Multimodal Hash retrieval method based on progressive fusion and hierarchical anchor learning

Abstract

The invention discloses a multi-modal hash retrieval method based on progressive fusion and hierarchical anchor point learning, and belongs to the field of image-text matching in multimedia information processing. And aligning the features of different scales by using a multi-head cross attention mechanism, and progressively fusing the aligned features to obtain a fused feature. And then, combining anchor point guidance learning, intra-layer anchor point learning, multi-label comparison learning and hierarchical anchor point consistency learning to obtain a total objective function, and jointly optimizing a feature encoder and a hash code generator by coordinating a loss function until a total objective function value is converged. And finally, carrying out multi-modal retrieval by utilizing the learned hash code, and calculating the retrieval accuracy. Compared with other methods, the method has the advantages that the retrieval accuracy is improved, and the performance is more stable.

Description

Technical Field

[0001] This invention belongs to the field of image-text matching in multimedia information processing, and specifically relates to a multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning. Background Technology

[0002] In the field of multimedia information processing, images and text are widely used and easily accessible data modalities. Hash retrieval analysis of multimodal image-text retrieval datasets can reduce the computational time complexity of models, allowing for the full extraction of potential semantic information from each modality. However, the high dimensionality and semantic complexity of raw image-text data lead to a decline in hash code discriminative power and the neglect of differences in semantic understanding between different levels. Existing multimodal retrieval methods sometimes directly use global features to generate hash codes or employ simple dimensionality reduction, ignoring fine-grained features and their correlation with global features. This poses a significant challenge to the accuracy of multimodal image-text data retrieval, necessitating effective technical methods to address the retrieval problem of multimodal image-text data. Summary of the Invention

[0003] To address the aforementioned problems, this invention provides a multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning, the method comprising the following steps:

[0004] The original features are extracted from the multimodal image and text retrieval dataset and normalized. The features corresponding to each modality are further processed using a perceptron and mapped to a latent space of the same dimension. Multi-head cross-attention is then used to align the processed image and text modal features at multiple levels, resulting in aligned fine-grained features, semantic-level features, and global features. These different levels of features are then progressively fused to obtain new features.

[0005] Based on the new features obtained, anchor points are constructed at different levels as semantic centers of each layer. At the same time, mid-level semantic anchor points are used as fine-grained anchor points for teacher guidance and global anchor points. Anchor point guidance loss and intra-layer anchor point loss are calculated.

[0006] The aligned features at different levels are compared and learned with the corresponding low-level fine-grained anchors, mid-level semantic anchors and high-level global anchors. The multi-label contrast loss is calculated, and the class prediction of the samples in the dataset is performed at different levels. The prediction results are subject to cross-level consistency constraints to obtain the hierarchical anchor consistency loss.

[0007] The overall objective loss function of the multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning is obtained by using anchor-guided loss, intra-layer anchor learning, hierarchical anchor consistency learning, and multi-label contrastive loss. Based on the obtained overall objective loss function, the network parameters are optimized using the Adam optimizer until the overall objective loss function value converges, generating hash codes for image and text retrieval.

[0008] The retrieval accuracy on the multimodal image and text retrieval dataset is calculated by using the obtained hash code for retrieval.

[0009] Furthermore, the step of normalizing the feature vectors to obtain the feature matrix... The formula is as follows:

[0010]

[0011] Where I represents the image modality, T represents the text modality, and x ini Let x represent the original feature vector. min and x max These represent the minimum and maximum values ​​of the vector across all samples, respectively.

[0012] The formula for refining the normalized feature vector using a multilayer perceptron to obtain feature X is as follows:

[0013]

[0014] Where MLP(·) represents a multilayer perceptron containing a ReLU activation function and a softmax normalization function, θ represents trainable parameters, and ⊙ represents matrix multiplication.

[0015] The multi-head cross-attention mechanism projects I and T into the query Q, key K, and value V space, expressed by the following formulas:

[0016]

[0017] Where dim represents the hierarchy, low represents the low level, mid represents the mid level, and high represents the high level. dim and T dim This represents the features of I and T at the dim level. and These represent the query Q, key K, and value V weight matrices at the dim level, respectively. and For I, the query Q, key K, and value V vector at the dim level. and Given a query Q, key K, and value V vector of T at the dim level; concatenate the key and value sequences of the image modality and the text modality to obtain the key K. XSum V X The formula is as follows:

[0018]

[0019] in, This represents the key K vector of mode X at the dim level. V represents the value vector of mode X at the dim level.

[0020] The aligned multi-level features are obtained through cross-modal bidirectional alignment. and The formula is as follows:

[0021]

[0022] Where, d t The dimension of query Q is τ1, which is the attention temperature parameter used to smooth the distribution of the normalization function softmax.

[0023] The adaptive fusion weight α of the dim level is calculated based on the aligned dim level features. dim The formula is as follows:

[0024]

[0025] Here, GatingNet(·) represents a lightweight gating network.

[0026] The obtained adaptive fusion weight α is utilized dim A progressive fusion process is performed to obtain the final fused representation. The formula is as follows:

[0027]

[0028] Where ⊙ denotes element-wise multiplication, α dim This represents the adaptive weights at different levels.

[0029] Furthermore, the construction of anchor points at the dim level... The formula is as follows:

[0030]

[0031] in, N represents the feature matrices at different levels. c Let l be the number of samples in the c-th category. If the label of the i-th sample is l i If it is class c, then the indicator function W is 1; otherwise, it is 0.

[0032] The anchor point guidance loss L teach The formula is as follows:

[0033]

[0034] Where B represents the current batch size, These are the lower-level weight matrix and the higher-level weight matrix on mode X, respectively. Let ||·|| represent the anchor centers of the lower and higher layers of mode X, respectively, and ||·|| represent the Euclidean norm.

[0035] The intralayer anchor point loss L proto The formula is as follows:

[0036]

[0037] Where C represents the total number of categories in all samples, and τ2 is a temperature parameter that controls the distribution size.

[0038] Furthermore, the similarity s between the label of sample i at the dim level and all anchor points at the corresponding level. i,c The formula is as follows:

[0039]

[0040] Where cos(·) represents the cosine similarity, Let X be the feature of the i-th sample at the dim level. Let τ be the anchor point of category c at the dim level, and τ3 be the temperature hyperparameter.

[0041] The similarity s between the label of sample i at the dim level and the corresponding positive anchor point at the same level. i,c' The formula is as follows:

[0042]

[0043] in, τ4 is the positive anchor point of category c at the dim level, and τ4 is the temperature hyperparameter.

[0044] The contrastive learning loss L at each level InfoNCE (·), expressed by the formula:

[0045]

[0046] Where B represents the current batch size, and C represents the total number of anchor point categories at the current level. L represents the total number of positive anchor point categories at the current level. i Let |·| represent the multi-label vector of the i-th sample, and |·| represent the magnitude.

[0047] The multi-label contrast loss L proxy The formula is as follows:

[0048]

[0049] Among them, L i Let i represent the multi-label vector of the i-th sample.

[0050] The dataset samples are then used for category prediction based on different levels of anchor points, with low-level category prediction... The formula is as follows:

[0051]

[0052] Among them, D low The Euclidean distance between the low-level fine-grained features of the sample and the low-level anchor point is specifically defined as:

[0053]

[0054] Among them, P low This indicates the center of the anchor point at the lower level.

[0055] The mid-level prediction results The formula is as follows:

[0056]

[0057] Among them, P mid τ5 represents the center of the anchor point of the middle layer, and τ5 represents the temperature parameter corresponding to the middle layer.

[0058] The high-level forecast results The formula is as follows:

[0059]

[0060] Wherein, τ6 represents the temperature parameter corresponding to the high-rise building;

[0061] The cross-layer consistency loss L consist The formula is as follows:

[0062]

[0063] Among them, when the loss is minimized, the difference between anchor points between different levels is minimized, so the feature information maintains a high degree of similarity at different levels.

[0064] Furthermore, the overall objective loss function L is expressed as follows:

[0065] L=αL proto +βL teach +ηL proxy +λL consistency

[0066] Where α is the loss L proto The weighting parameter, β is L teach The weighting parameter, η, is L. proxy The weighting parameter, λ, is L. consistency The weight parameters are then used. The Adam optimizer is applied to the obtained overall objective loss function to optimize the network parameters until the loss function converges, generating a binary hash code for multimodal retrieval.

[0067] Furthermore, multimodal retrieval is performed based on the obtained hash code, and the retrieval accuracy on the multimodal image and text retrieval dataset is calculated.

[0068] This invention provides a multimodal image and text retrieval method, which has the following advantages:

[0069] (1) The method uses a multi-head cross-attention mechanism to perform hierarchical feature alignment on image and text data. The resulting features can make up for the semantic gap caused by modal heterogeneity at different levels, and can select more accurate semantic features between different levels and different modalities, so that the resulting features can fully represent the semantic information of image and text data at different levels.

[0070] (2) The method adopts an anchor point construction strategy, which makes similar samples close to each other and heterogeneous samples far apart. The resulting anchor points, as category centers, can better distinguish easily confused information in the data, effectively enhancing the discriminative power of the generated hash code.

[0071] (3) The method constructs a hierarchical anchor system including low-level detail anchors, mid-level semantic anchors, and high-level global anchors, and uses cross-level consistency loss to constrain the semantic consistency of each anchor. The resulting anchors at different levels can better utilize the semantic information at different levels in the data, thereby fully considering comprehensive semantic features and more effectively reducing the impact of outliers and noise.

[0072] (4) The method adopts a multi-label contrastive learning approach, which can better handle multi-label data and maintain a good retrieval accuracy when dealing with the retrieval problem of complex data in real-world scenarios. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1This is a flowchart of the multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning provided by the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0076] Exemplary methods

[0077] like Figure 1 This invention provides a multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning, the method steps of which are as follows:

[0078] Step S110: Obtain the features by normalizing the feature vectors. Its formula is as follows:

[0079]

[0080] Where, x ini Let x represent the original feature vector. min and x max Let I and T represent the minimum and maximum values ​​of the vector across all samples, respectively. Let I represent the image modality and T represent the text modality. The features are further processed using an image multilayer perceptron and a text multilayer perceptron to obtain feature X, expressed as follows:

[0081]

[0082] in, θ represents the normalized feature vector, MLP(·) represents a three-layer sequential network containing the ReLU activation function and the softmax normalization function, θ represents the trainable parameters, and ⊙ represents the matrix multiplication operation.

[0083] The I and T are projected into the query Q, key K, and value V space using a multi-head cross-attention mechanism, expressed as follows:

[0084]

[0085] Where dim represents the hierarchy, low represents the low level, mid represents the mid level, and high represents the high level. and For I, the query Q, key K, and value V vector at the dim level. and Let I be the query Q, key K, and value V vector of T at the dim level. dim and T dim This represents the features of I and T at the dim level. and This represents the weight matrix of query Q, key K, and value V at the dim level.

[0086] The key K is obtained by concatenating the key K vector and value V vector sequences of the image modality and the text modality. X Sum V X The formula is as follows:

[0087]

[0088] in, This represents the key K vector of mode X at the dim level. V represents the value vector of mode X at the dim level.

[0089] Different hierarchical features were obtained through cross-modal bidirectional alignment. and The formula is as follows:

[0090]

[0091] Where, d t The dimension of query Q is defined by τ1, which is the attention temperature parameter used to smooth the softmax distribution. The adaptive fusion weight α of the dim level is calculated using the aligned dim level features. dim The formula is as follows:

[0092]

[0093] Here, GatingNet(·) represents a lightweight gating network.

[0094] The final fused representation is obtained by progressively fusing features at different levels. The formula is as follows:

[0095]

[0096] Here, ⊙ represents element-wise multiplication.

[0097] Step S120: Using the obtained fusion features Construct anchor center at the dim level Its formula is as follows:

[0098]

[0099] in, This represents the anchor center of class c in modality X along the dim dimension. N represents the feature matrices at different levels. c Let c be the number of samples in the c-th category, and l be the number of samples in the i-th category. i If it is class c, then the indicator function W is 1; otherwise, it is 0.

[0100] The obtained mid-level semantic anchors are used to guide the generation process of high-level global anchors and low-level fine-grained anchors through anchor guidance loss, and the formula is as follows:

[0101]

[0102] Where B represents the current batch size, These are the weight matrices for lower-level and higher-level teachers in the X-mode, respectively. Let ||·|| represent the anchor center of the lower and higher layers of the X mode, respectively, and let ||·|| represent the Euclidean norm.

[0103] The anchor point loss L within each layer is calculated using the obtained anchor points at each level. proto Its formula is as follows:

[0104]

[0105] Where C represents the total number of categories in all samples, and τ2 is a temperature parameter that controls the distribution size.

[0106] Step S130: Using aligned features Category anchors at the corresponding level Comparative learning is performed, and the similarity s between the label of sample i at the dim level and all anchor points at the corresponding level is calculated. i,c Its formula is as follows:

[0107]

[0108] Where cos(·) represents the cosine similarity, Let X be the feature of the i-th sample at the dim level. Let τ be the anchor point of category c at the dim level, and τ3 be the temperature hyperparameter.

[0109] The similarity s between the label of sample i at the dim level and the corresponding positive anchor point at the dim level i,c' Its formula is as follows:

[0110]

[0111] in, τ4 is the positive anchor point of category c at the dim level, and τ4 is the temperature hyperparameter.

[0112] Contrastive learning loss L at each level InfoNCE (·), its formula is as follows:

[0113]

[0114] Where B represents the current batch size, and C represents the total number of anchor point categories at the current level. L represents the total number of positive anchor point categories at the current level. i Let |·| represent the multi-label vector of the i-th sample, and |·| represent the magnitude.

[0115] Calculate the multi-label contrast loss L proxy Its formula is as follows:

[0116]

[0117] Among them, L i Let i represent the multi-label vector of the i-th sample.

[0118] Class prediction is performed on samples in the dataset hierarchically using anchor points at different levels. The low-level prediction results... Its formula is as follows:

[0119]

[0120] Among them, D low The Euclidean distance between the low-level fine-grained features of the sample and the low-level anchor point is defined as follows:

[0121]

[0122] Among them, P low This indicates the center of the anchor point at the lower level.

[0123] Mid-level prediction results Its formula is as follows:

[0124]

[0125] Where τ5 represents the temperature parameter corresponding to the middle layer.

[0126] High-level forecast results Its formula is as follows:

[0127]

[0128] Where τ6 represents the temperature parameter corresponding to the high-rise building.

[0129] The cross-layer consistency loss L is calculated by performing cross-layer consistency learning on the obtained multi-layer anchor points. consist Its formula is as follows:

[0130]

[0131] Among them, when the loss is minimized, the difference between anchor points between different levels is minimized, so the feature information maintains a high degree of similarity at different levels.

[0132] Step S140: Obtain the overall objective loss function L through intra-layer anchor point constraints, anchor point guidance constraints, hierarchical anchor point consistency constraints, and multi-label contrast constraints, as shown below:

[0133] L=αL proto +βL teach +ηL proxy +λL consistency

[0134] Where α is the loss L proto The weighting parameter, β is L teach The weighting parameter, η, is L. proxy The weighting parameter, λ, is L. consistency The weight parameters are then used. The network parameters are optimized using the Adam optimizer until the loss function converges, generating binary hash codes for multimodal retrieval.

[0135] Step S150: Calculate the retrieval accuracy on the multimodal image and text retrieval dataset using the obtained hash code.

[0136] In this implementation, the original features extracted from the multimodal image-text retrieval dataset are first refined using a multilayer perceptron. The features from different layers, aligned using multi-head cross-attention, are then progressively fused to obtain fused features. Next, a combined multi-label contrastive learning, anchor-guided learning, intra-layer anchor learning, and multi-layer anchor consistency learning are used to obtain the overall objective loss function. The feature encoder and hash code generator are jointly optimized by coordinating the loss function until the overall objective loss function converges, generating a binary hash code. Finally, the learned hash code is used for multimodal retrieval, and the retrieval accuracy is calculated.

[0137] To further illustrate, if a multimodal image and text retrieval dataset is retrieved according to this implementation method, a retrieval result with an accuracy higher than most methods will be obtained.

[0138] Detailed Implementation Results

[0139] This implementation uses the publicly available MIRFlickr-25K dataset, specifically 20,015 valid instances. The dataset details are described below:

[0140] The MIRFlickr-25K dataset contains image-text pairs, with image feature dimensions of 4096 and text feature dimensions of 1386, and includes 24 semantic categories.

[0141] To verify the superiority of this implementation method, it will be compared with several existing multimodal retrieval methods, including PMD-PCH, CLIPMH, PMFH, and DBSTH. The MAP values ​​of these methods when generating 16-bit, 32-bit, 64-bit, and 128-bit hash codes on the MIRFlickr-25K public dataset will be compared. The specific data comparison is shown in Table 1.

[0142] Table 1. Retrieval accuracy (%) of the MIRFlickr-25K dataset.

[0143]

[0144] The data comparison in the table above clearly shows that Ours achieves the best performance, significantly improving the accuracy of image and text retrieval. The quantitative results fully demonstrate the superiority of Ours, as it can more fully utilize fine-grained information and its correlation with global information, obtaining richer semantic representations by fully utilizing features at different levels. Ours improves the retrieval accuracy of image and text retrieval by performing multi-level anchor point processing on image and text pairs, fully utilizing semantic information while maintaining semantic consistency at different levels. Extensive experiments show that this method outperforms existing methods. Regarding the parameter settings for this implementation, in all experiments, the parameters α, β, η, and λ were uniformly tuned between 1e-6 and 1e2, the learning rate was set to 0.001, the batch size was set to 1024, and the number of training epochs was 150.

[0145] This implementation proposes a multimodal image-text retrieval method for analyzing common multimodal image-text data. It utilizes a multilayer perceptron and multi-head cross-attention mechanism to obtain a feature matrix that contains more semantic information and reduces the semantic gap between modalities. Then, hierarchical anchors are constructed at multiple levels, and anchor guidance loss, intra-layer anchor loss, and cross-layer consistency loss are calculated. A total objective loss function is obtained through multi-label contrastive loss, combined with anchor guidance learning, intra-layer anchor learning, and cross-layer consistency learning. This function is then optimized using an Adam optimizer until convergence. A hash code generation layer is then used to obtain binary hash codes, and finally, the final retrieval result is obtained using these hash codes. Experimental results on the public dataset MIRFlickr-25K, generating hash codes of four different lengths, demonstrate that this implementation achieves higher retrieval accuracy and superior performance compared to other methods.

[0146] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning, characterized in that, The method includes the following steps: Extract raw features from a multimodal image-text retrieval dataset and normalize them; then, analyze the normalized image features... and text features Image features I and text features T are generated by mapping them to a latent space of the same dimension using an image perceptron and a text perceptron, respectively. The resulting image features I and text features T are then aligned at different levels using a multi-head cross-attention mechanism, yielding aligned fine-grained features, semantic-level features, and global features. These features are then progressively fused to obtain new features. Based on the obtained fusion characteristics Anchor points are constructed at different levels as semantic centers for each level, including low-level fine-grained anchor points P. low Mid-level semantic anchor P mid and high-level global anchor point P high and using mid-level semantic anchor P mid P, as a fine-grained anchor point for teacher guidance low and global anchor point P high The construction of anchor point guidance loss and intra-layer anchor point loss is performed. Based on the alignment of different hierarchical features, comparative learning is performed with all learnable category anchors, and multi-label contrastive loss is calculated; the obtained hierarchical anchors are used to predict the categories of samples in the dataset at different levels, and cross-level consistency constraints are applied to the prediction results, and hierarchical anchor consistency loss is calculated. The overall objective loss function of the multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning is obtained by in-layer anchor learning, anchor guidance loss, hierarchical anchor consistency learning, and multi-label contrastive loss. Based on the obtained overall objective loss function, the network parameters are optimized using the Adam optimizer until the loss function converges, generating binary hash codes for multimodal retrieval. Multimodal retrieval is performed based on the obtained hash code, and the retrieval accuracy on the multimodal image and text retrieval dataset is calculated.

2. The multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning according to claim 1, characterized in that, The original features extracted from the multimodal image and text retrieval dataset are normalized to obtain new features. As shown below: Where, x ini Represents the original feature, x min and x max Let I and T represent the minimum and maximum values ​​of the feature across all samples, respectively. Let I represent the image modality and T represent the text modality. The normalized image and text features are further processed by an image perceptron and a text perceptron, respectively, to obtain the refined feature X, as shown below: in, Let represent the normalized features, MLP(·) denote a multilayer perceptron containing ReLU activation and softmax functions, θ represent trainable parameters, and ⊙ represent matrix multiplication. Project I and T onto the query Q, key K, and value V space, as shown below: Where dim represents hierarchy, low represents low hierarchy, mid represents mid hierarchy, high represents high hierarchy, and I represents high hierarchy. dim and T dim This represents the features of I and T at the dim level. and These represent the query Q, key K, and value V weight matrices at the dim level, respectively. and Let I be the query q, key K, and value V vector at the dim level. and Given a query Q, key K, and value V vector of T at the dim level; concatenate the key and value sequences of the image modality and the text modality to obtain the key K. X Sum V X As shown below: in, This represents the key K vector of mode X at the dim level. Represent the value vector V of mode X at the dim level; the resulting key K X Sum V X And query Q X Perform cross-modal bidirectional alignment to obtain aligned multi-level features. and As shown below: Where, d t To define the dimension of query Q, τ1 is the attention temperature parameter used to smooth the distribution of the normalization function softmax(·); the adaptive fusion weight α of the dim level is calculated based on the aligned dim level features. dim As shown below: Wherein, GatingNet(·) represents a lightweight gated network; using the obtained adaptive fusion weights α dim A progressive fusion process is performed to obtain the final fused representation. As shown below: Here, ⊙ represents element-wise multiplication.

3. The multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning according to claim 1, characterized in that, Based on the obtained fusion characteristics Build anchors at the dim level As shown below: in, N represents the feature matrices at different levels. x Let l be the number of samples in the x-th category. If the label of the i-th sample is l i If the class is c, the indicator function W is 1; otherwise, it is 0. Based on the obtained mid-level semantic anchors, the loss L is guided by the anchors. teach This guides the generation process of high-level global anchors and low-level fine-grained anchors, as shown below: Where B represents the current batch size, These are the weight matrices for lower-level teachers and the weight matrices for higher-level teachers, respectively. Let L represent the anchor point centers of the lower and upper levels, respectively, and ||·|| represent the Euclidean norm. Calculate the intra-level anchor point loss L using the obtained anchor points for each level. proto As shown below: Where C represents the total number of categories in all samples, and τ2 is a temperature parameter that controls the distribution size.

4. The multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning according to claim 1, characterized in that, Based on the aligned features Category anchors at the corresponding level Comparative learning is performed, and the similarity s between the label of sample i at the dim level and all anchor points at the corresponding level is calculated. i,c As shown below: in, Let X be the feature of the i-th sample at the dim level. Let τ3 be the anchor point of category c at the dim level, τ3 be the temperature hyperparameter, and cos(·) represent the cosine similarity; s is the similarity s between the label of sample i at the dim level and the corresponding positive anchor point at the dim level. i,c′ As shown below: in, Let τ4 be the positive anchor point for category c at the dim level, and let τ4 be the temperature hyperparameter; calculate the contrastive learning loss L at each level. InfoNCE (·), as shown below: Where B represents the current batch size, and C represents the total number of label categories at the current level. L represents the total number of positive anchor point categories at the current level. i Let L represent the multi-label vector of the i-th sample, and |·| represent the magnitude; calculate the multi-label contrastive loss L. proxy As shown below: Among them, L i This represents the multi-label vector of the i-th sample; based on the anchor points obtained at each level, class prediction is performed on the samples in the dataset hierarchically, and the low-level prediction results are... As shown below: Among them, D low The Euclidean distance between the low-level fine-grained features and the low-level anchor points of the sample is shown below: Among them, P low Indicates the anchor center of the lower layer; the prediction result of the middle layer. As shown below: Among them, P mid The anchor point center of the middle layer is indicated, and τ5 represents the temperature parameter corresponding to the middle layer; the prediction results for the upper layers are as follows. As shown below: Among them, P high The anchor point center of the high-rise is represented by τ6, and the temperature parameter corresponding to the high-rise is represented by τ6. Cross-layer consistency loss L is calculated by performing cross-layer consistency learning on the obtained multi-layer anchor points. consist As shown below: Among them, when the loss is minimized, the difference between anchor points between different levels is minimized, so the feature information maintains a high degree of similarity at different levels.

5. The multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning according to claim 1, characterized in that, The overall objective loss function is obtained based on intra-layer anchor point constraints, anchor point guidance constraints, hierarchical anchor point consistency constraints, and multi-label comparison constraints, as shown below: L=αL proto +βL teach +ηL proxy +λL consistency Where α is the loss L proto The weighting parameter, β is L teach The weighting parameter, η, is L. proxy The weighting parameter, λ, is L. consistency The weight parameters are determined. Based on the obtained overall objective loss function, the Adam optimizer is used to optimize the network parameters until the loss function converges, generating binary hash codes for multimodal retrieval.

6. The multimodal hash retrieval method based on progressive fusion and hierarchical anchor learning according to claim 1, characterized in that, Multimodal retrieval is performed based on the obtained hash code, and the retrieval accuracy on the multimodal image and text retrieval dataset is calculated.

Images