A Zero-Sample Cross-Modal Retrieval Method Based on Adaptive Class-Related Discrete Hashing

Abstract

This invention discloses a zero-shot cross-modal retrieval method based on adaptive class-related discrete hashing. A novel cross-modal zero-shot hashing method is proposed to effectively transfer class attribute knowledge. This method constructs a semantically enhanced embedding by fusing label information with class attribute information, which can solve the problem of class attribute correspondence for multi-label instances. By learning the semantically enhanced embedding, more semantic information is embedded into the feature representation, thereby balancing the retrieval results between images and text. This method fully considers the correlation between class attributes and adaptively embeds more class attribute semantic information into the hash code. Simultaneously, the hash code can effectively capture the relationship between visible and invisible classes, thus transferring attribute knowledge from visible classes to invisible classes. Finally, pairwise similarity is embedded in the hash code learning process to enhance the semantic information in the hash code. This invention improves retrieval accuracy in zero-shot cross-modal retrieval scenarios.

Description

Technical Field

[0001] This invention relates to a zero-sample cross-modal retrieval method based on adaptive class-related discrete hashing, belonging to the field of zero-sample cross-modal hashing retrieval. Background Technology

[0002] In recent years, with the continuous development of the internet and technology, a wealth of multimedia data, including images, text, audio, and video, has emerged on various social media platforms. Although these multimedia data belong to different modalities, they contain data expressing the same meaning. How to search for data expressing the same meaning from such multimodal data is a current need in information mining. Furthermore, with the continuous emergence of new concepts, we cannot train on all classes. Therefore, achieving high-precision retrieval of new concepts (invisible classes) without retraining the model with new concept samples is our goal. However, most existing cross-modal hashing retrieval methods solve standard cross-modal retrieval problems, meaning they are only suitable for retrieval scenarios where the number of classes in the training and query sets completely overlap. For the constantly emerging cross-modal datasets containing new concepts, the existing standard cross-modal retrieval methods are not well-suited to such retrieval scenarios.

[0003] Zero-shot hashing retrieval methods have attracted widespread attention in order to solve the retrieval problem of new concepts (invisible classes). However, most existing zero-shot hashing retrieval methods are proposed for single-modal retrieval tasks, and these methods cannot be directly applied to cross-modal zero-shot retrieval tasks. Methods for cross-modal zero-shot retrieval tasks are also constantly emerging. Some researchers learn hash codes using features and class attributes simultaneously, and embed the local structural information of the modality into the hash codes. Other researchers learn three different hash codes for images, text, and class attributes respectively, and use the hash codes corresponding to the class attributes to connect the image and text modalities. This method can not only reduce the gap between modalities, but also realize the transfer of attribute information of visible classes to invisible classes. Although existing cross-modal zero-shot hashing methods have been applied to zero-shot retrieval tasks, there are still some challenges that need to be further addressed: (1) In cross-modal retrieval tasks, there are some imbalanced retrieval results between text query images and image query text. (2) Most algorithms are only applicable to single-label datasets, so they cannot solve the problem of invisible class retrieval on multi-label datasets. (3) Existing methods cannot effectively transfer attribute knowledge between categories, resulting in unsatisfactory retrieval performance. Summary of the Invention

[0004] In view of the above challenges, this invention provides a zero-shot cross-modal retrieval method based on adaptive class-related discrete hashing. This invention constructs a semantically enhanced embedding by fusing tag information and class attribute information; then, it integrates the semantically enhanced embedding, class attribute correlation, pairwise similarity, and feature matrix into the same framework to establish a zero-shot cross-modal retrieval model based on adaptive class-related discrete hashing, which can be used to solve the cross-modal retrieval problem containing invisible class data.

[0005] To achieve the objectives of this invention, the technical solution of a zero-shot cross-modal retrieval method based on adaptive class-related discrete hashing is as follows: This invention proposes a novel zero-shot hashing framework to achieve more efficient inter-class attribute knowledge transfer. This method combines label information and category attribute information to construct semantically enhanced embeddings, solving the problem of category attribute correspondence for multi-label instances. Furthermore, by learning semantically enhanced embeddings to incorporate more semantic information for feature representation, its goal is to balance retrieval results between images and text. Simultaneously, this method adaptively learns the correlation between category attributes and then embeds more category attribute information into the hash code. Finally, pairwise similarity is used to constrain the learning of the hash code, thereby generating a more discriminative hash code. This invention includes the following steps:

[0006] Step 1: Feature extraction from cross-modal datasets: Feature extraction is performed on images and text in the original cross-modal dataset, and attribute vectors are extracted from the classes contained in the dataset;

[0007] Step 2, Dataset Partitioning: Based on the requirements of the zero-shot cross-modal retrieval scenario, the existing cross-modal dataset is divided into datasets that meet the zero-shot cross-modal retrieval scenario;

[0008] Step 3: Establish a zero-shot cross-modal retrieval model based on adaptive class-related discrete hashing. First, construct semantically enhanced embeddings using label information and class attribute information. Then, integrate semantically enhanced embeddings, class attribute correlations, pairwise similarity, and feature matrices into the same framework to establish a zero-shot cross-modal retrieval model based on adaptive class-related discrete hashing.

[0009] Step 4, Model Update and Optimization: Design an iterative discrete optimization algorithm to solve the model obtained in Step 3 until the model converges or reaches the maximum number of iterations, and obtain the hash code and hash function of the training set.

[0010] Step 5: Zero-shot cross-modal retrieval: First, obtain the hash codes of the retrieval set and the query set using the hash function trained in Step 4. Then, put the query set into the retrieval set for querying. Obtain the query result by calculating the Hamming distance between the hash code of the query set and the hash codes of each sample in the retrieval set. The result with the smallest Hamming distance is the final query result.

[0011] Furthermore, in Step 1, the cross-modal retrieval dataset is stored in pairs, with each sample pair including: text, image, and corresponding semantic label.

[0012] Furthermore, the feature extraction method in Step 1 is as follows: image features are extracted using the VGG-16 model; text features are extracted using the bag-of-words model; and a corresponding word vector is extracted for each category name using the GloVe method to form a class attribute matrix.

[0013] Furthermore, the dataset partitioning method described in Step 2 is as follows: First, select 10% of the samples in the original dataset as the query set, and the remaining samples as the training set; Second, randomly select 20% of the classes from all classes in the entire dataset as invisible classes, and the remaining classes as visible classes; Third, to conform to the zero-shot cross-modal retrieval scenario, repartition the training set, query set, and retrieval set in the dataset. The partitioning method is as follows: use the sample pairs corresponding to the invisible classes in the query set obtained in Step 1 as the new query set; use the sample pairs corresponding to the visible classes in the training set obtained in Step 1 as the new training set; the retrieval set consists of the training set obtained in Step 1.

[0014] Furthermore, in Step 2, to ensure the model's generalization ability, the visible and invisible classes are randomly re-split each time the model is trained to obtain new training and query sets. The average of multiple training iterations is then used as the final result.

[0015] Furthermore, in Step 3, when building the model, both linear and non-linear relationships exist in image and text features. A linear model can only capture the linear relationships within the features. To capture the non-linear relationships, a kernel function is added to the feature matrix to extend it into a non-linear model.

[0016] Furthermore, the semantically enhanced embedding in Step 3 is constructed by fusing label information and class attribute information; the constructed semantically enhanced embedding fuses multiple attribute vectors corresponding to a multi-label instance into a single attribute vector, thus making it suitable for multi-label datasets; at the same time, this semantically enhanced embedding can embed more semantic information into the feature representation to balance the retrieval results between images and text.

[0017] Furthermore, in Step 3, the correlation between class attributes is fully considered in the model, and more class attribute semantic information is adaptively embedded into the hash code; in addition, pairwise similarity is embedded in hash code learning to generate more discriminative hash codes.

[0018] Furthermore, the model established in Step 3 comprises two parts: hash code learning and hash function learning. First, semantically enhanced embeddings are constructed using label information and class attribute information. Then, the hash code is learned by integrating the semantically enhanced embeddings, class attribute correlations, pairwise similarities, and feature representations into the same framework. Second, the hash function is learned by minimizing a least-squares regression problem. Finally, hash code learning and hash function learning are trained in the same model, whose objective function is:

[0019]

[0020]

[0021] in, γ represents the regularization term of the model, used to prevent overfitting; γ is the parameter controlling the regularization term; φ(X) and φ(Y) are the matrices obtained by kernel functions for the features of the image and text modalities, respectively; E is the semantic enhancement embedding; t1 and t2 are the drifts corresponding to the two modalities, respectively; e n It is an n whose elements are all 1s s 12-dimensional column vectors; S is the pairwise similarity matrix, Z is the correlation matrix between class attributes; W1, W2, and P are projection matrices; V is the hash code library, B is the hash code; μ, α, β, and θ are non-negative parameters, n s is the total number of visible samples, and k is the hash code length.

[0022] Furthermore, the construction of semantically enhanced embeddings in Step 3 is achieved by fusing label information and class attribute information; the specific fusion method is as follows: the fusion method calculates a semantically enhanced embedding vector for each sample, and then combines these vectors into the semantically enhanced embedding of the entire training set. The method for calculating the semantically enhanced embedding vector for a sample is to multiply the class attribute matrix by the label vector corresponding to the sample and then divide by the total number of categories corresponding to the sample, thereby obtaining the semantically enhanced embedding vector corresponding to the sample.

[0023] Furthermore, the correlation between class attributes in Step 3 refers to exploring the correlation between each class attribute and other class attributes. A correlation matrix is ​​used to explore the relationship between class attributes, thereby improving the effectiveness of transferring attribute knowledge from visible to invisible classes.

[0024] Furthermore, in Step 3, the pairwise similarity is used to establish the relationship between pairwise similarity and hash code through an optimization model based on kernel-supervised hashing (KSH), so as to enhance the semantic information in the hash code by embedding similarity into the hash code. In order to avoid the NP-hard problem in the optimization process, this method adopts an asymmetric strategy to embed pairwise semantic similarity.

[0025] Furthermore, the model update optimization in Step 4 uses the objective function corresponding to the model established in Step 3 as the original function for updating. Clearly, the objective function corresponding to this model is not optimal and needs optimization. This objective function is a non-convex problem, which makes optimization difficult. However, when other variables are fixed to update a matrix variable, a non-convex problem is transformed into a convex problem, facilitating iterative updates. This alternating iterative algorithm is used to update the matrix variable until the objective function converges or reaches the maximum number of iterations, ultimately obtaining the optimal hash code and hash function.

[0026] Furthermore, the objective function optimization steps are as follows:

[0027] W1-step: With fixed variables W2, t1, t2, P, Z, B, and V, formula (1) simplifies to a problem concerning W1:

[0028]

[0029] By setting the partial derivative of W1 to zero, the closed-form solution of W1 is obtained as follows:

[0030] W1=(φ(X)φ(X) T +γI) -1 φ(X)(E-t1e n ) T (3)

[0031] W2-step: Similar to the update steps of W1, the closed solution of W2 is obtained as follows:

[0032] W2=(φ(Y)φ(Y) T +γI) -1 φ(Y)(E-t2e n ) T (4)

[0033] t1-step: With variables W1, W2, t2, P, Z, B, and V fixed, formula (1) simplifies to a problem concerning t1:

[0034]

[0035] By setting the partial derivative of t1 to zero, the closed-form solution of t1 is obtained as follows:

[0036]

[0037] t2-step: Similar to the update steps of t1, the closed-form solution of t2 can be obtained as follows:

[0038]

[0039] Z-step: With variables W1, W2, t1, t2, P, B, and V fixed, formula (1) simplifies to a problem concerning Z:

[0040]

[0041] By setting the partial derivative of Z to zero, the closed-form solution of Z is obtained as follows:

[0042] Z=(μP T P+βI) -1 (βEE T +μP T BE T )((β+μ)EE T ) -1 (9)

[0043] P-step: With variables W1, W2, t1, t2, Z, B, and V fixed, formula (1) simplifies to a problem concerning P:

[0044]

[0045] By setting the partial derivative of P to zero, the closed-form solution of P is obtained as follows:

[0046] P = μB(ZE) T (μZE(ZE) T +γI) -1 (11)

[0047] B-step: Fix variables W1, W2, t1, t2, Z, P, and V. Then formula (1) simplifies to a problem concerning B:

[0048]

[0049]

[0050] We use Discrete Cyclic Coordinate Descent (DCC) to update B; therefore, problem (12) is rewritten as follows:

[0051]

[0052]

[0053] Where Q = αkVS + μPZE + θV; specifically, B is updated bit by bit; let b be the l-th column of B, and Let V be the matrix B after removing b; let v be the l-th column of V, and Let q be the matrix V after removing v; let q be the l-th column of Q, and It is the matrix Q after removing q; according to the DCC method, the solution for b is as follows:

[0054]

[0055] V-step: Similar to the update step of B, the closed solution of v can be obtained as:

[0056]

[0057] Where Q1 = αkBS + θB, and let q1 be the l-th column of Q1;

[0058] The above steps are used to optimize formula (1) until the function converges or the maximum number of iterations is reached, at which point the iteration stops and the optimal hash code and hash function are obtained.

[0059] The beneficial effects of this invention are:

[0060] This invention proposes a zero-shot cross-modal retrieval method based on adaptive class-related discrete hashing. It overcomes the following limitations of existing zero-shot cross-modal retrieval methods: (1) In cross-modal retrieval tasks, there are imbalanced retrieval results between text-query-image and image-query-text. (2) Most algorithms are only applicable to single-label datasets, thus failing to address the problem of retrieval of invisible classes on multi-label datasets. (3) Existing methods cannot effectively transfer attribute knowledge between categories, resulting in unsatisfactory retrieval performance. Specifically, this method combines label information and category attribute information to construct a semantically enhanced embedding, solving the problem of category attribute correspondence for multi-label instances. Furthermore, by learning the semantically enhanced embedding to incorporate more semantic information for feature representation, its goal is to balance the retrieval results between images and text. Simultaneously, this method adaptively learns the correlation between category attributes and then embeds more category attribute information into the hash code. Then, pairwise similarity is used to constrain the learning of the hash code, thereby generating a more discriminative hash code. Furthermore, this invention proposes a discrete optimization scheme to solve the proposed model, effectively avoiding quantization errors. Attached Figure Description

[0061] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention.

[0062] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0063] Figure 2 This is a flowchart illustrating the iterative update process of the method of the present invention.

[0064] Figure 3 This is an architectural diagram of the method of the present invention.

[0065] Figure 4 This is an experimental graph comparing the top-N curves of the method of this invention with other cross-modal hash retrieval methods. Detailed Implementation

[0066] The following descriptions are illustrative and are intended to further illustrate the technical solutions of the present invention in conjunction with the accompanying drawings.

[0067] Example 1

[0068] Figure 1 This is a flowchart of a zero-sample cross-modal retrieval method based on adaptive class-related discrete hashing according to the present invention.

[0069] In this example, see Figure 1 The method of the present invention specifically includes the following processes:

[0070] Step 1: Cross-modal dataset feature extraction. This step involves extracting features from the images and text in the original cross-modal dataset, as well as extracting attribute vectors for the classes contained in the dataset. In this example, the dataset includes two modalities: images and text, and corresponding labels. In Step 1, the class attribute extraction uses the GloVe method to extract a corresponding word vector for each class name, forming a class attribute matrix.

[0071] Step 2: Dataset Partitioning: Since the problem this invention aims to solve is cross-modal retrieval that includes invisible classes, the already partitioned cross-modal dataset cannot be used directly. The dataset should be repartitioned according to the zero-shot application scenario to make it suitable for zero-shot cross-modal retrieval. The specific processing method is as follows:

[0072] The first step involves selecting 10% of the samples in the original dataset as the query set and the remaining samples as the training set. The second step involves randomly selecting 20% ​​of the classes from all classes in the entire dataset as invisible classes, and the remaining classes as visible classes. The third step, to conform to the zero-shot cross-modal retrieval scenario, re-partitions the training set, query set, and retrieval set in the dataset. The partitioning method is as follows: the sample pairs corresponding to the invisible classes in the query set obtained in the first step are used as the new query set; the sample pairs corresponding to the visible classes in the training set obtained in the first step are used as the new training set; and the retrieval set consists of the training set obtained in the first step.

[0073] In this invention, using Represents a cross-modal dataset, where o i =(x i ,y i ) represents a pair of image and text, where n is the total number of samples. and These are the feature matrices for the image and text modalities, respectively. Typically, the image feature dimension d... x With text feature dimension d y It should satisfy d x ≠d y Furthermore, the corresponding label matrix in the dataset is defined as L = {l1, l2, ..., l...} n}∈R c×n , where c represents the total number of categories. Using Represents the class attribute matrix, where d A This refers to the dimension of the class attributes. In the zero-sample experimental scenario, [the following will be used]: Data sets divided into visible classes and invisible class datasets Where n s and n u These represent the total number of visible class samples and the total number of invisible class samples, respectively. Similarly, label L is divided into... and Then L s and L u Should satisfy Constraints.

[0074] Step 3: Establish a zero-shot cross-modal retrieval model based on adaptive class-related discrete hashing. First, construct semantically enhanced embeddings using label information and class attribute information. Then, integrate semantically enhanced embeddings, class attribute correlations, pairwise similarity, and feature matrices into the same framework to establish a zero-shot cross-modal retrieval model based on adaptive class-related discrete hashing.

[0075] The semantically enhanced embedding in Step 3 is constructed by fusing label information and class attribute information. The constructed semantically enhanced embedding fuses multiple attribute vectors corresponding to a multi-label instance into a single attribute vector, thus making it suitable for multi-label datasets. At the same time, this semantically enhanced embedding can embed more semantic information into the feature representation to balance the retrieval results between images and text.

[0076] In Step 3, the correlation between class attributes is fully considered in the model, and more class attribute semantic information is adaptively embedded into the hash code; in addition, pairwise similarity is embedded in hash code learning to generate more discriminative hash codes.

[0077] The learning models for each module will be described in detail below:

[0078] Step 3.1, Semantic Enhancement Embedding Learning

[0079] Because the semantic information embedded in image visual features and text features differs, they cannot be guaranteed to fit well into a common semantic space, leading to imbalanced retrieval results between images and text. To address this issue, a semantically enhanced embedding is constructed to augment the semantic information of modal feature data with weak semantic information. This addresses the problem of embedding sufficient semantic information into manually extracted visual and text features. The method projects image and text features onto this semantically enhanced embedding to realign them. This embedded embedding contains enough semantic information to bridge the gap between image and text features and common semantics. The semantically enhanced embedding method involves fusing the class label matrix and the class attribute matrix. The representation corresponding to the i-th instance in the semantically enhanced embedding is calculated using the following formula:

[0080]

[0081] Where L si It is L s The i-th column, ∑L si Let E represent the sum of the number of labels of all semantic embeddings of different classes obtained by the i-th instance. However, semantic embedding E allows all instances to have a one-to-one corresponding semantic vector in the semantically enhanced embedding.

[0082] Secondly, by combining semantically enhanced embeddings and a hash function constructed using least-squares linear regression, this method projects the features of both modalities into the semantically enhanced embeddings. The function for learning the semantically enhanced embeddings can be expressed as:

[0083]

[0084] Where W1 and W2 are the projection matrices of the image and text, respectively. However, simply using the linear transformation described above to balance the query results between the image and text is far from sufficient. Therefore, this method adds a drift to the linear transformation in equation (2) above to narrow the gap between the query results of the image and text. Equation (2) can be reformulated as:

[0085]

[0086] Where t1 and t2 are the drifts of the two modes, e n It is an n whose elements are all 1 s The dimensional row vector. Furthermore, the semantically enhanced embedding E in formula (3) incorporates both labels and class attributes, as well as the feature matrix. Therefore, the semantically enhanced embedding can achieve the transfer of attribute knowledge from visible classes to invisible classes.

[0087] Step 3.2, Adaptive Class Relevance Learning

[0088] Supervised cross-modal hashing methods typically employ linear regression to embed label information into hash codes, as described below:

[0089]

[0090] Where P is the projection matrix and B is the hash code. By minimizing equation (4), the hash code containing supervised information can be obtained. However, combined with the semantically enhanced embedding learning described above, this method replaces the label information embedding in equation (4) with semantically enhanced embedding to learn hash codes richer in semantic information. Therefore, equation (4) can be reformulated as:

[0091]

[0092] In practical cross-modal retrieval applications, instances in the dataset are multi-labeled, and there are potential semantic correlations between the labels. Considering these semantic correlations could provide more useful information for hash code learning. However, similar to the potential semantic correlations between labels, semantic correlations between class attributes should also exist in the semantically enhanced embeddings used in this method. To explore the correlations between each class attribute and other class attributes, we assume that for each instance, the predicted information... Based on its own predictions and its correlation with other class attributes, namely:

[0093]

[0094] Where Z represents the correlation matrix between class attributes. It can be considered as a sample x belonging to the i-th class attribute. j The predicted value is determined by the correlation between the i-th class attribute and other class attributes. By incorporating the correlation of class attributes into hash code learning, formula (5) can be rewritten as follows:

[0095]

[0096]

[0097] Step 3.3: Preservation of Asymmetric Pairwise Similarity

[0098] By combining the kernel-based supervised hashing (KSH) optimization model, the asymmetric pairwise similarity preservation method proposed in this invention embeds pairwise similarity into the hash code to generate more discriminative hash codes. Its model expression is shown below:

[0099]

[0100] Where k is the hash code length, and S is the pairwise similarity constructed using labels. To avoid calculating large-scale similarity matrices, this invention directly uses L during the optimization process. s T ×L s Let S be used instead of S for calculation. Furthermore, the inner product of hash codes in formula (9) will lead to an NP-hard problem in the optimization process. Therefore, this invention introduces a discrete auxiliary variable V as the hash code of the training database into the inner product of hash codes, and maintains pairwise similarity by adopting an asymmetric strategy. Formula (8) can be rewritten as follows:

[0101]

[0102]

[0103] Step 3.4: Constructing the objective function;

[0104] The model constructed in Step 3 consists of two parts: hash code learning and hash function learning. First, semantically enhanced embeddings are constructed using label information and class attribute information. Then, the semantically enhanced embeddings, correlations between class attributes, pairwise similarity, and feature representations are integrated into the same framework to learn the hash code. Second, the hash function is learned by minimizing the least squares regression problem. Finally, the hash code learning and hash function learning are trained in the same model.

[0105] In reality, the distribution of multimodal data in the original feature space is often highly nonlinear. However, traditional linear models cannot effectively capture their nonlinear relationships. To address this issue, this invention uses a kernel function to map nonlinear multimodal data to a high-dimensional feature space. By synthesizing the above components, the objective function of this invention is obtained as follows:

[0106]

[0107] in, γ represents the regularization term of the model, used to prevent overfitting; γ is the parameter controlling the regularization term; φ(X) and φ(Y) are the matrices obtained by kernel functions for the features of the image and text modalities, respectively; E is the semantic enhancement embedding; t1 and t2 are the drifts corresponding to the two modalities, respectively; e n It is an n whose elements are all 1s s 12-dimensional column vectors; S is the pairwise similarity matrix, Z is the correlation matrix between class attributes; W1, W2, and P are projection matrices; V is the hash code library, B is the hash code; μ, α, β, and θ are non-negative parameters, n s is the total number of visible samples, and k is the hash code length.

[0108] Step 4: Model Update and Optimization: Design an iterative discrete optimization algorithm to solve the model obtained in Step 3 until the model converges or reaches the maximum number of iterations, and obtain the hash code and hash function of the training set.

[0109] The model update and optimization in Step 4 is an optimization of the model obtained in Step 3. Obviously, the objective function corresponding to this model is not optimal and needs to be optimized. The objective function is a non-convex problem, which will bring difficulties to the optimization. However, when other variables are fixed to update a matrix variable, a non-convex problem is transformed into a convex problem, which facilitates iterative updates. An alternating iterative algorithm is used to update the matrix variable until the objective function converges or the maximum number of iterations is reached, and finally the optimal hash code and hash function are obtained.

[0110] Function (10) is not optimal, so it needs to be iteratively updated. Clearly, the overall objective function is a non-convex optimization problem. Therefore, this invention proposes an efficient iterative discrete optimization algorithm to solve this problem. Specifically, see [link to relevant documentation]. Figure 2 The optimization steps for formula (10) are as follows:

[0111] W1-step: Fix variables W2, t1, t2, P, Z, B, and V. Then formula (10) can be simplified to a problem concerning W1:

[0112]

[0113] By setting the partial derivative of W1 to zero, we can obtain the closed solution of W1 as follows:

[0114] W1=(φ(X)φ(X) T +γI) -1 φ(X)(E-t1e n ) T (12)

[0115] W2-step: Similar to the update steps of W1, the closed solution of W2 can be obtained as follows:

[0116] W2=(φ(Y)φ(Y) T +γI) -1 φ(Y)(E-t2e n ) T (13)

[0117] t1-step: With variables W1, W2, t2, P, Z, B, and V fixed, formula (10) can be simplified to a problem concerning t1:

[0118]

[0119] By setting the partial derivative of t1 to zero, we can obtain the closed-form solution of t1 as follows:

[0120]

[0121] t2-step: Similar to the update steps of t1, the closed-form solution of t2 can be obtained as follows:

[0122]

[0123] Z-step: With fixed variables W1, W2, t1, t2, P, B, and V, formula (10) can be simplified to a problem concerning Z:

[0124]

[0125] By setting the partial derivative of Z to zero, we can obtain the closed-form solution of Z as follows:

[0126] Z=(μP T P+βI) -1 (βEE T +μP T BE T )((β+μ)EE T ) -1 (18)

[0127] P-step: Fix variables W1, W2, t1, t2, Z, B, and V. Then formula (10) can be simplified to a problem concerning P:

[0128]

[0129] By setting the partial derivative of P to zero, we can obtain the closed-form solution of P as follows:

[0130] P = μB(ZE) T (μZE(ZE) T +γI) -1 (20)

[0131] B-step: Fix variables W1, W2, t1, t2, Z, P, and V. Then formula (10) can be simplified to a problem concerning B:

[0132]

[0133]

[0134] This invention uses Discrete Cyclic Coordinate Descent (DCC) to update B. Therefore, problem (21) can be rewritten as follows:

[0135]

[0136]

[0137] Where Q = αkVS + μPZE + θV. Specifically, this invention updates B bit by bit. Let b be the l-th column of B, and This is the matrix B after removing b. Similarly, let v be the l-th column of V, and... Let q be the matrix V after removing v. Let q be the l-th column of Q, and It is the matrix Q divided by q. According to the DCC method, the solution for b can be obtained as follows:

[0138]

[0139] V-step: Similar to the update step of B, the closed solution of v can be obtained as:

[0140]

[0141] Where Q1 = αkBS + θB, and let q1 be the l-th column of Q1.

[0142] The above steps are used to optimize formula (10) until the function converges or the maximum number of iterations is reached, at which point the iteration stops and the optimal hash code and hash function are obtained.

[0143] Step 5, Zero-Shot Cross-Modal Retrieval: First, obtain the hash codes of the retrieval set and query set using the hash function trained in Step 4. Then, place the query set into the retrieval set for querying. The query result is obtained by calculating the Hamming distance between the hash code of the query set and the hash codes of each sample in the retrieval set. The result with the smallest Hamming distance is the final query result. The specific implementation steps are as follows:

[0144] The feature matrix corresponding to the query sample, given the image and text, is φ(X). query ) and φ(Y query This is combined with the projection matrices W1 and W2 obtained during model training, as well as the drifts t1 and t2. Using formula B... x =sgn(PZ(W1) T φ(x query )+t1e n )) and B y =sgn(PZ(W2) T φ(Y query )+t2e n )) Obtain the hash code corresponding to the query sample.

[0145] Since the query task of this invention is performed in a binary space, the query result is obtained by calculating the Hamming distance between the query sample and each sample in the retrieval set. The sample with the smallest Hamming distance in the retrieval set is the obtained query result.

[0146] To illustrate the effects of the present invention, the technical solution of the present invention will be further described below through specific embodiments:

[0147] 1. Simulation conditions

[0148] This invention utilizes Matlab software for experimental simulation. Experiments were conducted on the cross-modal dataset MIRFlickr (containing both image and text modalities). The experiments included two query tasks: (1) text query for image (Text2Img), and (2) image query for text (Img2Text). The parameters were set as μ = 1e-3, α = 1e6, β = 1e-3, γ = 1e-3, and θ = 1e-7.

[0149] 2. Simulation Content

[0150] The proposed method in this invention, a zero-sample cross-modal retrieval method based on adaptive class-related discrete hashing (ZSDH-ACC), is compared with existing non-zero-sample cross-modal hashing retrieval methods and zero-sample cross-modal hashing retrieval methods. In contrast, non-zero-sample cross-modal hashing retrieval methods include: (1) Collective Matrix Factorization Hashing (CMFH), (2) Joint and Individual Matrix Factorization Hashing (JIMFH), (3) Discrete Robust Matrix Factorization Hashing (DRMFH), (4) Label Consistent Matrix Factorization Hashing (LCMFH), (5) Label Category Supervised Matrix Factorization Hashing (LCSMFH), and (6) Label Consistent Flexible Matrix Factorization Hashing (LFMH); zero-sample unimodal hashing retrieval methods include: (1) Attribute Hashing (AH) for zero-sample image retrieval; and zero-sample cross-modal hashing retrieval methods include: (1) Cross-Modal Attribute Hashing (CMAH), (2) Discrete Asymmetric Zero-Sample Hashing (DRMH). Asymmetric Zero-Shot Hashing (DAZSH). For zero-shot unimodal hash retrieval, hash codes for both image and text modalities are obtained through a unimodal model, and then the following query task is performed.

[0151] 3. Simulation Results

[0152] The simulation experiments present the comparison method and the experimental results of the proposed method on the MIRFlickr dataset. To conform to the zero-shot cross-modal retrieval scenario, 20% of the classes in the random dataset MIRFlickr are selected as invisible classes. The MIRFlickr dataset contains a total of 24 classes. According to the experimental settings, this embodiment randomly selects 5 classes as invisible classes, and the processing method for the remaining datasets is the same as that of this invention.

[0153] In this simulation experiment, a widely used metric is used to measure the performance of the proposed ZSDH-ACC method and other comparative methods: mean average precision (mAP). Given a query and a list of search results, the mean precision (AP) is defined as:

[0154]

[0155] Where N is the number of relevant instances in the retrieval set, and P(r) is defined as the precision of the r-th retrieval instance. If the r-th retrieval instance is a true neighbor of the query, then δ(r) = 1; otherwise, δ(r) = 0. Then, the average AP of all queries is used to obtain mAP. The evaluation rule is that the larger the mAP value, the better the performance. Furthermore, to more comprehensively evaluate the ZSDH-ACC method proposed in this invention, the top-N precision curve is also used as an evaluation metric. In this experiment, the precision curve shows the precision curve of the top 1000 samples retrieved by all methods, i.e., the top-1000 precision curve.

[0156] The hash code lengths obtained from the simulation experiments were 8 bits, 16 bits, 32 bits, 64 bits, and 128 bits. Tables 1 and 2 show the corresponding mAP values ​​of the ZSDH-ACC method proposed in this invention and other comparative methods.

[0157] Table 1 shows the mAP values ​​of all methods on the Text Query Image (Text2Img) task on the MIRFlickr dataset.

[0158]

[0159] Table 2 shows the mAP values ​​of all methods on the Image Query Text (Img2Text) task on the MIRFlickr dataset.

[0160]

[0161]

[0162] As can be seen from Tables 1 and 2, the mAP values ​​of the ZSDH-ACC method proposed in this invention are higher than those of other comparative methods in both query tasks under the zero-shot cross-modal retrieval scenario of the MIRFlickr dataset. This further demonstrates the superiority of the ZSDH-ACC method proposed in this invention in zero-shot cross-modal retrieval.

[0163] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A zero-sample cross-modal retrieval method based on adaptive class-related discrete hashing, characterized in that: The specific steps of the method are as follows: Step 1: Feature extraction from cross-modal datasets: Feature extraction is performed on images and text in the original cross-modal dataset, and attribute vectors are extracted from the classes contained in the dataset; Step 2, Dataset Partitioning: Divide the existing cross-modal dataset into datasets that meet the zero-sample cross-modal retrieval scenario; Step 3: Establish a zero-sample cross-modal retrieval model based on adaptive class-related discrete hashing: First, construct semantically enhanced embeddings using label information and class attribute information; Then, semantic enhancement embedding, class attribute correlation, pairwise similarity, and feature matrix are integrated into the same framework to establish a zero-sample cross-modal retrieval model based on adaptive class-related discrete hashing. Step 4, Model Update and Optimization: Design an iterative discrete optimization algorithm to solve the model obtained in Step 3 until the model converges or reaches the maximum number of iterations, and obtain the hash code and hash function of the training set. Step 5: Zero-shot cross-modal retrieval: First, obtain the hash codes of the retrieval set and the query set using the hash function trained in Step 4; then, put the query set into the retrieval set for querying, and obtain the query results by calculating the Hamming distance between the hash code of the query set and the hash codes of each sample in the retrieval set. The one with the smallest Hamming distance is the final query result. The specific method of Step 2 is as follows: First, select 10% of the samples in the original dataset as the query set, and the remaining samples as the training set; Second, randomly select 20% of the classes from all classes in the entire dataset as invisible classes, and the remaining classes as visible classes; Third, in order to conform to the zero-shot cross-modal retrieval scenario, re-divide the training set, query set, and retrieval set in the dataset; The method of division is to take the sample pairs corresponding to the invisible classes in the query set obtained in the first step as the new query set; take the sample pairs corresponding to the visible classes in the training set obtained in the first step as the new training set; The retrieval set consists of the training set obtained in the first step. The model constructed in Step 3 consists of two parts: hash code learning and hash function. First, semantically enhanced embedding is constructed using label information and class attribute information. Then, semantically enhanced embeddings, class attribute correlations, pairwise similarities, and feature representations are integrated into the same framework to learn hash codes; Secondly, the hash function is learned by minimizing the least squares regression problem; finally, hash code learning and hash function learning are trained in the same model; the objective function of this model is: ； ； （1） ； ； where, represents the regularization term of the model, which is used to prevent overfitting; is the parameter that controls the regularization term, and are matrices obtained by applying the kernel function to the features of the image and text modalities respectively; is the semantic enhancement embedding; and are the drifts corresponding to the two modalities respectively; is a - dimensional column vector with all elements equal to 1; is the pairwise similarity matrix, is the correlation matrix between class attributes; , , are projection matrices; is the hash code library, is the hash code; , , and are non - negative parameters, is the total number of visible class samples, is the length of the hash code.​​​​ 2. The zero-sample cross-modal retrieval method based on adaptive class-related discrete hashing according to claim 1, characterized in that: In Step 1, the GloVe method is used to extract a corresponding word vector for each class name to form a class attribute matrix.

3. The zero-sample cross-modal retrieval method based on adaptive class-related discrete hashing according to claim 1, characterized in that: The semantically enhanced embedding in Step 3 is constructed by fusing label information and class attribute information. The constructed semantically enhanced embedding fuses multiple attribute vectors corresponding to a multi-label instance into a single attribute vector, thus making it suitable for multi-label datasets. At the same time, this semantically enhanced embedding can embed more semantic information into the feature representation to balance the retrieval results between images and text.

4. The zero-sample cross-modal retrieval method based on adaptive class-related discrete hashing according to claim 1, characterized in that: In Step 3, the correlation between class attributes is fully considered in the model, and more class attribute semantic information is adaptively embedded into the hash code; in addition, pairwise similarity is embedded in hash code learning to generate more discriminative hash codes.

5. The zero-sample cross-modal retrieval method based on adaptive class-related discrete hashing according to claim 1, characterized in that: The model update and optimization in Step 4 is an optimization of the model obtained in Step 3. Obviously, the objective function corresponding to this model is not optimal and needs to be optimized. The objective function is a non-convex problem, which will bring difficulties to the optimization. However, when other variables are fixed to update a matrix variable, a non-convex problem is transformed into a convex problem, which facilitates iterative updates. An alternating iterative algorithm is used to update the matrix variable until the objective function converges or the maximum number of iterations is reached, and finally the optimal hash code and hash function are obtained.

6. The zero-sample cross-modal retrieval method based on adaptive class-related discrete hashing according to claim 5, characterized in that: The objective function optimization steps are as follows: -step: Fixed variables , , , , , and Then formula (1) simplifies to about Question: （2） ； By setting The partial derivatives are zero, therefore we get The closed solution is: （3）； -step: with The update steps are similar, resulting in... The closed solution is: （4）； -step: Fixed variables , , , , , and Then formula (1) simplifies to about Question: （5） ； By setting The partial derivatives are zero, therefore we get The closed solution is: （6）； -step: with The update steps are similar, and we can obtain... The closed solution is: （7）； -step: Fixed variables , , , , , and Then formula (1) simplifies to about Question: （8） ； By setting The partial derivatives are zero, therefore we get The closed solution is: （9）； -step: Fixed variables , , , , , and Then formula (1) simplifies to about Question: （10） ； By setting The partial derivatives are zero, therefore we get The closed solution is: （11）； -step: Fixed variables , , , , , and Then formula (1) simplifies to about Question: ； （12）； Update using Discrete Cyclic Coordinate Descent (DCC) Therefore, problem (12) is rewritten as follows: ； （13）； in Specifically, it updates bit by bit. of; set yes The Column, and It is a matrix Remove The matrix after; let yes The Column, and It is a matrix Remove The matrix after; let yes The Column, and It is a matrix Remove The resulting matrix; according to the DCC method, we obtain... The solution is as follows: （14）； -step: with The update steps are similar, and we can obtain... The closed solution is: ， （15）； in And set yes The List; The above steps are used to optimize formula (1) until the function converges or the maximum number of iterations is reached, at which point the iteration stops and the optimal hash code and hash function are obtained.

Images