A two-stage graph hashing based cross-modal retrieval method

Abstract

This invention provides a cross-modal retrieval method based on two-stage graph hashing. The method includes: learning improved hash codes from a sample database to obtain the hash codes of samples in the database; inputting the sample to be retrieved; obtaining the hash code of the sample to be retrieved using an optimized hash function; comparing the hash code of the sample to be retrieved with the hash codes of samples in the database; and outputting the sample in the database with the highest similarity to the hash code of the sample to be retrieved. This invention can remove noise information and perform cross-modal retrieval more accurately and effectively.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval technology, and more specifically to a cross-modal retrieval method based on two-stage graph hashing. Background Technology

[0002] Technical Background: With the development of the information age, the volume of various types of data is unprecedentedly large, such as images, text, videos, and 3D models. Data from different modalities may express the same valuable high-level semantics, so how to capture the connections from one modality to another has become an important research area. In the past decade, a technique called cross-modal retrieval has developed rapidly, which aims to search for similarities between different modalities. For example, we can retrieve relevant text describing images or obtain videos introducing 3D models. However, the distribution and dimensionality of different modalities vary greatly, also known as heterogeneity between modalities, which brings great difficulties to retrieval. Original methods based on common latent subspaces have provided some solutions. Specifically, they map each modality to the same subspace so that the distance between them can be calculated using the same metric. However, some problems still need to be further considered, such as the slow speed when processing large-scale data. To achieve efficient retrieval, hash-based cross-modal retrieval methods have been proposed. Hash methods have become a research hotspot due to their low time and space costs. Specifically, hash methods compress the original multimodal data into a low-dimensional Hamming space and represent their features with binary code. Then, the similarity between different modalities can be determined through effective XOR operations, thereby effectively performing the retrieval. Summary of the Invention

[0003] The purpose of this invention is to provide a cross-modal retrieval method based on two-stage graph hashing, which can remove noise information and perform cross-modal retrieval more accurately and effectively.

[0004] This invention proposes a cross-modal retrieval method based on two-stage graph hashing, comprising:

[0005] Improved hash code learning is performed on the sample library to obtain the hash codes of the samples in the sample library;

[0006] Enter the sample you want to retrieve;

[0007] Use an optimized hash function to obtain the hash code of the sample to be retrieved;

[0008] The hash code of the sample to be retrieved is compared with the hash codes of samples in the sample library, and the sample with the highest similarity to the hash code of the sample to be retrieved is output.

[0009] Improved hash code learning is performed on the sample database to obtain the hash codes of the samples in the database, including:

[0010] The original features of the sample library are transformed into kernel feature representations of the sample library using radial basis function kernels.

[0011] The kernel features of the sample library are decomposed using matrix factorization, so that each modality corresponds to a subspace;

[0012] Extracting non-Euclidean features from the subspace;

[0013] Hash codes are learned by employing a non-symmetric strategy in the subspace.

[0014] Using radial basis function kernels to transform the original features of the sample library into kernel feature representations of the sample library includes:

[0015] The original features are transformed into kernel feature representations using radial basis function kernels:

[0016]

[0017] Where x represents image or text data, and E = {E1, E2, ..., E} K} represents k samples randomly selected from the training set, where n represents the number of training samples. Indicates the kernel width.

[0018] The kernel features of the sample database are decomposed using matrix factorization, so that each modality corresponds to a subspace, specifically:

[0019]

[0020] This represents the image data after kernelization. This represents the kernelized text data, where k1 represents... The feature dimension, k2 represents Feature dimensions, and Represents the mapping matrix. and V represents a specific latent semantic space, where the dimension of the latent space V is set to the length r of the hash code.

[0021] Extracting non-Euclidean features from the subspace specifically involves:

[0022] Calculate the label distance between two samples, in order to... To measure the similarity between two samples;

[0023] Construct an adjacency matrix:

[0024] The learned subspace should approximate the output of the graph convolutional network, achieved by the following loss term:

[0025]

[0026] Where α is the equilibrium parameter. and The mapping matrix that needs to be learned. It is a focus matrix, and each element in the matrix

[0027] A global approximation strategy is used to align the two potential subspaces:

[0028]

[0029] β is the equilibrium parameter.

[0030] Learning hash codes from subspaces using non-symmetric strategies includes:

[0031]

[0032] st B∈{-1,1} r×n .

[0033] Where S = A is the similarity matrix;

[0034] The overall objective function is:

[0035]

[0036] st B∈{-1,1} r×n .

[0037] Transform the objective function into trace form:

[0038]

[0039] st B∈{-1,1} r×n When optimizing one variable, keep the other variables fixed and set their gradients to zero to obtain the optimal solution for these variables.

[0040] The optimal solution for U1:

[0041] The optimal solution for U2:

[0042] The optimal solution for W1:

[0043] The optimal solution for W2:

[0044] The optimal solution for V1: V1 = P1 -1 F1;

[0045] The optimal solution for V2:

[0046] in,

[0047] When optimizing B, other parameters are fixed, and the objective function is simplified as follows:

[0048]

[0049] st B∈{-1,1} r×n .

[0050] After discarding the constant term:

[0051] The optimal solution for B is: B = sign(Y)

[0052] in,

[0053] Using an optimized hash function, the hash code of the sample to be retrieved is obtained, including:

[0054] Use a linear classifier as the hash function:

[0055]

[0056] Q t It is the mapping matrix for the t-th mode, and the balance parameter η is set to 0.01;

[0057] The hash function is modified as follows:

[0058]

[0059] Optimize the above equation and take its derivative:

[0060]

[0061] Q t Optimal solution:

[0062] The sample to be retrieved is processed by a hash function to obtain the hash code of the sample to be retrieved.

[0063] A cross-modal retrieval system based on two-stage graph hashing includes:

[0064] The data acquisition module is used to acquire the samples to be retrieved;

[0065] The data processing module is used to obtain the hash code of the sample to be retrieved using an optimized hash function, and compare the hash code of the sample to be retrieved with the hash codes of samples in the sample library.

[0066] The data output module is used to output the sample in the sample library that has the highest similarity to the sample whose hash code is to be retrieved.

[0067] A computer device includes a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the above-described cross-modal retrieval method based on two-stage graph hashing.

[0068] A storage medium, which is a computer-readable storage medium, stores a computer program that, when executed by a processor, implements the aforementioned cross-modal retrieval method based on two-stage graph hashing.

[0069] This invention improves existing hash code learning by removing noise information from hash codes, eliminating the loss of quantization errors, reducing the impact of noise information on hash codes, and improving accuracy. This invention also proposes an optimized hash function to balance efficiency and accuracy, improving the efficiency and accuracy of converting samples into hash codes, thereby improving the efficiency and accuracy of cross-modal retrieval. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 The flowchart shows a cross-modal retrieval method based on two-stage graph hashing proposed in this invention.

[0073] Figure 2 This is a system architecture diagram of a cross-modal retrieval method based on two-stage graph hashing proposed in this invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0075] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0076] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0077] Example 1

[0078] This invention proposes a cross-modal retrieval method based on two-stage graph hashing, comprising:

[0079] S100, Improved hash code learning is performed on the sample library to obtain the hash code of the sample in the sample library;

[0080] S200, Enter the sample to be retrieved;

[0081] S300, use an optimized hash function to obtain the hash code of the sample to be retrieved;

[0082] S400 compares the hash code of the sample to be retrieved with the hash codes of samples in the sample library, and outputs the sample in the sample library with the highest similarity to the hash code of the sample to be retrieved.

[0083] This invention improves existing hash code learning by removing noise information from hash codes, eliminating the loss of quantization errors, reducing the impact of noise information on hash codes, and improving accuracy. This invention also proposes an optimized hash function to balance efficiency and accuracy, improving the efficiency and accuracy of converting samples into hash codes, thereby improving the efficiency and accuracy of cross-modal retrieval.

[0084] S100 performs improved hash code learning on the sample database, obtaining the hash codes of the samples in the database, including:

[0085] S101, using radial basis function kernels to transform the original features of the sample library into kernel feature representations of the sample library;

[0086] S102, use matrix decomposition to decompose the kernel features of the sample library so that each modality corresponds to a subspace;

[0087] S103, extract non-Euclidean features in the subspace;

[0088] S104, learns hash codes from subspaces using a non-symmetric strategy.

[0089] S101, using radial basis function kernels to transform the original features of the sample library into kernel feature representations of the sample library includes:

[0090] The original features are transformed into kernel feature representations using radial basis function kernels:

[0091]

[0092] Where x represents image or text data, and E = {E1, E2, ..., E} K} represents k samples randomly selected from the training set, where n represents the number of training samples. Indicates the kernel width.

[0093] S102 uses matrix decomposition on the kernel features of the sample database, so that each modality corresponds to a subspace, specifically:

[0094]

[0095] This represents the image data after kernelization. This represents the kernelized text data, where k1 represents... The feature dimension, k2 represents Feature dimensions, and Represents the mapping matrix. and V represents a specific latent semantic space, where the dimension of the latent space V is set to the length r of the hash code.

[0096] Collective matrix factorization (CMF) is an excellent feature dimensionality reduction technique that can remove redundant information across different modalities. Some cross-modal methods utilize it to learn a common space, which may be unreasonable given the different distributions of images and text. Therefore, this invention learns a specific latent space for each modality.

[0097] S103 extracts non-Euclidean features in the subspace, specifically as follows:

[0098] Calculate the label distance between two samples, in order to... To measure the similarity between two samples;

[0099] Construct an adjacency matrix:

[0100] The learned subspace should approximate the output of the graph convolutional network, achieved by the following loss term:

[0101]

[0102] Where α is the equilibrium parameter. and The mapping matrix that needs to be learned. It is a focus matrix, and each element in the matrix

[0103] A global approximation strategy is used to align the two potential subspaces:

[0104]

[0105] β is the equilibrium parameter.

[0106] Subspaces learned through matrix factorization may be affected by noisy data and outliers in the Euclidean domain. Inspired by graph convolutional networks, this invention extracts non-Euclidean features for subspaces, which can reduce the impact of noisy data and outliers in the Euclidean domain.

[0107] S104 learns hash codes from subspaces using a non-symmetric strategy, including:

[0108]

[0109] st B∈{-1,1} r×n .

[0110] Where S = A is the similarity matrix;

[0111] The overall objective function is:

[0112]

[0113] st B∈{-1,1} r×n .

[0114] Transform the objective function into trace form:

[0115]

[0116] st B∈{-1,1} r×n When optimizing one variable, keep the other variables fixed and set their gradients to zero to obtain the optimal solution for these variables.

[0117] The optimal solution for U1:

[0118] The optimal solution for U2:

[0119] The optimal solution for W1:

[0120] The optimal solution for W2:

[0121] The optimal solution for V1: V1 = P1 -1 F1;

[0122] The optimal solution for V2:

[0123] in,

[0124] When optimizing B, other parameters are fixed, and the objective function is simplified as follows:

[0125]

[0126] st B∈{-1,1} r×n .

[0127] After discarding the constant term:

[0128] The optimal solution for B is: B = sign(Y)

[0129] in,

[0130] The S300 uses an optimized hash function to obtain the hash code of the sample to be retrieved, including:

[0131] Use a linear classifier as the hash function:

[0132]

[0133] Q t It is the mapping matrix for the t-th mode, and the balance parameter η is set to 0.01;

[0134] The hash function is modified as follows:

[0135]

[0136] Optimize the above equation and take its derivative:

[0137]

[0138] Q t Optimal solution:

[0139] The sample to be retrieved is processed by a hash function to obtain the hash code of the sample to be retrieved.

[0140] Although linear classifiers are effective, they ignore the correlation between samples, which can lead to poor robustness to outliers. Therefore, this invention further considers... To capture the relationship between a pair of samples in different spaces, where Similarity moments are constructed based on cosine relations. Specifically, if the class labels of two samples are strongly correlated, the mapping result (QX)i and the hash code bj should be similar to each other.

[0141] Example 2

[0142] A cross-modal retrieval system based on two-stage graph hashing includes:

[0143] The data acquisition module is used to acquire the samples to be retrieved;

[0144] The data processing module is used to obtain the hash code of the sample to be retrieved using an optimized hash function, and compare the hash code of the sample to be retrieved with the hash codes of samples in the sample library.

[0145] The data output module is used to output the sample in the sample library that has the highest similarity to the sample whose hash code is to be retrieved.

[0146] Example 3

[0147] A computer device includes a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the above-described cross-modal retrieval method based on two-stage graph hashing.

[0148] Example 4

[0149] A storage medium, which is a computer-readable storage medium, stores a computer program that, when executed by a processor, implements the aforementioned cross-modal retrieval method based on two-stage graph hashing.

[0150] This invention improves existing hash code learning by removing noise information from hash codes, eliminating the loss of quantization errors, reducing the impact of noise information on hash codes, and improving accuracy. This invention also proposes an optimized hash function to balance efficiency and accuracy, improving the efficiency and accuracy of converting samples into hash codes, thereby improving the efficiency and accuracy of cross-modal retrieval.

[0151] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A two-stage graph hashing based cross-modal retrieval method, characterized in that, include: Improved hash code learning is performed on the sample library to obtain the hash codes of the samples in the sample library; The improved hash code learning of the sample library, which obtains the hash codes of samples in the sample library, includes: The original features of the sample library are transformed into kernel feature representations of the sample library using radial basis function kernels. The kernel features of the sample library are decomposed using matrix factorization, so that each modality corresponds to a subspace; The step of using matrix decomposition to decompose the kernel features of the sample library so that each modality corresponds to a subspace is as follows: ； This represents the image data after kernelization. This represents the kernelized text data, where k1 represents... The feature dimension, k2 represents Feature dimensions, and Represents the mapping matrix. and V represents a specific latent semantic space, and the dimension of the latent space V is set to the length r of the hash code; Extracting non-Euclidean features from the subspace; Learn hash codes using an asymmetric strategy from subspaces; Enter the sample you want to retrieve; Use an optimized hash function to obtain the hash code of the sample to be retrieved; The process of obtaining the hash code of the sample to be retrieved using an optimized hash function includes: Use a linear classifier as the hash function: ； It is the mapping matrix for the t-th mode, and the balancing parameters are set. ; The hash function is modified as follows: ； Optimize the above formula and take its derivative: ； Optimal solution: ; The sample to be retrieved is processed by a hash function to obtain the hash code of the sample to be retrieved; The hash code of the sample to be retrieved is compared with the hash codes of samples in the sample library, and the sample with the highest similarity to the hash code of the sample to be retrieved is output.

2. The two-stage graph hashing based cross-modal retrieval method according to claim 1, characterized in that, The process of converting the original features of the sample library into kernel feature representations of the sample library using radial basis function kernels includes: The original features are transformed into kernel feature representations using radial basis function kernels: ； wherein x is data that is an image or text, represents k samples randomly selected from the training set, and n represents the number of training samples, represents the kernel width.

3. The two-stage graph hashing based cross-modal retrieval method according to claim 1, characterized in that, The extraction of non-Euclidean features in the subspace specifically involves: Calculate the label distance between two samples, in order to... To measure the similarity between two samples; Construct an adjacency matrix: ; The learned subspace should approximate the output of the graph convolutional network, achieved by the following loss term: ； where a is an equilibrium parameter, and is a mapping matrix to be learned, is a focusing matrix, each element of the matrix ; A global approximation strategy is used to align the two potential subspaces: ； β is the equilibrium parameter.

4. The two-stage graph hashing based cross-modal retrieval method according to claim 1, characterized in that, The method of learning hash codes from subspace using an asymmetric strategy includes: ； Where S = A is the similarity matrix; The overall objective function is: ； Transform the objective function into trace form: Fixing other variables and setting their gradients to 0 when optimizing a variable gives the optimal solution for those variables Optimal solution for U1: ; Optimal solution for U2: ; Optimal solution for W1: ; Optimal solution for W2: ; Optimal solution for V1: ; Optimal solution for V2: ; wherein , , , ; When optimizing B, other parameters are fixed, and the objective function is simplified as follows: ； After discarding the constant term: ; The optimal solution for B is: ; wherein .

5. A two-stage graph hashing based cross-modal retrieval system, applied to the two-stage graph hashing based cross-modal retrieval method of any one of claims 1-4, characterized in that, include: The data acquisition module is used to acquire the samples to be retrieved; The data processing module is used to obtain the hash code of the sample to be retrieved using an optimized hash function, and compare the hash code of the sample to be retrieved with the hash codes of samples in the sample library. The data output module is used to output the sample in the sample library that has the highest similarity to the sample whose hash code is to be retrieved.

6. A computer device comprising a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements a cross-modal retrieval method based on two-stage graph hashing as described in any one of claims 1-4.

7. A storage medium, which is a computer-readable storage medium, stores a computer program, wherein when the program is executed by a processor, the processor executes the computer program stored in the memory to implement the cross-modal retrieval method based on two-stage graph hashing as described in any one of claims 1-4.

Images