Unsupervised cross-modal hash retrieval method, system and device based on implicit features

Abstract

The present application discloses an unsupervised cross-modal hash retrieval method, system and device based on implicit features. The method constructs a pseudo-label matrix; obtains a target sample implicit feature matrix by alternately iteratively optimizing the pseudo-label implicit feature matrix decomposition of the pseudo-label matrix and the sample implicit feature matrix; initializes a consensus representation matrix, and determines the projection matrix of each modal feature matrix and the target sample implicit feature matrix according to the target sample implicit feature matrix, the consensus representation matrix and each modal feature matrix; binarizes the consensus representation matrix into a hash code matrix, and constructs a target loss function according to the target sample implicit feature matrix, each modal feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix and the pseudo-label matrix; and determines the target hash code matrix and the target projection matrix according to the target loss function to perform cross-modal hash retrieval on the target modal data. The present application can improve the accuracy of cross-modal retrieval.

Description

Technical Field

[0001] The present application relates to the field of cross-modal retrieval technology, and in particular to an unsupervised cross-modal hash retrieval method, system, and device based on implicit features. Background Art

[0002] With the deep integration of the internet, multimedia technologies, and artificial intelligence, data is undergoing a paradigm shift from a single modality to a multimodal collaborative representation of images, text, audio, and video. This exponential growth of multimodal data presents a dual challenge for cross-modal retrieval systems: addressing the semantic gap between heterogeneous modalities, such as text retrieval for images and video understanding for audio, while also meeting the engineering requirements of real-time retrieval of terabytes of data.

[0003] Traditional multimodal hashing methods achieve cross-modal retrieval by aligning feature spaces or constructing shared semantic representations in low-dimensional spaces. These methods offer significant advantages in training efficiency and retrieval speed. However, these methods rely heavily on handcrafted feature modeling and rely heavily on supervised information. However, these methods face a "supervision-semantics double gap" in their utilization of supervised information. On the one hand, they rely heavily on labeled data, significantly reducing retrieval accuracy as the missing label rate increases. On the other hand, they fail to fully exploit the hierarchical structure and semantic representation of semantic labels, resulting in the generated hash codes failing to fully utilize the effective semantic information of the labels.

[0004] Therefore, the accuracy of existing cross-modal hash retrieval methods for cross-modal retrieval is relatively low. Summary of the Invention

[0005] This application aims to propose an unsupervised cross-modal hash retrieval method, system and device based on implicit features, which can improve the accuracy of cross-modal retrieval.

[0006] In a first aspect, an embodiment of the present application provides an unsupervised cross-modal hash retrieval method based on implicit features, the method comprising:

[0007] Extracting modal features from raw data of different modalities to construct a multimodal feature matrix, wherein the raw data of different modalities include text data, image data, video data, and audio data;

[0008] Performing pairwise canonical correlation analysis on every two modal feature matrices in the multimodal feature matrix to obtain typical variable matrices of multiple different modalities, and constructing a common potential representation matrix based on the typical variable matrices of the multiple different modalities;

[0009] Clustering the features in the common latent representation matrix to obtain a plurality of cluster centers, and calculating the distance between the cluster centers and each sample in the common latent representation matrix to construct a distance matrix, wherein the sample is a column in the common latent representation matrix;

[0010] Calculating the pseudo label value of each sample according to the distance matrix, and constructing a pseudo label matrix according to the pseudo label value of each sample;

[0011] Decomposing the pseudo label matrix into the product of a pseudo label latent feature matrix and a sample latent feature matrix, and optimizing the pseudo label latent feature matrix and the sample latent feature matrix through alternating iterations to obtain a target sample latent feature matrix;

[0012] Initializing a consensus representation matrix according to each modal feature matrix in the multimodal feature matrix, and determining a projection matrix of each modal feature matrix and the target sample latent feature matrix according to the target sample latent feature matrix, the consensus representation matrix, and each modal feature matrix;

[0013] Binarizing the consensus representation matrix into a hash code matrix, and constructing a target loss function based on the target sample implicit feature matrix, each modality feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix, and the pseudo label matrix;

[0014] According to the target loss function, a target hash code matrix and a target projection matrix are determined, and cross-modal hash retrieval is performed on the target modality data according to the target hash code matrix and the target projection matrix.

[0015] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0016] This method constructs a multimodal feature matrix by extracting modal features from original data of different modalities, where the original data of different modalities include text data, image data, video data, and audio data; performs pairwise canonical correlation analysis on each two modal feature matrices in the multimodal feature matrix to obtain typical variable matrices of multiple different modalities, and constructs a common latent representation matrix based on the typical variable matrices of multiple different modalities; clusters the features in the common latent representation matrix to obtain multiple cluster centers, and calculates the distance between the cluster centers and each sample in the common latent representation matrix to construct a distance matrix, where the sample is a column in the common latent representation matrix; calculates the pseudo label value of each sample based on the distance matrix, and constructs a pseudo label matrix based on the pseudo label value of each sample; decomposes the pseudo label matrix into a pseudo label latent feature matrix and the product of the sample latent feature matrix, and the target sample latent feature matrix is ​​obtained by alternately iteratively optimizing the pseudo-label latent feature matrix and the sample latent feature matrix; according to each modal feature matrix in the multimodal feature matrix, the consensus representation matrix is ​​initialized, and according to the target sample latent feature matrix, the consensus representation matrix and each modal feature matrix, the projection matrix of each modal feature matrix and the target sample latent feature matrix is ​​determined; the consensus representation matrix is ​​binarized into a hash code matrix, and the target loss function is constructed according to the target sample latent feature matrix, each modal feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix and the pseudo-label matrix; according to the target loss function, the target hash code matrix and the target projection matrix are determined, and the target modal data is cross-modal hash retrieval is performed based on the target hash code matrix and the target projection matrix. In this way, a pseudo-label matrix is ​​constructed by calculating the pseudo-label value of each sample. Later, a target loss function is constructed based on the pseudo-label matrix to obtain the optimized hash code matrix (i.e., target hash code matrix) and the optimized projection matrix (i.e., target projection matrix), thereby realizing cross-modal hash retrieval of target modal data. That is, by generating a pseudo-label matrix and making full use of pseudo-label information to deeply explore the semantic consistency between modal data, the problem of "supervision-semantic double gap" that exists in traditional multimodal hashing when using supervised information is solved, thereby improving the accuracy of cross-modal retrieval.

[0017] In some implementations, calculating a pseudo label value for each sample based on the distance matrix includes:

[0018] ;

[0019] in, represents the pseudo label value of each sample, Represents the distance matrix Middle Rank Elements of the column, Represents the distance matrix Middle Rank Elements of the column, represents the total number of cluster centers, represents a hyperparameter controlling the rate of distance decay.

[0020] In some embodiments, the step of optimizing the pseudo-label latent feature matrix and the sample latent feature matrix through alternating iterations to obtain a target sample latent feature matrix includes:

[0021] Obtaining a score for each sample, and constructing an iterative optimization loss function based on the score for each sample, the pseudo-label latent feature matrix, and the sample latent feature matrix;

[0022] Fixing the pseudo-label latent feature matrix and optimizing the sample latent feature matrix;

[0023] Fixing the sample latent feature matrix and optimizing the pseudo-label latent feature matrix;

[0024] The alternating iterative optimization is performed until the iterative optimization loss function converges, and the implicit feature matrix of the target sample is obtained.

[0025] In some embodiments, determining the projection matrices of each modality feature matrix and the target sample implicit feature matrix based on the target sample implicit feature matrix, the consensus representation matrix, and each modality feature matrix includes:

[0026] According to the consensus representation matrix and each modal feature matrix, a projection matrix of each modal feature matrix is ​​determined:

[0027] ;

[0028] According to the consensus representation matrix and the target sample implicit feature matrix, a projection matrix of the target sample implicit feature matrix is ​​determined:

[0029] ;

[0030] in, Indicates the The projection matrix of the modal feature matrix, and represents the hyperparameter, represents the consensus representation matrix, Indicates the modal characteristic matrix, represents transpose, represents the identity matrix, Represents the projection matrix of the target sample implicit feature matrix, Represents the target sample implicit feature matrix.

[0031] In some embodiments, constructing a target loss function based on the target sample implicit feature matrix, the feature matrix of each modality, the consensus representation matrix, the projection matrix, the hash code matrix, and the pseudo label matrix includes:

[0032] ;

[0033] ;

[0034] in, Indicates the The projection matrix of the modal feature matrix, Represents the projection matrix of the target sample implicit feature matrix, represents the consensus representation matrix, represents the hash code matrix, Indicates the number of modes, Represents the contribution used to balance each feature matrix in constructing the consensus representation matrix, Indicates the modal characteristic matrix, represents the F2 norm, represents the target sample implicit feature matrix, represents the pseudo label matrix, The number of bits representing the hash code, represents the number of samples, Represents the consensus representation matrix and Weiquan The product of vectors, express dimension vector, represents the identity matrix, 、 、 and represents a hyperparameter.

[0035] In some embodiments, determining a target hash code matrix and a target projection matrix according to the target loss function includes:

[0036] When optimizing the projection matrix, fixing other parameters in the target loss function, and extracting a functional formula related to the projection matrix from the target loss function to construct a first optimization function;

[0037] When optimizing the consensus representation matrix, fixing other parameters in the target loss function, and extracting a functional formula related to the consensus representation matrix from the target loss function to construct a second optimization function;

[0038] When optimizing the hash code matrix, fixing other parameters in the target loss function, and extracting a functional formula related to the hash code matrix from the target loss function to construct a third optimization function;

[0039] The target loss function is minimized by minimizing the first optimization function, the second optimization function, and the third optimization function, and a target hash code matrix and a target projection matrix are determined.

[0040] In some implementations, performing cross-modal hash retrieval on target modality data based on the target hash code matrix and the target projection matrix includes:

[0041] Obtaining a target projection matrix corresponding to the target modal data;

[0042] Using the target hash code matrix as a hash code database;

[0043] Extracting target feature vectors of target modal data;

[0044] Calculating a hash code of the target modality data according to a target projection matrix and the target eigenvector corresponding to the target modality data;

[0045] Calculate the Hamming distance between the hash code of the target modality data and the hash codes of all data in the hash code database, and perform cross-modality hash retrieval on the target modality data according to the Hamming distance.

[0046] In a second aspect, an embodiment of the present application further provides an unsupervised cross-modal hash retrieval system based on implicit features, the system comprising:

[0047] A feature matrix construction unit is used to extract modal features from raw data of different modalities to construct a multimodal feature matrix, wherein the raw data of different modalities include text data, image data, video data, and audio data;

[0048] a representation matrix construction unit, configured to perform pairwise canonical correlation analysis on each two modal feature matrices in the multimodal feature matrix to obtain typical variable matrices of multiple different modalities, and to construct a common potential representation matrix based on the typical variable matrices of the multiple different modalities;

[0049] a distance matrix construction unit, configured to cluster the features in the common latent representation matrix to obtain a plurality of cluster centers, and calculate the distance between the cluster centers and each sample in the common latent representation matrix to construct a distance matrix, wherein the sample is a column in the common latent representation matrix;

[0050] A pseudo label matrix construction unit, configured to calculate a pseudo label value for each sample according to the distance matrix, and to construct a pseudo label matrix according to the pseudo label value for each sample;

[0051] an alternating iterative optimization unit, configured to decompose the pseudo-label matrix into a product of a pseudo-label latent feature matrix and a sample latent feature matrix, and obtain a target sample latent feature matrix by alternating iteratively optimizing the pseudo-label latent feature matrix and the sample latent feature matrix;

[0052] a projection matrix determining unit, configured to initialize a consensus representation matrix according to each modal feature matrix in the multimodal feature matrix, and determine a projection matrix for each modal feature matrix and the target sample latent feature matrix according to the target sample latent feature matrix, the consensus representation matrix, and each modal feature matrix;

[0053] a loss function construction unit, configured to binarize the consensus representation matrix into a hash code matrix, and construct a target loss function based on the target sample implicit feature matrix, each modality feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix, and the pseudo label matrix;

[0054] A cross-modal hash retrieval unit is used to determine a target hash code matrix and a target projection matrix according to the target loss function, and perform a cross-modal hash retrieval on the target modality data according to the target hash code matrix and the target projection matrix.

[0055] In a third aspect, an embodiment of the present application also provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the unsupervised cross-modal hash retrieval method based on implicit features as described above.

[0056] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the unsupervised cross-modal hash retrieval method based on implicit features as described above.

[0057] It can be understood that the beneficial effects of the above-mentioned second to fourth aspects compared with the relevant technologies are the same as the beneficial effects of the above-mentioned first aspect compared with the relevant technologies. Please refer to the relevant description in the above-mentioned first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0059] Figure 1 1 is a flow chart of an embodiment of an unsupervised cross-modal hash retrieval method based on implicit features provided by the present application;

[0060] Figure 2 1 is a schematic diagram of the overall framework of the best embodiment of the unsupervised cross-modal hash retrieval method based on implicit features provided by this application;

[0061] Figure 3 1 is a schematic structural diagram of an embodiment of an unsupervised cross-modal hash retrieval system based on implicit features provided by the present application;

[0062] Figure 4 It is a structural diagram of an embodiment of the electronic device provided by this application. DETAILED DESCRIPTION

[0063] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0064] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0065] In the description of this application, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0066] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technical personnel in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0067] Traditional multimodal hashing methods achieve cross-modal retrieval by aligning feature spaces or constructing shared semantic representations in low-dimensional spaces. While these methods offer significant advantages in training efficiency and retrieval speed, they rely heavily on supervised information due to their handcrafted feature modeling. However, these methods suffer from a "supervision-semantics gap" in their utilization of supervised information. On the one hand, they rely heavily on labeled data, significantly decreasing retrieval accuracy when the missing label rate increases. On the other hand, they fail to fully exploit the hierarchical structure and semantic representation of semantic labels, resulting in the generated hash codes failing to fully utilize the effective semantic information of the labels. Consequently, existing cross-modal hashing retrieval methods exhibit relatively low cross-modal retrieval accuracy.

[0068] The problem of low cross-modal retrieval accuracy in existing cross-modal hash retrieval methods has not been solved. This application proposes an unsupervised cross-modal hash retrieval method, system and device based on implicit features.

[0069] Reference Figure 1 , a flow chart of an unsupervised cross-modal hash retrieval method based on implicit features provided in an embodiment of the present application. The unsupervised cross-modal hash retrieval method based on implicit features is applied to an electronic device, which may be a server or a mobile terminal. Figure 1 As shown, the unsupervised cross-modal hash retrieval method based on implicit features may include the following steps:

[0070] Step S100: extracting modal features from raw data of different modalities to construct a multimodal feature matrix, where the raw data of different modalities include text data, image data, video data, and audio data;

[0071] Step S200: performing pairwise canonical correlation analysis on every two modal feature matrices in the multimodal feature matrix to obtain typical variable matrices of multiple different modalities, and constructing a common potential representation matrix based on the typical variable matrices of the multiple different modalities;

[0072] Step S300: clustering the features in the common latent representation matrix to obtain multiple cluster centers, and calculating the distance between the cluster centers and each sample in the common latent representation matrix to construct a distance matrix, where a sample is a column in the common latent representation matrix;

[0073] Step S400: Calculate the pseudo label value of each sample according to the distance matrix, and construct a pseudo label matrix according to the pseudo label value of each sample;

[0074] Step S500: decompose the pseudo label matrix into the product of the pseudo label latent feature matrix and the sample latent feature matrix, and optimize the pseudo label latent feature matrix and the sample latent feature matrix through alternating iterations to obtain the target sample latent feature matrix;

[0075] Step S600: Initialize a consensus representation matrix based on each modal feature matrix in the multimodal feature matrix, and determine the projection matrix of each modal feature matrix and the target sample latent feature matrix based on the target sample latent feature matrix, the consensus representation matrix, and each modal feature matrix;

[0076] Step S700: Binarize the consensus representation matrix into a hash code matrix, and construct a target loss function based on the target sample implicit feature matrix, each modality feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix, and the pseudo label matrix;

[0077] Step S800: Determine a target hash code matrix and a target projection matrix according to a target loss function, and perform cross-modal hash retrieval on target modality data according to the target hash code matrix and the target projection matrix.

[0078] In this embodiment, a multimodal feature matrix is ​​constructed by extracting modal features from original data of different modalities, where the original data of different modalities include text data, image data, video data, and audio data; pairwise canonical correlation analysis is performed on each two modal feature matrices in the multimodal feature matrix to obtain typical variable matrices of multiple different modalities, and a common latent representation matrix is ​​constructed based on the typical variable matrices of multiple different modalities; features in the common latent representation matrix are clustered to obtain multiple cluster centers, and the distance between the cluster centers and each sample in the common latent representation matrix is ​​calculated to construct a distance matrix, wherein the sample is a column in the common latent representation matrix; a pseudo-label value of each sample is calculated based on the distance matrix, and a pseudo-label matrix is ​​constructed based on the pseudo-label value of each sample; the pseudo-label matrix is ​​decomposed into pseudo-label latent features The product of the matrix and the sample latent feature matrix is ​​optimized by alternating iteration to obtain the target sample latent feature matrix; according to each modal feature matrix in the multimodal feature matrix, the consensus representation matrix is ​​initialized, and according to the target sample latent feature matrix, the consensus representation matrix and each modal feature matrix, the projection matrix of each modal feature matrix and the target sample latent feature matrix is ​​determined; the consensus representation matrix is ​​binarized into a hash code matrix, and the target loss function is constructed according to the target sample latent feature matrix, each modal feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix and the pseudo label matrix; according to the target loss function, the target hash code matrix and the target projection matrix are determined, and the target modal data is cross-modal hash retrieval is performed based on the target hash code matrix and the target projection matrix. In this way, a pseudo-label matrix is ​​constructed by calculating the pseudo-label value of each sample. Later, a target loss function is constructed based on the pseudo-label matrix to obtain the optimized hash code matrix (i.e., target hash code matrix) and the optimized projection matrix (i.e., target projection matrix), thereby realizing cross-modal hash retrieval of target modal data. That is, by generating a pseudo-label matrix and making full use of pseudo-label information to deeply explore the semantic consistency between modal data, the problem of "supervision-semantic double gap" that exists in traditional multimodal hashing when using supervised information is solved, thereby improving the accuracy of cross-modal retrieval.

[0079] In some implementations, calculating a pseudo label value for each sample based on the distance matrix includes:

[0080] ;

[0081] in, represents the pseudo label value of each sample, Represents the distance matrix Middle Rank Elements of the column, Represents the distance matrix Middle Rank Elements of the column, represents the total number of cluster centers, represents a hyperparameter controlling the rate of distance decay.

[0082] In some embodiments, the target sample latent feature matrix is ​​obtained by alternately iteratively optimizing the pseudo-label latent feature matrix and the sample latent feature matrix, including:

[0083] Obtain the score of each sample, and construct an iterative optimization loss function based on the score of each sample, the pseudo-label latent feature matrix, and the sample latent feature matrix;

[0084] Fix the pseudo-label latent feature matrix and optimize the sample latent feature matrix;

[0085] Fix the sample latent feature matrix and optimize the pseudo-label latent feature matrix;

[0086] Until the alternating iterative optimization is performed and the iterative optimization loss function converges, the implicit feature matrix of the target sample is obtained.

[0087] In this embodiment, by obtaining the score of each sample, an iterative optimization loss function is constructed based on the score of each sample, the pseudo-label latent feature matrix, and the sample latent feature matrix; the pseudo-label latent feature matrix is ​​fixed and the sample latent feature matrix is ​​optimized; the sample latent feature matrix is ​​fixed and the pseudo-label latent feature matrix is ​​optimized; until the iterative optimization loss function converges, the target sample latent feature matrix is ​​obtained. In this way, by alternately iteratively optimizing the target sample latent feature matrix, a relatively accurate target sample latent feature matrix can be obtained, laying a good data foundation for the later guidance of consensus representation learning and hash code matrix generation.

[0088] In some embodiments, determining the projection matrices of each modality feature matrix and the target sample latent feature matrix based on the target sample latent feature matrix, the consensus representation matrix, and each modality feature matrix includes:

[0089] According to the consensus representation matrix and each modal feature matrix, the projection matrix of each modal feature matrix is ​​determined:

[0090] ;

[0091] According to the target sample latent feature matrix and the consensus representation matrix, the projection matrix of the target sample latent feature matrix is ​​determined:

[0092] ;

[0093] in, Indicates the The projection matrix of the modal feature matrix, and represents the hyperparameter, represents the consensus representation matrix, Indicates the modal characteristic matrix, represents transpose, represents the identity matrix, Represents the projection matrix of the target sample implicit feature matrix, Represents the target sample implicit feature matrix.

[0094] In some embodiments, a target loss function is constructed based on the target sample implicit feature matrix, each modality feature matrix, consensus representation matrix, projection matrix, hash code matrix, and pseudo label matrix, including:

[0095] ;

[0096] ;

[0097] in, Indicates the The projection matrix of the modal feature matrix, Represents the projection matrix of the target sample implicit feature matrix, represents the consensus representation matrix, represents the hash code matrix, Indicates the number of modes, Represents the contribution used to balance each feature matrix in constructing the consensus representation matrix, Indicates the modal characteristic matrix, represents the F2 norm, represents the target sample implicit feature matrix, represents the pseudo label matrix, The number of bits representing the hash code, represents the number of samples, Represents the consensus representation matrix and Weiquan The product of vectors, express dimension vector, represents the identity matrix, 、 、 and represents a hyperparameter.

[0098] In this embodiment, a target loss function is constructed based on the target sample implicit feature matrix, each modal feature matrix, consensus representation matrix, projection matrix, hash code matrix and pseudo label matrix. In this way, by integrating multiple parameters to construct the target loss function and subsequently optimizing the target loss function, more accurate results can be obtained.

[0099] In some embodiments, determining a target hash code matrix and a target projection matrix according to a target loss function includes:

[0100] When optimizing the projection matrix, other parameters in the target loss function are fixed, and the functional formula related to the projection matrix is ​​extracted from the target loss function to construct the first optimization function;

[0101] When optimizing the consensus representation matrix, other parameters in the target loss function are fixed, and a functional formula related to the consensus representation matrix is ​​extracted from the target loss function to construct a second optimization function;

[0102] When optimizing the hash code matrix, other parameters in the target loss function are fixed, and a functional formula related to the hash code matrix is ​​extracted from the target loss function to construct a third optimization function;

[0103] By minimizing the first optimization function, the second optimization function, and the third optimization function, the target loss function is minimized, and the target hash code matrix and the target projection matrix are determined.

[0104] In this embodiment, the target loss function is minimized by minimizing the first optimization function, the second optimization function, and the third optimization function, thereby determining the target hash code matrix and the target projection matrix. In this way, the obtained target hash code matrix and the target projection matrix are more accurate, thereby laying a good data foundation for later cross-modal data retrieval. In addition, this embodiment does not use label information. Instead, it deeply mines the semantic consistency between the modal data by generating a pseudo-label matrix and making full use of the pseudo-label information, solving the "supervision-semantics double gap" problem that exists in traditional multimodal hashing when using supervised information. This embodiment is suitable for efficient retrieval of large-scale multimodal data.

[0105] In some implementations, performing cross-modal hash retrieval on target modality data based on a target hash code matrix and a target projection matrix includes:

[0106] Obtain the target projection matrix corresponding to the target modality data;

[0107] Use the target hash code matrix as the hash code database;

[0108] Extracting target feature vectors of target modal data;

[0109] Calculating a hash code of the target modal data according to a target projection matrix and a target eigenvector corresponding to the target modal data;

[0110] The Hamming distance between the hash code of the target modality data and the hash codes of all data in the hash code database is calculated, and cross-modal hash retrieval of the target modality data is performed based on the Hamming distance.

[0111] In this embodiment, cross-modal hash retrieval is performed on target modality data by obtaining an accurate target hash code matrix and a target projection matrix, thereby improving the accuracy of the cross-modal retrieval.

[0112] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0113] With the deep integration of the internet, multimedia technologies, and artificial intelligence, data is undergoing a paradigm shift from a single modality to a multimodal collaborative representation of images, text, audio, and video. This exponential growth of multimodal data presents a dual challenge for cross-modal retrieval systems: addressing the semantic gap between heterogeneous modalities, such as text retrieval for images and video understanding for audio, while also meeting the engineering requirements of real-time retrieval of terabytes of data.

[0114] Because hashing technology offers significant advantages in fast querying and efficient storage of large amounts of multimodal data, it provides a viable solution for large-scale cross-modal retrieval applications. The core of cross-modal hashing lies in constructing a unified Hamming space mapping model, using binary hash coding to achieve comparability metrics for data from different modalities. Numerous cross-modal hashing methods have been proposed, each focusing on projecting cross-modal data into the Hamming space of binary hash codes while preserving inherent semantic similarity.

[0115] While deep learning-based cross-modal hashing methods have achieved breakthrough performance in recent years, their enormous time and computing power consumption have also led to severe bottlenecks in engineering implementation. This computational intensiveness leads to two major challenges in industrial-scale and terabyte-scale data scenarios: 1) the exponential growth of gradient synchronization overhead during distributed training; and 2) the difficulty in achieving real-time response in edge computing scenarios. Consequently, this method is limited in applications involving extremely large datasets and tasks with high real-time requirements.

[0116] Traditional multimodal hashing methods achieve cross-modal retrieval by aligning feature spaces or constructing shared semantic representations in low-dimensional spaces. While these methods offer significant advantages in training efficiency and retrieval speed, they rely heavily on supervised information due to their handcrafted feature modeling. However, these methods face a "supervision-semantics double gap" in their use of supervised information. On the one hand, they rely heavily on labeled data, significantly reducing retrieval accuracy as the missing label rate increases. On the other hand, they fail to fully exploit the hierarchical structure and semantic representation of semantic labels, resulting in insufficient utilization of the effective semantic information in the labels, which in turn leads to reduced retrieval accuracy.

[0117] This embodiment relates to an unsupervised cross-modal hash learning method based on pseudo-label generation of sample implicit features, which is applied to the field of multimodal data retrieval. It aims to solve the problem of insufficient retrieval accuracy in multimodal data retrieval due to lack of label information or insufficient use of label information. This method is suitable for paired multimodal data retrieval without label information. Figure 2 The technical solution of this embodiment specifically includes the following steps:

[0118] Step S1. Extract the multimodal feature matrix. The raw data of different modalities have significant differences in structure and form. Feature extraction can convert this heterogeneous data into vectors that are easy to process uniformly, eliminating formal differences and providing a basis for subsequent cross-modal alignment.

[0119] Specifically, in multimodal retrieval tasks, the original data forms of different modalities are very different (such as text is discrete symbols, images are pixel matrices), and direct comparison or fusion is almost impossible. In order to solve the heterogeneity problem of data in different modalities, it is necessary to first extract the multimodal feature matrix so that it can be effectively compared and calculated in a unified space. For data of different modalities, it is necessary to select an appropriate feature extraction model based on the modal characteristics and task requirements. The multimodal hash retrieval method of this embodiment has greater advantages in training efficiency and retrieval speed than the traditional multimodal hash retrieval method. Therefore, the traditional feature extraction method with higher computational efficiency is preferred. Next, take the bag-of-words model to extract text features and the convolutional neural network to extract image features as an example:

[0120] 1. Bag-of-words model extracts text features.

[0121] The bag-of-words model quickly captures global keyword information in a text by counting word frequencies, and is suitable for efficiently processing high-dimensional sparse text data. Specifically:

[0122] A. Segment the original text, remove stop words, and convert it into standardized word units;

[0123] B. Extract unique words from all documents, build a vocabulary, and assign a unique index to each word;

[0124] C. For each document, mark whether the word appears in the document (1 or 0), thereby mapping it into a vector with a dimension equal to the length of the vocabulary.

[0125] 2. Convolutional neural network extracts image features.

[0126] Convolutional neural networks automatically learn local features through convolutional layers and abstract high-level semantic information layer by layer. Specifically:

[0127] A. Use methods such as size normalization and channel normalization to convert the original image into a standardized format suitable for model processing;

[0128] B. Use convolution kernels to extract local spatial features, and then introduce nonlinearity through nonlinear activation functions to enhance the model's expressiveness;

[0129] C. Through the pooling layer, the spatial dimension of the feature map is reduced to reduce the amount of computation;

[0130] D. Repeatedly stack the "convolution-activation-pooling" module to extract more complex features layer by layer.

[0131] Step S2: Learn the common latent representation. Apply canonical correlation analysis to the multimodal feature matrix to find the weight matrix and canonical variable matrix for each modality, so as to maximize the correlation between the canonical variables after the transformation of various modalities. Then, the common latent representation is obtained through the canonical variable matrix.

[0132] Specifically, canonical correlation analysis is applied to the multimodal feature matrix to find the weight matrix of each modality and the canonical variate matrix , so that the correlation between the typical variables after various modal transformations is maximized. If there are only two modal data, canonical correlation analysis can be directly applied; for more than two modal data, multiple groups of data are sequentially subjected to pairwise canonical correlation analysis, and the weight matrix is ​​gradually optimized. Taking two modes as an example, the canonical variable matrices obtained from their characteristic matrices are respectively recorded as and , only retain the dimensional features with cumulative correlation coefficients exceeding 10%, still recorded as and , the asymmetric fusion of the two is the common potential representation matrix :

[0133] ;

[0134] ;

[0135] in, Represents the weight coefficient.

[0136] Step S3. Construct a pseudo-label matrix. Apply a clustering method to the features in the common latent representation matrix to obtain several cluster centers. Treat each cluster center as a pseudo-label, and treat the similarity between each sample in the common latent representation matrix and the cluster center as the value of the sample under this pseudo-label. Calculate the similarity between all pairs of column vectors in the cluster center matrix and the common latent representation matrix to obtain the pseudo-label matrix.

[0137] Specifically, for the common latent representation matrix The features in the clustering method are clustered, such as the k-means clustering method. The clustering method in this embodiment can adopt a clustering method known to those skilled in the art, and is not specifically limited in this embodiment. Center points (cluster centers), denoted as , each cluster center is regarded as a pseudo label. Calculate the Cluster centers and No. samples (i.e., common latent representation matrix The The square of the distance between columns:

[0138] ;

[0139] The real label value is often that each sample is only labeled under a few labels, and unlabeled under other labels. In order to make the pseudo-label closer to the nature of the real label, and also to enhance the guidance role of the more neighboring pseudo-label, the distance matrix (Distance Matrix Contains multiple elements) are sparse, that is, only the The elements with the smallest value are set to Finally, according to the distance matrix Calculate the pseudo label matrix (Distance Matrix Contains multiple elements), and in order to balance the impact of pseudo labels on each sample, the pseudo label value of each sample is normalized:

[0140] ;

[0141] in, and The distance matrix No. Rank Elements of the column and Rank Elements of the column, is a hyperparameter used to control the rate of distance decay.

[0142] Step S4. Obtaining sample latent features. The pseudo-label matrix can be viewed as the product of the pseudo-label factor matrix and the sample factor matrix. Alternating least squares is used to alternately optimize these two product matrices to obtain the optimal pseudo-label factor matrix and sample factor matrix. The sample factor matrix is ​​then viewed as the sample latent feature matrix for subsequent consensus representation learning.

[0143] Specifically, the pseudo-label matrix can be regarded as the product of the pseudo-label factor matrix (i.e., the pseudo-label implicit feature matrix) and the sample factor matrix (i.e., the sample implicit feature matrix), that is:

[0144] ;

[0145] in, 、 、 ( ), represents the number of pseudo labels (i.e. the number of cluster centers), represents the number of samples of a certain mode, Denotes the latent factor dimension. Using the alternating least squares method, alternately fixing one matrix and optimizing the other matrix to decompose the problem:

[0146] Fixed pseudo-label factor matrix , optimize the sample factor matrix :

[0147] For each sample , only consider the pseudo-label set that has a score (pseudo-label value) for the sample .definition , for the sample Factor matrix of pseudo labels with scores; , that is, the sample The loss function is about The part is:

[0148] ;

[0149] The closed-form solution is:

[0150] ;

[0151] in, represents the hyperparameter that controls the strength of the regularization term, express dimensional identity matrix, represents transpose, Represents pseudo labels The factor matrix of the scored samples (i.e. ), Indicates the sample The factor matrix of the pseudo labels of the scores (i.e. submatrix of ).

[0152] Fixed sample factor matrix , optimize the pseudo-label factor matrix :

[0153] Similarly, we can get:

[0154] ;

[0155] The alternating least squares method is used to iterate until convergence.

[0156] The optimized sample factor matrix It is regarded as the sample implicit feature matrix and used in subsequent consensus representation learning.

[0157] Step S5. Learn the consensus representation. The consensus representation matrix and the corresponding projection matrix are learned using the feature matrix and the sample latent feature matrix. Specifically, the feature matrix and the sample latent feature matrix are multiplied by the corresponding projection matrix, and the result should be as close as possible to the consensus representation matrix. To balance the contribution of each feature matrix in constructing the consensus representation matrix, the loss function between each feature matrix and the consensus representation matrix is ​​multiplied by the corresponding hyperparameter.

[0158] Specifically, the consensus representation matrix and the corresponding projection matrix are learned using each modal feature matrix and the sample latent feature matrix. Specifically, the feature matrix and the sample latent feature matrix are multiplied by the corresponding projection matrix, and the result should be as close as possible to the consensus representation matrix. Among them, the feature matrix of one modality obtains an initialized consensus representation matrix, and then the projection matrix corresponding to the sample latent feature matrix is ​​calculated based on the sample latent feature matrix and the consensus representation matrix. The projection matrix corresponding to each modal feature matrix is ​​calculated based on the feature matrix of each modality and the consensus representation matrix, and then the projection matrix and the consensus representation matrix are gradually iteratively optimized in step S8.

[0159] Will Denoted as consensus representation matrix, Recorded as The projection matrix of the modal feature matrix, Denoted as the projection matrix of the sample implicit feature matrix, is the number of modes, and the optimization objective can be written as:

[0160] ;

[0161] ;

[0162] in, Indicates the modal characteristic matrix, Used to balance the contribution of each feature matrix in constructing the consensus representation matrix, and The regularization term is to limit the complexity of the projection matrix and improve the generalization of the model. Representation matrix and Weiquan The product of vectors, express dimension vector, represents the number of samples, The number of bits representing the hash code, Represents the identity matrix. According to information entropy theory, the ratio of each value in each bit of the hash code should be equal, and the values ​​between bits should be independent of each other, so as to prevent information redundancy. Directly restricting these two conditions to the hash code itself will make the optimization itself an NP-hard problem. Therefore, this embodiment introduces the hash code continuous value estimation method. (can also be viewed as a consensus representation), and restrict these two conditions to : .

[0163] Step S6. Learn the hash code. Intuitively, the hash code matrix is ​​the ultimate learning target for each sample feature matrix, while the consensus representation matrix is ​​a continuous-valued estimate of the hash code matrix. Therefore, the simplest approach is to directly binarize the consensus representation matrix to obtain the hash code matrix. However, considering that this will result in a certain loss of accuracy, the norm of the difference between the two is added to the overall loss function.

[0164] Specifically, from an intuitive sense, the hash code matrix is the final learning goal of each sample feature matrix, and the consensus representation matrix It is the continuous value estimate of the hash code matrix, so the simplest approach is to directly binarize the consensus representation matrix to obtain the hash code matrix, that is:

[0165] ;

[0166] in, Represents a sign function, which outputs a positive value when the input is positive. , otherwise output .

[0167] However, considering that this will cause a certain loss of accuracy, we instead optimize the norm of the difference between the two:

[0168] .

[0169] Step S7. Utilization of pseudo label information. In order to utilize the label information of the pseudo label matrix to guide the generation of the consensus representation matrix and the hash code matrix, the product of the consensus representation matrix and the hash code matrix should be equal to the inner product of the pseudo label matrix. (number of bits in the hash code) times as close as possible.

[0170] Specifically, in step S5, this embodiment uses the sample implicit feature matrix decomposed from the pseudo label matrix to guide the learning of consensus representation, which is one aspect of the use of pseudo label information. On the other hand, the label information of the pseudo label matrix is ​​directly used to guide the generation of the consensus representation matrix and the hash code matrix. ,in The number of bits representing the hash code, the optimization goal is written as:

[0171] ;

[0172] Intuitively, if The larger the The sample and The higher the similarity of samples, the higher the consensus representation matrix Column samples and hash code matrix The larger the inner product of the column samples should be.

[0173] Step S8. Optimization of the main loss function and each variable.

[0174] Combining steps S5 to S7, we can obtain the overall optimization goal, that is, the main loss function:

[0175] ;

[0176] ;

[0177] in, are all hyperparameters. From the perspective of optimizing each variable, optimizing and The method is the same as Recorded as , Record , the above formula can be simplified to:

[0178] ;

[0179] ;

[0180] in, Represents the characteristic matrix of each mode Used to learn the consensus representation matrix is a hyperparameter that balances the contribution of each mode.

[0181] optimization :

[0182] Fix other variables and only consider The relevant functional form, the optimization problem is transformed into:

[0183] ;

[0184] Let the first-order derivative be 0, and we get:

[0185] ;

[0186] The solution is:

[0187] ;

[0188] when hour, .

[0189] optimization :

[0190] Fix other variables and only consider The relevant functional form, the optimization problem is transformed into:

[0191] ;

[0192] ;

[0193] Converted to the following form:

[0194] ;

[0195] ;

[0196] This is an orthogonal Procrustes problem with a zero mean constraint. , , in order to obtain the optimal ,right Performing feature decomposition yields:

[0197] ;

[0198] in, is a diagonal matrix of positive eigenvalues, is a matrix rank. is the corresponding eigenvector, Contains the remaining eigenvectors corresponding to the eigenvalue 0, By Schmidt orthogonalization we get .remember , is a random orthogonal matrix. Finally, The optimal solution is:

[0199] ;

[0200] optimization :

[0201] Fix other variables and only consider The relevant functional form, the optimization problem is transformed into:

[0202] ;

[0203] ;

[0204] Converted to the following form:

[0205] ;

[0206] ;

[0207] Binarization can be solved:

[0208] ;

[0209] Step S9. Retrieval of new data.

[0210] In step S8, the hash code of the training data has been obtained (as a hash code database) and the projection matrix of each modality For new data that needs to be retrieved (i.e. target modality data), use the following steps to retrieve relevant data in the database:

[0211] Use the corresponding method provided in step S1 to extract the feature vector of the new data ;

[0212] If the new data belongs to modality, get the corresponding projection matrix ,use Get the hash code of the new data;

[0213] The Hamming distance calculation is performed on the hash code of the new data and the hash codes of all the data in the database. and , the Hamming distance calculation formula is:

[0214] ;

[0215] in, is the hash code length, is an indicator function, if Then , otherwise .

[0216] According to the Hamming distance of the new data, another modality data related to the Hamming distance of the new data is returned in order from small to large as the retrieval result.

[0217] Compared with the existing technology, the technical solution of this embodiment has the following advantages:

[0218] This embodiment is a type of unsupervised cross-modal hash learning that only uses the pairwise correspondence of multimodal data without using label information. By generating a pseudo-label matrix and making full use of pseudo-label information to deeply explore the semantic consistency between the data of each modality, it can improve the accuracy of cross-modal hash retrieval and solve the "supervision-semantics double gap" problem that exists in traditional multimodal hashing when using supervised information. In other words, it solves the problem that traditional methods are highly dependent on supervised information and do not fully explore supervised information. The technical solution of this embodiment is suitable for efficient retrieval of large-scale multimodal data.

[0219] Reference Figure 3 The embodiment of the present application further provides an unsupervised cross-modal hash retrieval system based on implicit features, which includes a feature matrix construction unit 100, a representation matrix construction unit 200, a distance matrix construction unit 300, a pseudo-label matrix construction unit 400, an alternating iterative optimization unit 500, a projection matrix determination unit 600, a loss function construction unit 700, and a cross-modal hash retrieval unit 800, wherein:

[0220] A feature matrix construction unit 100 is used to extract modal features from raw data of different modalities to construct a multimodal feature matrix. The raw data of different modalities include text data, image data, video data, and audio data.

[0221] The representation matrix construction unit 200 is used to perform pairwise canonical correlation analysis on each two modal feature matrices in the multimodal feature matrix to obtain typical variable matrices of multiple different modalities, and to construct a common potential representation matrix based on the typical variable matrices of the multiple different modalities;

[0222] A distance matrix construction unit 300 is used to cluster the features in the common latent representation matrix to obtain multiple cluster centers, and calculate the distance between the cluster centers and each sample in the common latent representation matrix to construct a distance matrix, where a sample is a column in the common latent representation matrix;

[0223] A pseudo label matrix construction unit 400 is used to calculate the pseudo label value of each sample according to the distance matrix, and to construct a pseudo label matrix according to the pseudo label value of each sample;

[0224] An alternating iterative optimization unit 500 is used to decompose the pseudo-label matrix into the product of the pseudo-label latent feature matrix and the sample latent feature matrix, and obtain the target sample latent feature matrix by alternating iteratively optimizing the pseudo-label latent feature matrix and the sample latent feature matrix;

[0225] A projection matrix determination unit 600 is configured to initialize a consensus representation matrix based on each modal feature matrix in the multimodal feature matrix, and determine a projection matrix for each modal feature matrix and the target sample latent feature matrix based on the target sample latent feature matrix, the consensus representation matrix, and each modal feature matrix;

[0226] A loss function construction unit 700 is used to binarize the consensus representation matrix into a hash code matrix and construct a target loss function based on the target sample implicit feature matrix, each modality feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix, and the pseudo label matrix;

[0227] The cross-modal hash retrieval unit 800 is used to determine a target hash code matrix and a target projection matrix according to a target loss function, and perform a cross-modal hash retrieval on the target modality data according to the target hash code matrix and the target projection matrix.

[0228] It should be noted that since the unsupervised cross-modal hash retrieval system based on implicit features in this embodiment and the above-mentioned unsupervised cross-modal hash retrieval method based on implicit features are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0229] Reference Figure 4 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0230] at least one memory;

[0231] at least one processor;

[0232] at least one program;

[0233] The program is stored in the memory, and the processor executes at least one program to implement the above-mentioned unsupervised cross-modal hash retrieval method based on implicit features in the present disclosure.

[0234] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0235] The electronic device according to the embodiment of the present application is described in detail below.

[0236] The processor 1600 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0237] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called by the processor 1600 to execute the unsupervised cross-modal hash retrieval method based on implicit features of the embodiments of the present disclosure.

[0238] Input / output interface 1800, used for information input and output;

[0239] Communication interface 1900, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0240] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0241] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0242] An embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned unsupervised cross-modal hash retrieval method based on implicit features.

[0243] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0244] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0245] Those skilled in the art will understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than shown in the drawings, or a combination of certain steps, or different steps.

[0246] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0247] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0248] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0249] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0250] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0251] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0252] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0253] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

[0254] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. An unsupervised cross-modal hash retrieval method based on implicit features, characterized in that: The method comprises: Extracting modal features from raw data of different modalities to construct a multimodal feature matrix, wherein the raw data of different modalities include text data, image data, video data, and audio data; Performing pairwise canonical correlation analysis on every two modal feature matrices in the multimodal feature matrix to obtain typical variable matrices of multiple different modalities, and constructing a common potential representation matrix based on the typical variable matrices of the multiple different modalities; Clustering the features in the common latent representation matrix to obtain a plurality of cluster centers, and calculating the distance between the cluster centers and each sample in the common latent representation matrix to construct a distance matrix, wherein the sample is a column in the common latent representation matrix; Calculating the pseudo label value of each sample according to the distance matrix, and constructing a pseudo label matrix according to the pseudo label value of each sample; Decomposing the pseudo label matrix into the product of a pseudo label latent feature matrix and a sample latent feature matrix, and optimizing the pseudo label latent feature matrix and the sample latent feature matrix through alternating iterations to obtain a target sample latent feature matrix; Initializing a consensus representation matrix according to each modal feature matrix in the multimodal feature matrix, and determining a projection matrix of each modal feature matrix and the target sample latent feature matrix according to the target sample latent feature matrix, the consensus representation matrix, and each modal feature matrix; Binarizing the consensus representation matrix into a hash code matrix, and constructing a target loss function based on the target sample implicit feature matrix, each modality feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix, and the pseudo label matrix; According to the target loss function, a target hash code matrix and a target projection matrix are determined, and cross-modal hash retrieval is performed on the target modality data according to the target hash code matrix and the target projection matrix.

2. The unsupervised cross-modal hash retrieval method based on implicit features according to claim 1 is characterized in that: Calculating the pseudo label value of each sample according to the distance matrix includes: ； in, represents the pseudo label value of each sample, Represents the distance matrix Middle Rank Elements of the column, Represents the distance matrix Middle Rank Elements of the column, represents the total number of cluster centers, represents a hyperparameter controlling the rate of distance decay.

3. The unsupervised cross-modal hash retrieval method based on implicit features according to claim 1 is characterized in that: The step of optimizing the pseudo-label latent feature matrix and the sample latent feature matrix through alternating iteration to obtain a target sample latent feature matrix includes: Obtaining a score for each sample, and constructing an iterative optimization loss function based on the score for each sample, the pseudo-label latent feature matrix, and the sample latent feature matrix; Fixing the pseudo-label latent feature matrix and optimizing the sample latent feature matrix; Fixing the sample latent feature matrix and optimizing the pseudo-label latent feature matrix; The alternating iterative optimization is performed until the iterative optimization loss function converges, and the implicit feature matrix of the target sample is obtained.

4. The unsupervised cross-modal hash retrieval method based on implicit features according to claim 1 is characterized in that: The step of determining the projection matrices of each modal feature matrix and the target sample implicit feature matrix according to the target sample implicit feature matrix, the consensus representation matrix, and each modal feature matrix includes: According to the consensus representation matrix and each modal feature matrix, a projection matrix of each modal feature matrix is ​​determined: ； According to the consensus representation matrix and the target sample implicit feature matrix, a projection matrix of the target sample implicit feature matrix is ​​determined: ； in, Indicates the The projection matrix of the modal feature matrix, and represents the hyperparameter, represents the consensus representation matrix, Indicates the modal characteristic matrix, represents transpose, represents the identity matrix, Represents the projection matrix of the target sample implicit feature matrix, Represents the target sample implicit feature matrix.

5. The unsupervised cross-modal hash retrieval method based on implicit features according to claim 1 is characterized in that: The constructing of a target loss function according to the target sample implicit feature matrix, each modality feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix, and the pseudo label matrix includes: ； ； in, Indicates the The projection matrix of the modal feature matrix, Represents the projection matrix of the target sample implicit feature matrix, represents the consensus representation matrix, represents the hash code matrix, Indicates the number of modes, Represents the contribution used to balance each feature matrix in constructing the consensus representation matrix, Indicates the modal characteristic matrix, represents the F2 norm, represents the target sample implicit feature matrix, represents the pseudo label matrix, The number of bits representing the hash code, represents the number of samples, Represents the consensus representation matrix and Weiquan The product of vectors, express dimension vector, represents the identity matrix, 、 、 and represents a hyperparameter.

6. The unsupervised cross-modal hash retrieval method based on implicit features according to claim 1 is characterized in that: Determining a target hash code matrix and a target projection matrix according to the target loss function includes: When optimizing the projection matrix, fixing other parameters in the target loss function, and extracting a functional formula related to the projection matrix from the target loss function to construct a first optimization function; When optimizing the consensus representation matrix, fixing other parameters in the target loss function, and extracting a functional formula related to the consensus representation matrix from the target loss function to construct a second optimization function; When optimizing the hash code matrix, fixing other parameters in the target loss function, and extracting a functional formula related to the hash code matrix from the target loss function to construct a third optimization function; The target loss function is minimized by minimizing the first optimization function, the second optimization function, and the third optimization function, and a target hash code matrix and a target projection matrix are determined.

7. The unsupervised cross-modal hash retrieval method based on implicit features according to claim 1 is characterized in that: The performing cross-modal hash retrieval on the target modality data according to the target hash code matrix and the target projection matrix includes: Obtaining a target projection matrix corresponding to the target modal data; Using the target hash code matrix as a hash code database; Extracting target feature vectors of target modal data; Calculating a hash code of the target modality data according to a target projection matrix and the target eigenvector corresponding to the target modality data; Calculate the Hamming distance between the hash code of the target modality data and the hash codes of all data in the hash code database, and perform cross-modality hash retrieval on the target modality data according to the Hamming distance.

8. An unsupervised cross-modal hash retrieval system based on implicit features, characterized by: The system comprises: A feature matrix construction unit is used to extract modal features from raw data of different modalities to construct a multimodal feature matrix, wherein the raw data of different modalities include text data, image data, video data, and audio data; a representation matrix construction unit, configured to perform pairwise canonical correlation analysis on each two modal feature matrices in the multimodal feature matrix to obtain typical variable matrices of multiple different modalities, and to construct a common potential representation matrix based on the typical variable matrices of the multiple different modalities; a distance matrix construction unit, configured to cluster the features in the common latent representation matrix to obtain a plurality of cluster centers, and calculate the distance between the cluster centers and each sample in the common latent representation matrix to construct a distance matrix, wherein the sample is a column in the common latent representation matrix; A pseudo label matrix construction unit, configured to calculate a pseudo label value for each sample according to the distance matrix, and to construct a pseudo label matrix according to the pseudo label value for each sample; an alternating iterative optimization unit, configured to decompose the pseudo-label matrix into a product of a pseudo-label latent feature matrix and a sample latent feature matrix, and obtain a target sample latent feature matrix by alternating iteratively optimizing the pseudo-label latent feature matrix and the sample latent feature matrix; a projection matrix determining unit, configured to initialize a consensus representation matrix according to each modal feature matrix in the multimodal feature matrix, and determine a projection matrix for each modal feature matrix and the target sample latent feature matrix according to each modal feature matrix, the target sample latent feature matrix, and the consensus representation matrix; a loss function construction unit, configured to binarize the consensus representation matrix into a hash code matrix, and construct a target loss function based on the target sample implicit feature matrix, each modality feature matrix, the consensus representation matrix, the projection matrix, the hash code matrix, and the pseudo label matrix; A cross-modal hash retrieval unit is used to determine a target hash code matrix and a target projection matrix according to the target loss function, and perform a cross-modal hash retrieval on the target modality data according to the target hash code matrix and the target projection matrix.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the unsupervised cross-modal hash retrieval method based on implicit features as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the unsupervised cross-modal hash retrieval method based on implicit features according to any one of claims 1 to 7.

Images