Unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering

Abstract

This application discloses an unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering. The method constructs an alignment loss function based on a similarity matrix, a first encoding hash code, and a second encoding hash code; obtains a first soft assignment corresponding to the first encoding hash code and a second soft assignment corresponding to the second encoding hash code through a pseudo-classifier, and constructs a cluster-level contrast loss function based on the first soft assignment and the second soft assignment; fuses the first encoding hash code and the second encoding hash code to obtain a fused encoding hash code; constructs a steady-state loss function and a quantization loss function based on the first encoding hash code, the second encoding hash code, and the fused encoding hash code; constructs a total loss function by combining the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function; and determines the trained unsupervised cross-modal hash retrieval model based on the convergence of the total loss function. This application can improve the accuracy of cross-modal retrieval.

Description

Technical Field

[0001] The present application relates to the field of cross-modal retrieval technology, and in particular to an unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering. Background Art

[0002] With the rapid growth of multimedia data, cross-modal retrieval technology has become a research hotspot in the field of information retrieval. This technology aims to achieve cross-modal information alignment and efficient retrieval by querying related content in one modality (such as text) from another modality (such as images). In recent years, hashing methods have been widely used in cross-modal retrieval tasks due to their high encoding efficiency, low storage cost, and fast retrieval speed. However, unsupervised cross-modal hashing methods still face significant challenges in feature alignment and semantic preservation due to the lack of explicit semantic annotation information.

[0003] To solve the problem of supervised dependency, unsupervised cross-modal hashing methods have emerged. These methods usually learn to share semantic space by constructing inter-modal similarity graphs or introducing pseudo-labeling mechanisms. However, current unsupervised methods face some core challenges: (1) The semantic gap between modalities is difficult to bridge. Images and texts have natural differences in perceptual structure and feature distribution. Traditional unsupervised methods lack effective alignment mechanisms and find it difficult to learn consistent cross-modal representations. (2) Graph structure construction is prone to noisy connections. When constructing sample similarity graphs under unsupervised conditions, they are often based on low-level features or initial similarity metrics, which may cause semantically unrelated samples to be misconnected, thereby affecting the accuracy of semantic structure modeling. (3) There is a lack of modeling capabilities for global semantic structures. Existing methods mostly focus on local sample relationships and find it difficult to fully express the global distribution and aggregation characteristics of data in the latent semantic space. In addition, some methods introduce clustering mechanisms to assist in modal representation learning. Although this alleviates the label missing problem to a certain extent, the clustering process often lacks cross-modal consistency modeling, resulting in the final generated hash code being unable to accurately match between different modalities.

[0004] In summary, the existing unsupervised cross-modal hash retrieval methods have relatively low accuracy for cross-modal retrieval. Summary of the Invention

[0005] This application aims to propose an unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering, which can improve the accuracy of cross-modal retrieval.

[0006] In a first aspect, an embodiment of the present application provides an unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering, the method comprising:

[0007] Acquiring multimodal data including image modal data and text modal data, and extracting features of first modal data and second modal data respectively to obtain first extracted features and second extracted features, wherein the first modal data and the second modal data are any modal data in the multimodal data, and the first modal data and the second modal data are different modal data;

[0008] Performing nonlinear transformation on the first extracted feature and the second extracted feature respectively to obtain a first encoded hash code and a second encoded hash code;

[0009] Calculating a similarity matrix based on the first extracted features and the second extracted features, and constructing an alignment loss function based on the similarity matrix, the first encoded hash code, and the second encoded hash code;

[0010] Obtaining, by a pseudo classifier, a first soft assignment corresponding to the first encoded hash code and a second soft assignment corresponding to the second encoded hash code, and constructing a cluster-level contrast loss function based on the first soft assignment and the second soft assignment;

[0011] Fusing the first encoded hash code and the second encoded hash code to obtain a fused encoded hash code;

[0012] Constructing a steady-state loss function according to the first encoding hash code, the second encoding hash code, and the fused encoding hash code;

[0013] constructing a quantization loss function according to the first encoded hash code and the second encoded hash code;

[0014] Constructing a total loss function by combining the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function;

[0015] A trained unsupervised cross-modal hash retrieval model is determined according to the convergence of the total loss function, so as to perform cross-modal hash retrieval according to the trained unsupervised cross-modal hash retrieval model.

[0016] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0017] This method obtains multimodal data including image modal data and text modal data, and extracts features of the first modal data and the second modal data respectively to obtain first extracted features and second extracted features, wherein the first modal data and the second modal data are any modal data in the multimodal data, and the first modal data and the second modal data are different modal data. The first extracted features and the second extracted features are subjected to nonlinear transformation respectively to obtain first encoded hash codes and second encoded hash codes. The nonlinear transformation can map the first extracted features and the second extracted features to a unified hash space, thereby achieving alignment of the first modal data and the second modal data in the hash space. Then, a similarity matrix is calculated based on the first extracted features and the second extracted features, and an alignment loss function is constructed based on the similarity matrix, the first encoded hash code and the second encoded hash code, which can ensure that the features of the first modal data and the second modal data can be effectively matched in a common semantic space. Then, a pseudo classifier is used to obtain a first soft assignment corresponding to the first encoded hash code and a second soft assignment corresponding to the second encoded hash code, and a cluster-level contrast loss function is constructed based on the first soft assignment and the second soft assignment, which can enhance cross-modal semantic consistency. The first encoded hash code and the second encoded hash code are fused to obtain a fused encoded hash code. A steady-state loss function is constructed based on the first, second, and fused encoded hash codes. A quantization loss function is also constructed based on the first and second encoded hash codes. This allows for a more robust feature representation from a global optimization perspective, with strong stability and insensitivity to outliers, thereby improving the reliability of the hash representation. Finally, a total loss function is constructed by combining the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function. The trained unsupervised cross-modal hash retrieval model is determined based on the convergence of the total loss function, so that cross-modal hash retrieval can be performed based on the trained unsupervised cross-modal hash retrieval model. By comprehensively considering the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function, the total loss function is constructed, which can improve the accuracy of cross-modal retrieval based on the trained unsupervised cross-modal hash retrieval model.

[0018] In some embodiments, the step of subjecting the first extracted feature and the second extracted feature to nonlinear transformation to obtain a first encoded hash code and a second encoded hash code includes:

[0019] ;

[0020] ;

[0021] in, represents the first encoded hash code, represents the hyperbolic tangent function, Represents the scaling factor that is adaptively adjusted with the number of training rounds, and represents the weight matrix of the mapping layer, represents the regularization layer, represents a nonlinear activation function, and represents the weight matrix of the fully connected layer, represents the first extracted feature, and represents the bias matrix of the fully connected layer, and represents the bias matrix of the mapping layer, represents the second encoded hash code, represents the second extracted feature.

[0022] In some embodiments, calculating a similarity matrix based on the first extracted features and the second extracted features, and constructing an alignment loss function according to the similarity matrix, the first encoded hash code, and the second encoded hash code, includes:

[0023] Calculating a first similarity matrix based on the first extracted features;

[0024] Calculating a second similarity matrix based on the second extracted features;

[0025] fusing the first similarity matrix and the second similarity matrix to obtain a fused similarity matrix;

[0026] Calculating a first loss based on the fused similarity matrix and the first encoded hash code;

[0027] Calculating a second loss based on the fused similarity matrix and the second encoded hash code;

[0028] calculating a third loss based on the fused similarity matrix, the first encoded hash code, and the second encoded hash code;

[0029] An alignment loss function is constructed by combining the first loss, the second loss, and the third loss.

[0030] In some embodiments, constructing a cluster-level contrast loss function based on the first soft assignment and the second soft assignment includes:

[0031] ;

[0032] ;

[0033] ;

[0034] in, Indicates the first modal data corresponding to the The contrastive loss of the cluster assignment statistics vector, Indicates the second modal data corresponding to the The contrastive loss of the cluster assignment statistics vector, represents the cluster-level contrast loss function, Indicates the first modal data corresponding to the The first soft assignment of clusters, Indicates the second modal data corresponding to the The second soft assignment of clusters, represents the cluster-level temperature parameter, represents the number of clusters, Indicates the calculation of cosine similarity function, Indicates the first modal data corresponding to the The first soft assignment of clusters, Indicates the second modal data corresponding to the The second soft assignment of clusters, represents the information entropy corresponding to the first modal data, represents the information entropy corresponding to the second modal data, Represents the exponential function.

[0035] In some embodiments, constructing a steady-state loss function based on the first encoded hash code, the second encoded hash code, and the fused encoded hash code includes:

[0036] Normalizing the first encoded hash code to obtain a first state vector;

[0037] Normalizing the second encoded hash code to obtain a second state vector;

[0038] Normalizing the fused encoding hash code to obtain a third state vector;

[0039] Calculating cosine similarity based on the fused encoding hash code, and constructing state transition probability based on the cosine similarity;

[0040] Iterate the third state vector multiple times using the state transition probability to determine a steady-state distribution of the state vector;

[0041] A steady-state loss function is constructed according to the steady-state distribution of the state vector, the first state vector, the second state vector, and the third state vector.

[0042] In some embodiments, constructing a steady-state loss function based on the steady-state distribution of the state vector, the first state vector, the second state vector, and the third state vector includes:

[0043] ;

[0044] in, represents the steady-state loss function, Indicates the number of samples corresponding to each modality, represents the third state vector, represents the steady-state distribution of the state vector, represents the first state vector, represents the second state vector, Represents the logarithmic function.

[0045] In some embodiments, constructing a quantization loss function based on the first encoded hash code and the second encoded hash code includes:

[0046] ;

[0047] in, represents the quantization loss function, represents the first encoded hash code, represents the symbolic function, Represents the second encoded hash code.

[0048] In some implementations, fusing the first encoded hash code and the second encoded hash code to obtain a fused encoded hash code includes:

[0049] Concatenate the first encoded hash code and the second encoded hash code to obtain a concatenated encoded hash code;

[0050] The concatenated encoding hash codes are nonlinearly fused to obtain a fused encoding hash code.

[0051] In a second aspect, an embodiment of the present application further provides an unsupervised cross-modal hash retrieval system based on steady-state distribution and clustering, the system comprising:

[0052] a data acquisition unit, configured to acquire multimodal data including image modal data and text modal data, and extract features of first modal data and second modal data respectively to obtain first extracted features and second extracted features, wherein the first modal data and the second modal data are any modal data of the multimodal data, and the first modal data and the second modal data are different modal data;

[0053] a nonlinear transformation unit, configured to perform nonlinear transformation on the first extracted feature and the second extracted feature respectively to obtain a first encoded hash code and a second encoded hash code;

[0054] a first construction unit, configured to calculate a similarity matrix based on the first extracted features and the second extracted features, and construct an alignment loss function according to the similarity matrix, the first encoded hash code, and the second encoded hash code;

[0055] a second construction unit, configured to obtain, through a pseudo classifier, a first soft assignment corresponding to the first encoded hash code and a second soft assignment corresponding to the second encoded hash code, and construct a cluster-level contrast loss function according to the first soft assignment and the second soft assignment;

[0056] a data fusion unit, configured to fuse the first encoding hash code and the second encoding hash code to obtain a fused encoding hash code;

[0057] A third construction unit is configured to construct a steady-state loss function according to the first encoding hash code, the second encoding hash code, and the fused encoding hash code;

[0058] a fourth constructing unit, configured to construct a quantization loss function according to the first encoded hash code and the second encoded hash code;

[0059] a fifth construction unit, configured to construct a total loss function by combining the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function;

[0060] The cross-modal retrieval unit is used to determine a trained unsupervised cross-modal hash retrieval model according to the convergence of the total loss function, so as to perform cross-modal hash retrieval according to the trained unsupervised cross-modal hash retrieval model.

[0061] In a third aspect, an embodiment of the present application also provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering.

[0062] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering.

[0063] It can be understood that the beneficial effects of the above-mentioned second to fourth aspects compared with the relevant technologies are the same as the beneficial effects of the above-mentioned first aspect compared with the relevant technologies. Please refer to the relevant description in the above-mentioned first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0065] Figure 1 This is a flow chart of an embodiment of an unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering provided by the present application;

[0066] Figure 2 This is a schematic diagram of the overall architecture of the best embodiment of the unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering provided by this application;

[0067] Figure 3 1 is a schematic structural diagram of an embodiment of an unsupervised cross-modal hash retrieval system based on steady-state distribution and clustering provided by the present application;

[0068] Figure 4 It is a structural diagram of an embodiment of the electronic device provided by this application. DETAILED DESCRIPTION

[0069] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0070] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0071] In the description of this application, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0072] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technical personnel in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0073] First, let’s analyze some of the terms used in this application:

[0074] Deep semantic hash coding: uses deep neural networks to extract high-level semantic features of data and compresses them into discriminative binary hash codes for efficient retrieval.

[0075] Semantic alignment: refers to the semantic correspondence between different modalities (such as images and text) in the feature space, so that similar content has similar representations.

[0076] Pseudo classifier: simulates the real classifier in unsupervised learning to generate pseudo labels or similarity measures to guide the model to learn discriminative features.

[0077] Soft assignment: When a sample is assigned to multiple categories or cluster centers, the assignment weight is a continuous value rather than a hard label of only one category.

[0078] Cluster assignment statistic vector (ASV): represents the probability statistics of the sample distribution on the cluster center, which is used to characterize its positional relationship relative to the overall data distribution.

[0079] Steady-state distribution: refers to the state distribution that eventually tends to be stable during the iteration of the Markov chain, reflecting the average state probability after the system has run for a long time.

[0080] Transition probability matrix: describes the probability of a Markov process transitioning from one state to another and is used to model the evolutionary relationship between states.

[0081] To solve the problem of supervised dependency, unsupervised cross-modal hashing methods have emerged. These methods usually learn to share semantic space by constructing inter-modal similarity graphs or introducing pseudo-labeling mechanisms. However, current unsupervised methods face some core challenges: (1) The semantic gap between modalities is difficult to bridge. Images and texts have natural differences in perceptual structure and feature distribution. Traditional unsupervised methods lack effective alignment mechanisms and find it difficult to learn consistent cross-modal representations. (2) Graph structure construction is prone to noisy connections. When constructing sample similarity graphs under unsupervised conditions, they are often based on low-level features or initial similarity metrics, which may cause semantically unrelated samples to be misconnected, thereby affecting the accuracy of semantic structure modeling. (3) There is a lack of modeling capabilities for global semantic structures. Existing methods mostly focus on local sample relationships and find it difficult to fully express the global distribution and aggregation characteristics of data in the latent semantic space. In addition, some methods introduce clustering mechanisms to assist in modal representation learning. Although this alleviates the label missing problem to a certain extent, the clustering process often lacks cross-modal consistency modeling, resulting in the final generated hash code being unable to accurately match between different modalities.

[0082] In order to solve the problem of low accuracy of existing unsupervised cross-modal hash retrieval methods for cross-modal retrieval, this application proposes an unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering.

[0083] Reference Figure 1 , a flow chart of an unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering provided in an embodiment of the present application. The unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering is applied to an electronic device, which may be a server or a mobile terminal. Figure 1 As shown, the unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering may include the following steps:

[0084] Step S100: acquiring multimodal data including image modal data and text modal data, and extracting features of the first modal data and the second modal data, respectively, to obtain first extracted features and second extracted features, wherein the first modal data and the second modal data are any modal data in the multimodal data, and the first modal data and the second modal data are different modal data;

[0085] Step S200: The first extracted feature and the second extracted feature are subjected to nonlinear transformation respectively to obtain a first encoded hash code and a second encoded hash code;

[0086] Step S300: calculating a similarity matrix based on the first extracted features and the second extracted features, and constructing an alignment loss function according to the similarity matrix, the first encoded hash code, and the second encoded hash code;

[0087] Step S400: obtaining a first soft assignment corresponding to the first encoded hash code and a second soft assignment corresponding to the second encoded hash code through a pseudo classifier, and constructing a cluster-level contrast loss function based on the first soft assignment and the second soft assignment;

[0088] Step S500: Fusing the first encoded hash code and the second encoded hash code to obtain a fused encoded hash code;

[0089] Step S600: construct a steady-state loss function according to the first encoding hash code, the second encoding hash code, and the fused encoding hash code;

[0090] Step S700: construct a quantization loss function according to the first encoded hash code and the second encoded hash code;

[0091] Step S800: Construct a total loss function by combining the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function;

[0092] Step S900: Determine a trained unsupervised cross-modal hash retrieval model according to the convergence of the total loss function, so as to perform cross-modal hash retrieval according to the trained unsupervised cross-modal hash retrieval model.

[0093] In this embodiment, by obtaining multimodal data including image modal data and text modal data, and extracting features of the first modal data and the second modal data respectively, a first extracted feature and a second extracted feature are obtained, wherein the first modal data and the second modal data are any modal data in the multimodal data, and the first modal data and the second modal data are different modal data, the first extracted feature and the second extracted feature are respectively subjected to a nonlinear transformation to obtain a first encoded hash code and a second encoded hash code, and the first extracted feature and the second extracted feature can be mapped to a unified hash space by the nonlinear transformation, thereby achieving alignment of the first modal data and the second modal data in the hash space. Then, a similarity matrix is calculated based on the first extracted feature and the second extracted feature, and an alignment loss function is constructed based on the similarity matrix, the first encoded hash code and the second encoded hash code, which can ensure that the features of the first modal data and the second modal data can be effectively matched in a common semantic space. Then, a first soft assignment corresponding to the first encoded hash code and a second soft assignment corresponding to the second encoded hash code are obtained by a pseudo classifier, and a cluster-level contrast loss function is constructed based on the first soft assignment and the second soft assignment, which can enhance cross-modal semantic consistency. The first encoded hash code and the second encoded hash code are fused to obtain a fused encoded hash code. A steady-state loss function is constructed based on the first, second, and fused encoded hash codes. A quantization loss function is also constructed based on the first and second encoded hash codes. This allows for a more robust feature representation from a global optimization perspective, with strong stability and insensitivity to outliers, thereby improving the reliability of the hash representation. Finally, a total loss function is constructed by combining the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function. The trained unsupervised cross-modal hash retrieval model is determined based on the convergence of the total loss function, so that cross-modal hash retrieval can be performed based on the trained unsupervised cross-modal hash retrieval model. By comprehensively considering the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function, the total loss function is constructed, which can improve the accuracy of cross-modal retrieval based on the trained unsupervised cross-modal hash retrieval model.

[0094] The multimodal data may include image data, text data, audio data, and video data, and is not specifically limited in this embodiment. Images, text, audio, and video are data of different modalities, and images, text, audio, and video can be used in practical scenarios such as cross-modal search, content recommendation, and multimodal intelligent analysis. For example, cross-modal search can be used to find corresponding images based on known text information.

[0095] The above-mentioned obtaining of the first soft assignment corresponding to the first encoded hash code and the second soft assignment corresponding to the second encoded hash code by the pseudo classifier can be performed by clustering the first encoded hash code and the second encoded hash code by the pseudo classifier to obtain the soft assignment of the cluster (i.e., including the first soft assignment and the second soft assignment).

[0096] In some embodiments, the first extracted feature and the second extracted feature are subjected to nonlinear transformation respectively to obtain a first encoded hash code and a second encoded hash code, including:

[0097] ;

[0098] ;

[0099] in, represents the first encoded hash code, represents the hyperbolic tangent function, Represents the scaling factor that is adaptively adjusted with the number of training rounds, and represents the weight matrix of the mapping layer, represents the regularization layer, represents a nonlinear activation function, and represents the weight matrix of the fully connected layer, represents the first extracted feature, and represents the bias matrix of the fully connected layer, and represents the bias matrix of the mapping layer, represents the second encoded hash code, represents the second extracted feature.

[0100] In this embodiment, by subjecting the first extracted features and the second extracted features to nonlinear transformation respectively, the first extracted features and the second extracted features can be mapped to a unified hash space, thereby achieving alignment of the first modal data and the second modal data in the hash space.

[0101] The above formula is first processed by a nonlinear activation function, then passed through a regularization layer, and then the scaling factor is adjusted. Finally, it is passed through a hyperbolic tangent function to obtain the first encoded hash code and the second encoded hash code.

[0102] In some embodiments, calculating a similarity matrix based on the first extracted features and the second extracted features, and constructing an alignment loss function based on the similarity matrix, the first encoded hash code, and the second encoded hash code, includes:

[0103] Calculating a first similarity matrix based on the first extracted features;

[0104] Calculating a second similarity matrix based on the second extracted features;

[0105] Fusing the first similarity matrix and the second similarity matrix to obtain a fused similarity matrix;

[0106] Calculating a first loss based on the fused similarity matrix and the first encoded hash code;

[0107] Calculate a second loss based on the fused similarity matrix and the second encoded hash code;

[0108] calculating a third loss based on the fused similarity matrix, the first encoded hash code, and the second encoded hash code;

[0109] Combining the first loss, the second loss and the third loss, an alignment loss function is constructed.

[0110] In this embodiment, a first similarity matrix is calculated based on the first extracted features; a second similarity matrix is calculated based on the second extracted features; the first similarity matrix and the second similarity matrix are fused to obtain a fused similarity matrix; a first loss is calculated based on the fused similarity matrix and the first encoded hash code; a second loss is calculated based on the fused similarity matrix and the second encoded hash code; a third loss is calculated based on the fused similarity matrix, the first encoded hash code, and the second encoded hash code; and an alignment loss function is constructed by combining the first loss, the second loss, and the third loss. In this way, by using cosine similarity to measure the similarity between data of different modalities and fusing the cross-modal similarity matrices, it is possible to ensure that the first extracted features and the second extracted features can be effectively matched in a common semantic space.

[0111] In some embodiments, constructing a cluster-level contrast loss function based on the first soft assignment and the second soft assignment includes:

[0112] ;

[0113] ;

[0114] ;

[0115] in, Indicates the first modal data corresponding to the The contrastive loss of the cluster assignment statistics vector, Indicates the second modal data corresponding to the The contrastive loss of the cluster assignment statistics vector, represents the cluster-level contrast loss function, Indicates the first modal data corresponding to the The first soft assignment of clusters, Indicates the second modal data corresponding to the The second soft assignment of clusters, represents the cluster-level temperature parameter, represents the number of clusters, Indicates the calculation of cosine similarity function, Indicates the first modal data corresponding to the The first soft assignment of clusters, Indicates the second modal data corresponding to the The second soft assignment of clusters, represents the information entropy corresponding to the first modal data, represents the information entropy corresponding to the second modal data, Represents the exponential function.

[0116] above and The formula is based on the idea of contrastive learning to construct a cluster-level contrastive loss function. Traditional contrastive learning learns discriminative feature representations by narrowing the representation distance of positive sample pairs and pushing the representation distance of negative sample pairs further apart. The loss function is as follows:

[0117] ;

[0118] in, and are two augmented views of the same data sample (positive sample pair), is the representation of other samples (negative samples), Represents the temperature parameter.

[0119] Using this idea, the representation of the same cluster in different modes and As a positive sample pair, the representations of different clusters and as well as and Treating them as negative sample pairs, we construct a cluster-level contrastive loss function. By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, we achieve the goal of pulling samples from the same cluster together and pushing samples from different clusters away.

[0120] above In order to avoid all samples being assigned to the same cluster, the formula introduces information entropy and , by maximizing entropy, we force the samples to be evenly distributed to clusters and avoid degenerate solutions.

[0121] In some embodiments, constructing a steady-state loss function based on the first encoded hash code, the second encoded hash code, and the fused encoded hash code includes:

[0122] Normalizing the first encoded hash code to obtain a first state vector;

[0123] Normalizing the second encoded hash code to obtain a second state vector;

[0124] Normalize the fused encoding hash code to obtain the third state vector;

[0125] Calculate the cosine similarity based on the fusion encoding hash code, and construct the state transition probability based on the cosine similarity;

[0126] Iterate the third state vector multiple times through the state transition probability to determine the steady-state distribution of the state vector;

[0127] A steady-state loss function is constructed according to the steady-state distribution of the state vector, the first state vector, the second state vector, and the third state vector.

[0128] In this embodiment, the first state vector is obtained by normalizing the first encoded hash code; the second state vector is obtained by normalizing the second encoded hash code; the third state vector is obtained by normalizing the fused encoded hash code; cosine similarity is calculated based on the fused encoded hash code, and state transition probabilities are constructed based on the cosine similarity; the steady-state distribution of the state vector is determined by performing multiple iterations through the state transition probabilities; and a steady-state loss function is constructed based on the steady-state distribution of the state vector, the first state vector, the second state vector, and the third state vector. In this way, the steady-state distribution can be regarded as an ideal global feature distribution form in the context of deep learning, used to capture the overall structure of the data. By performing multiple iterations through the state transition probabilities to determine the steady-state distribution of the state vector, a more robust feature representation can be constructed from the perspective of global optimization, which can have the characteristics of strong stability and insensitivity to outliers, thereby improving the reliability of the hash representation.

[0129] In some embodiments, constructing a steady-state loss function based on the steady-state distribution of the state vector, the first state vector, the second state vector, and the third state vector includes:

[0130] ;

[0131] in, represents the steady-state loss function, Indicates the number of samples corresponding to each modality, represents the third state vector, represents the steady-state distribution of the state vector, represents the first state vector, represents the second state vector, Represents the logarithmic function.

[0132] above The derivation of the formula is as follows:

[0133] Traditional hash retrieval methods are usually based on local optimization. This embodiment optimizes from a global perspective and uses the core concept of Markov chain, steady-state distribution, to improve robustness to noise and outliers. Under the same transfer matrix, state changes tend to be stable. According to the Perron-Frobenius theorem, an irreducible and non-periodic Markov chain has a unique stationary distribution. , and for any initial distribution , the tth iteration gets , through iterative updates, it will converge to That is to say,

[0134] ;

[0135] ;

[0136] According to this principle, this embodiment first initializes the state vector , , then construct the transition probability matrix for:

[0137] ;

[0138] ;

[0139] in, It is the fusion feature No. OK, It is Features and The cosine similarity between features, It is Features to The transition probability of a feature, is the Euclidean norm. Since and , which proves that this Markov chain satisfies irreducibility and non-periodicity, so there must be a unique stationary distribution .

[0140] After reaching the steady state, the loss function is further designed to guide the model to learn feature representations close to the steady state structure. In order to constrain the consistency between the current model output distribution and the ideal steady state distribution, the steady state distribution loss function is introduced:

[0141] ;

[0142] in, represents the Kullback-Leibler divergence, which is used to measure the difference between distributions. It is specifically expanded as follows:

[0143] .

[0144] In some embodiments, constructing a quantization loss function based on the first encoded hash code and the second encoded hash code includes:

[0145] ;

[0146] in, represents the quantization loss function, represents the first encoded hash code, represents the symbolic function, Represents the second encoded hash code.

[0147] The above quantization loss function is based on minimizing the mean square error between the network output and its corresponding binary hash code, and making continuous features gradually close to The idea of space is constructed.

[0148] In some implementations, fusing the first encoded hash code and the second encoded hash code to obtain a fused encoded hash code includes:

[0149] Concatenate the first encoded hash code and the second encoded hash code to obtain a concatenated encoded hash code;

[0150] The concatenated encoding hash codes are nonlinearly fused to obtain a fused encoding hash code.

[0151] In this embodiment, by fusing the first encoded hash code and the second encoded hash code, abstract extraction of shared semantics between modalities can be achieved.

[0152] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0153] With the rapid growth of multimedia data, cross-modal retrieval technology has become a research hotspot in the field of information retrieval. To address the issue of supervised dependency, unsupervised cross-modal hashing methods have emerged. These methods typically learn a shared semantic space by constructing inter-modal similarity graphs or introducing pseudo-labeling mechanisms. However, current unsupervised methods face the following core challenges:

[0154] (1) The semantic gap between modalities is difficult to bridge. Images and texts have natural differences in perceptual structure and feature distribution. Traditional unsupervised methods lack an effective alignment mechanism and are difficult to learn consistent cross-modal representations.

[0155] (2) Graph structure construction is prone to noisy connections. When constructing sample similarity graphs under unsupervised conditions, they are often based on low-level features or initial similarity metrics, which may cause semantically irrelevant samples to be misconnected, thereby affecting the accuracy of semantic structure modeling.

[0156] (3) Lack of modeling capabilities for global semantic structures. Existing methods mostly focus on local sample relationships, making it difficult to fully express the global distribution and aggregation characteristics of data in the latent semantic space.

[0157] In addition, some methods introduce clustering mechanisms to assist in modal representation learning. Although this alleviates the label missing problem to a certain extent, the clustering process often lacks cross-modal consistency modeling, resulting in the final generated hash code being unable to accurately match between different modalities.

[0158] In summary, how to effectively model cross-modal global semantic relationships in an unsupervised environment and improve the clustering consistency and semantic alignment effects between different modalities has become a key issue that urgently needs to be broken through in current cross-modal hashing research.

[0159] In response to the problems of existing unsupervised cross-modal hashing methods such as weak semantic modeling capabilities, inaccurate modal alignment, and large quantization errors, this embodiment proposes an unsupervised cross-modal hash retrieval method based on steady-state optimization and semantic clustering. This method jointly optimizes the semantic structure perception and modal alignment process by constructing a steady-state probability transfer mechanism and a cross-modal comparative clustering framework, thereby obtaining consistent cross-modal hash codes without the need for label supervision. The method of this embodiment mainly includes the following contents:

[0160] (1) In view of the characteristic differences between image and text modalities, a convolutional neural network (VGG) and a bag-of-words (BoW) model are used to extract multi-level feature representations, respectively, to ensure the expressiveness between the modalities from the source. Subsequently, a deep nonlinear hash coding network with two branches for image and text is constructed. Through ReLU activation, Dropout regularization and Tanh transformation, the original modal features are mapped to a consistent hash space and a continuous hash code is output.

[0161] (2) To further enhance the semantic alignment between different modalities, this embodiment designs a cross-modal alignment loss. Through feature normalization and cosine similarity constraints, it ensures that the representation of image and text modalities in the embedding space has strong consistency. In addition, to solve the semantic modeling problem caused by missing labels in an unsupervised environment, a multimodal clustering mechanism based on pseudo classifiers is proposed. This mechanism uses soft assignment to generate cluster assignment statistical vectors (ASVs), and constructs cross-modal contrast loss and entropy regularization terms to generate cluster-level contrast loss to promote the alignment and distribution diversity of clustering results between modalities, thereby improving the model's ability to capture multimodal semantic structures.

[0162] (3) In terms of feature stability, this embodiment introduces the theory of steady-state distribution in Markov chains, performs probabilistic modeling on the fused features to generate a state vector, and constructs a transition probability matrix to guide the model output to approach a stable feature distribution state. By minimizing the Kullback-Leibler divergence between the current output distribution and the ideal steady-state distribution, the robustness and stability of the feature representation to noise, outliers, and complex semantics are effectively improved.

[0163] (4) In addition, considering that hash retrieval ultimately relies on a compact binary code representation, this embodiment introduces a quantization loss function to constrain continuous hash codes to their binary form. This loss term, together with the modality alignment loss, cluster-level contrast loss, and steady-state loss, constitutes the total loss function and is jointly optimized through an end-to-end training mechanism.

[0164] Ultimately, this embodiment enables efficient cross-modal hash retrieval under unsupervised conditions. Users can select either image or text as a query term, and through fast Hamming distance calculation, potentially relevant content in the other modality is matched and ranked. Thanks to this embodiment's comprehensive optimization of semantic modeling, structural stability, and binary quantification, the proposed method demonstrates superior performance and broad application potential in scenarios such as image-text matching, image description generation, and cross-modal recommendation.

[0165] To further illustrate the specific implementation of this embodiment, the following will explain in detail the implementation process and technical details of this embodiment through the fusion and hash retrieval tasks of two typical modalities, image and text. Although this embodiment mainly uses images and text as examples in the description, its implementation principle is applicable to the processing of other multimodal data and can be flexibly applied to retrieval tasks in different fields. Figure 2 , the specific implementation method includes the following implementation steps:

[0166] Step S1: multimodal feature extraction.

[0167] To extract rich feature information from multimodal data and capture the semantic differences and relevance between different modalities, we first construct a preliminary multimodal feature representation. For image data (i.e., first modality data), a deep learning model based on a convolutional neural network (VGG) is used. This model effectively extracts hierarchical features of images through multi-layer convolution operations and captures image information at different scales through pooling operations. The ReLU activation function is combined to further enhance the network's nonlinear mapping capabilities. On this basis, a fully connected layer outputs a feature vector of the image (i.e., the first extracted feature). , the feature vector represents the high-level semantic information of the image. For text data (i.e., the second modality data), the classic bag-of-words model (BoW) is used to model the text data. Through this model, the frequency of words in the text and their co-occurrence relationship can be effectively captured to form a sparse feature (i.e., the second extracted feature) representation. .

[0168] Step S2: Deep semantic hash coding of multimodal data.

[0169] In order to alleviate the problem of feature scale differences between different modalities, this embodiment designs a dual-branch nonlinear projection network to achieve cross-modal feature alignment and unified encoding. The image branch and text branch each consist of a fully connected network layer, a ReLU activation function, and a Dropout regularization layer. The network structure passes the extracted feature vectors layer by layer. and , ensuring that the features of each modality can be mapped to a unified hash space after nonlinear transformation. Subsequently, it is mapped to the target hash code length through linear transformation, and then a continuous hash code is output through the Tanh function. The final outputs of the network are:

[0170] ;

[0171] ;

[0172] in, , is the sample size, is the target hash code length; and is the weight matrix of the fully connected layer, and is the bias matrix of the fully connected layer; and is the weight matrix of the mapping layer, and is the bias matrix of the mapping layer; is a scaling factor that is adaptively adjusted with the number of training rounds.

[0173] This process not only aligns the image and text modalities in the hash space but also, through end-to-end training, facilitates the model's learning of the underlying semantic relationships between the different modalities during optimization. This design enables this embodiment to effectively process heterogeneous data and ensure the consistency of multimodal data within the same semantic space.

[0174] Step S3: cross-modal semantic alignment.

[0175] In multimodal hash retrieval, in order to ensure that the features of image and text modalities can be effectively matched in a common semantic space, this embodiment introduces a cross-modal semantic alignment mechanism. and text features , first normalize it so that it fluctuates within a uniform range. The normalized features are used to calculate the cross-modal similarity matrix, which uses cosine similarity to measure the similarity between different modal data, and the cross-modal similarity matrix is fused, where A trade-off parameter to adjust the importance of information from different modalities:

[0176] ;

[0177] ;

[0178] ;

[0179] For deep semantic hash codes (i.e. first encoding hash code) and deep semantic hash code (i.e., the second encoded hash code), normalized and then constructed as follows:

[0180] ;

[0181] ;

[0182] ;

[0183] ;

[0184] in, and To balance the parameters, represents the first similarity matrix, represents the second similarity matrix, represents the fusion similarity matrix, represents the first loss, represents the second loss, represents the third loss, represents the alignment loss function.

[0185] This mechanism optimizes the modality alignment effect by minimizing the distribution difference between the hash codes of the image modality and the text modality, effectively ensuring the representation consistency of different modality features in the hash space, and laying the foundation for the subsequent retrieval process.

[0186] Step S4: clustering optimization and cross-modal semantic consistency enhancement.

[0187] In order to further optimize cross-modal representation, this embodiment proposes a clustering method based on pseudo-classifiers, which enables different modalities to share clustering weights.

[0188] Specifically, instances can be passed through a pseudo-classifier to obtain a soft assignment to a cluster, i.e. ,in is the sample size, is the number of clusters, Indicates two modes: image and text. No. Ranked as:

[0189] ;

[0190] in, Collect and assign to The probability values of all instances in a cluster (i.e., soft assignment), for When corresponds to the first soft assignment, for , corresponds to the second soft assignment, which in this embodiment is called a cluster assignment statistical vector (ASV).

[0191] ASVs from different clusters should be mutually exclusive and ideally orthogonal, while ASVs from different modalities in the same cluster should be consistent to enhance cross-modal semantic consistency. To achieve this goal, this embodiment extends contrastive clustering to multimodal data, defines a cluster-level contrastive loss, and introduces entropy regularization to prevent all samples from being assigned to the same cluster, thereby improving retrieval robustness. The contrast loss of an ASV is calculated as follows:

[0192] ;

[0193] ;

[0194] in, is the cluster-level temperature parameter, is the number of clusters, Indicates the calculation of the cosine similarity function. Then for the entire data, the cluster-level contrast loss is expressed as:

[0195] ;

[0196] in, is information entropy, , by maximizing entropy to avoid all samples being assigned to the same cluster, where is the sample size, is the number of clusters.

[0197] Step S5: Steady-state distribution optimization and global characteristic stability.

[0198] In unsupervised cross-modal hash retrieval, traditional hashing methods mostly rely on local similarity or reconstruction error, often ignoring the structural information of the data at the overall distribution level, which causes the generated hash codes to be unstable when faced with noise, abnormal samples or complex semantic structures.

[0199] To address this issue, this embodiment introduces the concept of steady-state distribution in Markov chains, suppresses the influence of outliers through iterative diffusion, and constructs a more robust feature expression from a global optimization perspective. In the context of deep learning, steady-state distribution can be considered an ideal form of global feature distribution to capture the overall structure of the data. This embodiment generates a state vector from the fused feature matrix and iterates multiple times through the transition probability matrix until a steady state is reached, making it highly stable and insensitive to outliers, thereby improving the reliability of the hash representation.

[0200] Step S5.1: Cross-modal deep semantic hash code fusion.

[0201] In order to solve the problem of semantic fragmentation and insufficient fusion caused by independent modeling of hash representations of image modality and text modality in cross-modal retrieval tasks, this embodiment fuses cross-modal deep semantic hash codes for subsequent operations. Specifically, the deep semantic hash codes of the two modalities are first combined by feature splicing. and Connect on the channel dimension. In order to compress the concatenated vector back to a unified hash code dimension and abstractly extract the shared semantics between modalities, the module introduces a fully connected layer (i.e., linear transformation layer) to map the fused features to an output vector of the target hash code length and uses the hyperbolic tangent (tanh) activation function to map the vector to Continuous hash codes on (i.e. fused encoding hash code).

[0202] Step S5.2: Steady-state distribution constrained optimization.

[0203] This embodiment uses the fused hash code to perform subsequent steady-state distribution optimization. First, the fused feature ( is the number of samples, is the target hash code length) normalized to a probability distribution (i.e., the third state vector), which is called the state vector in this embodiment, is used to express the relative importance of the sample in the feature space and eliminate the scale effect of the original feature, where express No. OK:

[0204] ;

[0205] At the same time, independent distributions corresponding to image modality and text modality are constructed respectively To characterize the relative importance of each modal hash feature in the feature space, they are expressed as:

[0206] ;

[0207] ;

[0208] in, and Respectively and No. OK, represents the first state vector, represents the second state vector.

[0209] Next, construct the transition probability matrix between samples ,in is the number of samples. This matrix is based on the cosine similarity between features and is normalized to transition probability through the Softmax function to ensure that similar samples have a high state transition probability:

[0210] ;

[0211] ;

[0212] On this basis, through ,in For iteration The state vector after the iteration continuously updates the distribution state until it converges to the state that satisfies Steady-state distribution of This process can be viewed as an explicit global optimization process to capture the overall structure of the data. After reaching a steady state, this embodiment further designs a loss function to guide the model to learn feature representations close to the steady-state structure. In order to constrain the consistency between the current model output distribution and the ideal steady-state distribution, the steady-state loss function is introduced as follows:

[0213] ;

[0214] in, represents the Kullback-Leibler divergence, which is used to measure the difference between distributions. It is specifically expanded as follows:

[0215] ;

[0216] Through the above mechanism, this embodiment can explicitly model and optimize the global steady-state distribution of features, so that the fused hash representation is more stable and has global structure perception capabilities, thereby improving the accuracy and robustness of multimodal hash retrieval.

[0217] Step S6: Generate total loss space.

[0218] Although continuous hash representations can be learned through deep networks, hash retrieval essentially relies on binary hash codes. If the difference between the continuous output and the ideal binary code is not constrained, the model may learn floating-point features that deviate significantly from the binary code, thereby weakening the discriminative power and consistency of the retrieval stage. To address this problem, this paper introduces a quantization loss term to constrain the continuous hash representation. This loss term minimizes the mean square error between the network output and its corresponding binary hash code, so that the continuous features gradually approach the binary code. space. Formally, the quantization loss is defined as:

[0219] ;

[0220] On this basis, combined with the aforementioned modality alignment loss, cluster-level contrast loss, steady-state loss and other optimization objectives, we further construct a joint total loss function. and To adjust the parameters of the loss weight:

[0221] ;

[0222] Step S7: Repeat steps S2 to S6 until the model converges.

[0223] This embodiment jointly optimizes the multiple loss functions constructed above and continuously monitors the total loss during the training process. When the loss tends to be stable or its change amplitude is lower than the set threshold in multiple training rounds, the entire model is judged to have reached a convergence state and the training process is terminated.

[0224] Reference Figure 2 The entire model is an unsupervised cross-modal hash retrieval model, including convolutional neural network (VGG), bag-of-words model (BoW), two-branch nonlinear projection network, cross-modal semantic alignment mechanism, soft clustering layer and steady-state distribution module.

[0225] Furthermore, to prevent the model from falling into local optimality or overfitting, weight decay, dropout, and regularization are introduced during training, while periodic evaluations are performed on the quantization level and semantic preservation of the hash output. Through this iterative and steadily optimized training mechanism, the model effectively learns and aligns semantic relationships in the multimodal feature space, thereby consistently outputting highly discriminative and quantifiable hash representations.

[0226] Step S8: Efficient cross-modal retrieval and result ranking.

[0227] This embodiment, based on an optimized multimodal hash coding mechanism, maps the image and text modalities into a unified hash space, ensuring consistent semantic representation across heterogeneous modalities. Specifically, given a query term in any modality (e.g., an image), the hash code efficiently matches potentially semantically related terms in the other modality (e.g., text) using a fast Hamming distance calculation.

[0228] During the actual search process, the system first calculates the hash code of the query sample and then sorts the hash codes of all samples in the target database by Hamming distance. Because hash codes are compact binary values, this process greatly improves efficiency through bitwise operations, achieving sub-second response times. Furthermore, thanks to the semantic preservation, modality alignment, and steady-state optimization mechanisms introduced during early training, the sorting results are semantically more consistent and discriminative.

[0229] Therefore, this embodiment can achieve cross-modal retrieval with high accuracy, high response speed, and high robustness, and demonstrates significant performance advantages in practical applications such as multimodal image and text retrieval, image description matching, and visual question answering.

[0230] Compared with the existing technology, the technical solution of this embodiment has the following advantages:

[0231] This embodiment proposes an unsupervised cross-modal hash retrieval method that integrates semantic alignment, clustering optimization and steady-state modeling, which effectively alleviates the core bottlenecks of existing technologies in terms of modal heterogeneity, feature instability and hash code quantifiability. This method realizes continuous hash code generation through a dual-branch deep encoding network of images and texts, and combines cross-modal alignment loss with a multimodal clustering mechanism guided by pseudo-labels to capture shared semantic structures under unsupervised conditions. At the same time, the features are globally modeled with the help of Markov chain steady-state distribution theory to further improve the stability and robustness of the representation. Finally, by jointly optimizing multiple loss functions, a binary hash code with strong discriminability and excellent compressibility is effectively obtained, realizing efficient and accurate retrieval matching between image and text modalities. This embodiment has the characteristics of unified structure, efficient training and strong adaptability, and is widely applicable to practical scenarios such as cross-modal search, content recommendation and multimodal intelligent analysis.

[0232] Reference Figure 3 The embodiment of the present application further provides an unsupervised cross-modal hash retrieval system based on steady-state distribution and clustering, the system comprising a data acquisition unit 100, a nonlinear transformation unit 200, a first construction unit 300, a second construction unit 400, a data fusion unit 500, a third construction unit 600, a fourth construction unit 700, a fifth construction unit 800, and a cross-modal retrieval unit 900, wherein:

[0233] A data acquisition unit 100 is configured to acquire multimodal data including image modal data and text modal data, and extract features of the first modal data and the second modal data, respectively, to obtain first extracted features and second extracted features, wherein the first modal data and the second modal data are any modal data in the multimodal data, and the first modal data and the second modal data are different modal data;

[0234] The nonlinear transformation unit 200 is configured to perform nonlinear transformation on the first extracted feature and the second extracted feature to obtain a first encoded hash code and a second encoded hash code;

[0235] A first construction unit 300 is configured to calculate a similarity matrix based on the first extracted features and the second extracted features, and to construct an alignment loss function according to the similarity matrix, the first encoded hash code, and the second encoded hash code;

[0236] A second construction unit 400 is configured to obtain, through a pseudo classifier, a first soft assignment corresponding to the first encoded hash code and a second soft assignment corresponding to the second encoded hash code, and construct a cluster-level contrast loss function based on the first soft assignment and the second soft assignment;

[0237] The data fusion unit 500 is configured to fuse the first encoded hash code and the second encoded hash code to obtain a fused encoded hash code;

[0238] A third construction unit 600 is configured to construct a steady-state loss function according to the first encoding hash code, the second encoding hash code, and the fused encoding hash code;

[0239] A fourth constructing unit 700 is configured to construct a quantization loss function according to the first encoded hash code and the second encoded hash code;

[0240] A fifth construction unit 800 is configured to construct a total loss function by combining the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function;

[0241] The cross-modal retrieval unit 900 is configured to determine a trained unsupervised cross-modal hash retrieval model according to the convergence of the total loss function, so as to perform cross-modal hash retrieval based on the trained unsupervised cross-modal hash retrieval model.

[0242] It should be noted that since the unsupervised cross-modal hash retrieval system based on steady-state distribution and clustering in this embodiment and the above-mentioned unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the present system embodiment and will not be described in detail here.

[0243] Reference Figure 4 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0244] at least one memory;

[0245] at least one processor;

[0246] at least one program;

[0247] The program is stored in the memory, and the processor executes at least one program to implement the above-mentioned unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering implemented in the present disclosure.

[0248] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0249] The electronic device according to the embodiment of the present application is described in detail below.

[0250] The processor 1600 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0251] Memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 1700 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in memory 1700 and is called by processor 1600 to execute the unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering in the embodiments of the present disclosure.

[0252] Input / output interface 1800, used for information input and output;

[0253] Communication interface 1900, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0254] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0255] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0256] An embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering.

[0257] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0258] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0259] Those skilled in the art will understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than shown in the drawings, or a combination of certain steps, or different steps.

[0260] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0261] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0262] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0263] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0264] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0265] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0266] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0267] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

[0268] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. An unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering, characterized in that: The method comprises: Acquiring multimodal data including image modal data and text modal data, and extracting features of first modal data and second modal data respectively to obtain first extracted features and second extracted features, wherein the first modal data and the second modal data are any modal data in the multimodal data, and the first modal data and the second modal data are different modal data; Performing nonlinear transformation on the first extracted feature and the second extracted feature respectively to obtain a first encoded hash code and a second encoded hash code; Calculating a similarity matrix based on the first extracted features and the second extracted features, and constructing an alignment loss function based on the similarity matrix, the first encoded hash code, and the second encoded hash code; Obtaining, by a pseudo classifier, a first soft assignment corresponding to the first encoded hash code and a second soft assignment corresponding to the second encoded hash code, and constructing a cluster-level contrast loss function based on the first soft assignment and the second soft assignment; Fusing the first encoded hash code and the second encoded hash code to obtain a fused encoded hash code; Constructing a steady-state loss function according to the first encoding hash code, the second encoding hash code, and the fused encoding hash code; constructing a quantization loss function according to the first encoded hash code and the second encoded hash code; Constructing a total loss function by combining the alignment loss function, the cluster-level contrast loss function, the steady-state loss function, and the quantization loss function; A trained unsupervised cross-modal hash retrieval model is determined according to the convergence of the total loss function, so as to perform cross-modal hash retrieval according to the trained unsupervised cross-modal hash retrieval model.

2. The unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering according to claim 1 is characterized in that: The step of subjecting the first extracted feature and the second extracted feature to nonlinear transformation to obtain a first encoded hash code and a second encoded hash code includes: ； ； in, represents the first encoded hash code, represents the hyperbolic tangent function, Represents the scaling factor that is adaptively adjusted with the number of training rounds, and represents the weight matrix of the mapping layer, represents the regularization layer, represents a nonlinear activation function, and represents the weight matrix of the fully connected layer, represents the first extracted feature, and represents the bias matrix of the fully connected layer, and represents the bias matrix of the mapping layer, represents the second encoded hash code, represents the second extracted feature.

3. The unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering according to claim 1 is characterized in that: The calculating a similarity matrix based on the first extracted features and the second extracted features, and constructing an alignment loss function according to the similarity matrix, the first encoded hash code, and the second encoded hash code, includes: Calculating a first similarity matrix based on the first extracted features; Calculating a second similarity matrix based on the second extracted features; fusing the first similarity matrix and the second similarity matrix to obtain a fused similarity matrix; Calculating a first loss based on the fused similarity matrix and the first encoded hash code; Calculating a second loss based on the fused similarity matrix and the second encoded hash code; calculating a third loss based on the fused similarity matrix, the first encoded hash code, and the second encoded hash code; An alignment loss function is constructed by combining the first loss, the second loss, and the third loss.

4. The unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering according to claim 1 is characterized in that: The constructing a cluster-level contrast loss function according to the first soft assignment and the second soft assignment includes: ； ； ； in, Indicates the first modal data corresponding to the The contrastive loss of the cluster assignment statistics vector, Indicates the second modal data corresponding to the The contrastive loss of the cluster assignment statistics vector, represents the cluster-level contrast loss function, Indicates the first modal data corresponding to the The first soft assignment of clusters, Indicates the second modal data corresponding to the The second soft assignment of clusters, represents the cluster-level temperature parameter, represents the number of clusters, Indicates the calculation of cosine similarity function, Indicates the first modal data corresponding to the The first soft assignment of clusters, Indicates the second modal data corresponding to the The second soft assignment of clusters, represents the information entropy corresponding to the first modal data, represents the information entropy corresponding to the second modal data, Represents the exponential function.

5. The unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering according to claim 1 is characterized in that: The constructing a steady-state loss function according to the first encoding hash code, the second encoding hash code, and the fused encoding hash code includes: Normalizing the first encoded hash code to obtain a first state vector; Normalizing the second encoded hash code to obtain a second state vector; Normalizing the fused encoding hash code to obtain a third state vector; Calculating cosine similarity based on the fused encoding hash code, and constructing state transition probability based on the cosine similarity; Iterate the third state vector multiple times using the state transition probability to determine a steady-state distribution of the state vector; A steady-state loss function is constructed according to the steady-state distribution of the state vector, the first state vector, the second state vector, and the third state vector.

6. The unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering according to claim 5 is characterized in that: The constructing a steady-state loss function according to the steady-state distribution of the state vector, the first state vector, the second state vector, and the third state vector includes: ； in, represents the steady-state loss function, Indicates the number of samples corresponding to each modality, represents the third state vector, represents the steady-state distribution of the state vector, represents the first state vector, represents the second state vector, Represents the logarithmic function.

7. The unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering according to claim 1 is characterized in that: The constructing a quantization loss function according to the first encoded hash code and the second encoded hash code includes: ； in, represents the quantization loss function, represents the first encoded hash code, represents the symbolic function, Represents the second encoded hash code.

8. The unsupervised cross-modal hash retrieval method based on steady-state distribution and clustering according to claim 1 is characterized in that: The fusing the first encoding hash code and the second encoding hash code to obtain a fused encoding hash code includes: Concatenate the first encoded hash code and the second encoded hash code to obtain a concatenated encoded hash code; The concatenated encoding hash codes are nonlinearly fused to obtain a fused encoding hash code.

Images