Cross-modal hashing retrieval method and system for maintaining data and semantic consistency

Abstract

The application discloses a cross-modal hash retrieval method and system for keeping data and semantic consistency. The method comprises the following steps: obtaining common feature representation and unique feature representation of multi-modal data, and constructing a latent data consistency learning model based on the common feature representation and the unique feature representation; constructing a latent semantic consistency learning model in the form of an asymmetric norm according to the optimal feature representation of the multi-modal data; respectively iteratively optimizing the latent data consistency learning model and the latent semantic consistency learning model, and generating optimal hash codes according to the two optimized learning models; learning a hash function according to the optimal hash codes, and completing cross-modal hash retrieval according to the learned hash function. The method can maximize the data and semantic consistency of the multi-modal data, thereby improving the cross-modal retrieval effect, and can reduce the quantization error, thereby reducing the adverse effects of the quantization error on the learning and generation of the hash codes.

Description

Technical Field

[0001] This invention relates to the field of computer information retrieval technology, specifically to a cross-modal hash retrieval method and system that maintains data and semantic consistency. Background Technology

[0002] With the continuous development of computer technology, multimedia data is experiencing explosive growth, and its forms of representation are becoming increasingly diverse. To quickly and effectively retrieve relevant information from large-scale multimodal data, cross-modal retrieval has become an important research topic. Faced with the ever-increasing volume of data, traditional cross-modal retrieval methods have become impractical due to their high computational costs. Hash methods map data points to Hamming space to obtain compact, low-dimensional representations while preserving their similarity in the original feature space. Using XOR operations in Hamming space can accelerate the search process. Simultaneously, the use of binary encoding significantly reduces computational costs and storage overhead. Therefore, hashing techniques are widely used in cross-modal retrieval, and many cross-modal hashing methods have been proposed and widely applied in information retrieval, recommender systems, and other fields.

[0003] Cross-modal hashing methods map data from different modalities to the same feature space to achieve cross-modal data matching and retrieval. Based on their learning methods, they can be divided into unsupervised and supervised methods. Unsupervised methods do not require supervisory information and only learn from the features of the data itself. However, due to the diversity and complexity of data, the hash codes learned by unsupervised methods have poor discriminative power. Supervised methods use labeled data for training, which helps to learn more discriminative features. Therefore, supervised methods usually achieve better retrieval performance (e.g., the SCRATCH method uses collective matrix factorization and semantic embedding of labels to explore the latent semantic space to maintain intra-modal and inter-modal similarity of hash codes; the ALECH method adaptively uses high-order semantic label relevance to guide latent feature learning, improving cross-modal retrieval performance). However, most existing methods mainly focus on maintaining semantic similarity between heterogeneous modalities in a shared subspace, often ignoring certain underlying data features and potential semantic relevance, thus failing to fully maintain the data and semantic consistency of cross-modal data. To avoid the high computational complexity and memory overhead caused by using pairwise similarity matrices, methods such as SRLCH only use semantic label information to supervise the learning of hash codes. Because they do not fully utilize the supervisory information, they partially ignore the semantic relevance of the labels, thus reducing the retrieval results. Furthermore, most algorithms solve the binary optimization problem in the real-valued space by relaxing binary constraints, and then threshold the obtained real-valued solutions to generate binary hash codes. This two-stage process can potentially cause large quantization errors and generate suboptimal hash codes. These methods are not yet ideal in terms of data matching and retrieval accuracy and require further improvement and optimization. Therefore, there is an urgent need to provide a cross-modal hash retrieval method and system that can fully maintain the consistency of multimodal data and semantics and can learn and generate better hash codes. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a solution to the technical problems existing in the prior art.

[0005] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0006] A cross-modal hash retrieval method that maintains data and semantic consistency includes the following steps:

[0007] S101, Obtain the common feature representation and unique feature representation of the multimodal data, and construct a potential data consistency learning model for the multimodal data based on the common feature representation and unique feature representation;

[0008] S102, Construct a latent semantic consistency learning model for multimodal data in the form of an asymmetric paradigm based on the optimal feature representation of multimodal data;

[0009] S103, iteratively optimize the potential data consistency learning model and the potential semantic consistency learning model respectively, and generate the optimal hash code of the multimodal data based on the optimized two learning models;

[0010] S104, perform hash function learning based on the optimal hash code, and complete cross-modal hash retrieval of multimodal data based on the learned hash function.

[0011] As a further improvement to the above technical solution:

[0012] Step S101 includes the following steps:

[0013] Kerneling is performed on multimodal data to obtain its nonlinear structure;

[0014] Multimodal data is mapped to a common latent semantic space through collaborative semantic label matrix decomposition to obtain common feature representations of multimodal data;

[0015] Each modality of multimodal data is projected into the latent semantic space unique to that modality by using unique semantic label matrix decomposition to obtain a unique feature representation for each modality.

[0016] Based on the common feature representation and the unique feature representation, a potential data consistency learning model for multimodal data is constructed to obtain the optimal feature representation of the multimodal data.

[0017] The function expression for kernelizing multimodal data is:

[0018] ,

[0019] In the above formula, k∈{1,2} represents the image and text modalities, respectively. For the data features of the k-th mode, For random selection A sample of anchor points, The number of anchor points is set. The Gaussian kernel function;

[0020] The functional expression for the collaborative semantic tag matrix decomposition is:

[0021] ,

[0022] In the above formula, The fundamental matrix for matrix decomposition. This is the projection auxiliary matrix of the matrix decomposition. For the label matrix, For the feature matrices of image and text modalities, These are the equilibrium parameters for the k-th mode;

[0023] The functional expression for the unique semantic tag matrix decomposition is:

[0024] ,

[0025] In the above formula, Let be the fundamental matrix of the matrix decomposition of the k-th mode. Let be the projection auxiliary matrix of the matrix decomposition of the k-th mode. For the correlation matrix, For non-negative trade-off parameters, This is the projection auxiliary matrix for the matrix decomposition of the image modality. The projection auxiliary matrix for the matrix decomposition of the text modality;

[0026] The functional expression of the latent data consistency learning model is:

[0027] ,

[0028] In the above formula, This represents a regularization term to avoid overfitting. and The parameter represents a non-negative trade-off.

[0029] The functional expression representing the optimal feature is:

[0030] .

[0031] Step S102 includes the following steps:

[0032] The hash code is learned by minimizing the quantization loss between the hash code and the projection matrix after orthogonal transformation to obtain the first hash code learning representation, and the hash code is learned with Hamming affinity preservation as the objective to obtain the second hash code learning representation.

[0033] The second hash code learning representation is optimized by using an asymmetric similarity matrix. Based on the optimized second hash code learning representation and the optimal feature representation, a latent semantic consistency learning model for multimodal data is constructed.

[0034] The functional expression for the first hash code learning representation is:

[0035] st ,

[0036] In the above formula, The unified hash code matrix of the training set, It is an orthogonal rotation matrix. It is the identity matrix. This represents the optimal feature representation.

[0037] The functional expression for the second hash code learning representation is:

[0038] st ,

[0039] In the above formula, To standardize the length of hash codes, The amount of data in the training set, This is a pairwise semantic similarity matrix;

[0040] The functional expression of the latent semantic consistency learning model is:

[0041] ,

[0042] In the above formula, For auxiliary real-valued matrix, The parameter represents a non-negative trade-off.

[0043] In step S103, the latent data consistency learning model and the latent semantic consistency learning model are iteratively optimized, including the following steps:

[0044] Each variable in the latent data consistency learning model is updated sequentially, while other variables are fixed when updating one variable, and the updating is stopped when the objective function of the latent data consistency learning model is determined to be converged, so as to obtain the optimal feature representation when the objective function of the latent data consistency learning model converges.

[0045] The latent semantic consistency learning model is updated based on the optimal feature representation. Each variable in the latent semantic consistency learning model is updated sequentially. When updating one variable, the other variables are fixed. The update stops when the objective function of the latent semantic consistency learning model is determined to have converged, so as to obtain the optimized latent semantic consistency learning model.

[0046] The step of sequentially updating each variable in the potential data consistency learning model includes the following steps:

[0047] renew ,fixed and ,get The closed-form solution is:

[0048] ,

[0049] In the above formula, The parameter represents a non-negative trade-off.

[0050] renew ,fixed and ,get The closed-form solution is:

[0051] ;

[0052] renew ,fixed and ,get The closed-form solution is:

[0053] ;

[0054] renew ,fixed and ,get The closed-form solution is:

[0055] ,

[0056] In the above formula, These are the equilibrium parameters for the first mode;

[0057] renew ,fixed and ,get The closed-form solution is:

[0058] ,

[0059] In the above formula, These are the equilibrium parameters for the second mode;

[0060] renew ,fixed and ,get The closed-form solution is:

[0061] ;

[0062] The step of sequentially updating each variable in the latent semantic consistency learning model includes the following steps:

[0063] renew ,fixed Singular value decomposition is used to obtain The solution is:

[0064] ,

[0065] In the above formula, It is a unitary matrix;

[0066] renew ,fixed Through auxiliary real-valued matrix Alternative ,get The closed-form solution is:

[0067] ,

[0068] In the above formula, It is a symbolic function. A column vector consisting entirely of 1s. For non-negative trade-off parameters, For auxiliary matrix variables;

[0069] renew ,fixed ,get The closed-form solution is:

[0070] .

[0071] Step S104 includes the following steps:

[0072] Feature matrices from image and text modalities and the hash code of learning Come to train The objective function expression for the learning process of a binary classifier is:

[0073] ,

[0074] In the above formula, Let be the optimal projection matrix for the k-th modality, and let represent the linear mapping matrix from kernel features to hash codes. The parameter represents a non-negative trade-off.

[0075] get The optimal solution is:

[0076] ;

[0077] In the process of cross-modal hash retrieval of multimodal data based on the learned hash function, a mapping matrix is ​​used. The hash code is obtained, where the expression of the hash function is:

[0078] ,

[0079] In the above formula, This represents the search term for the k-th mode. This indicates the extraction of kernelized features.

[0080] The present invention also provides a cross-modal hash retrieval system that maintains data and semantic consistency, comprising an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to perform the above-described cross-modal hash retrieval method that maintains data and semantic consistency.

[0081] The present invention also provides a computer-readable storage medium storing a computer program that is programmed or configured by a microprocessor to perform the above-described cross-modal hash retrieval method for maintaining data and semantic consistency.

[0082] Compared with the prior art, the advantages of the present invention are as follows:

[0083] This invention employs a two-step learning method—latent data consistency learning and latent semantic consistency learning—to learn hash codes for multimodal data. This maximizes the preservation of data and semantic consistency across multimodal data, thereby improving cross-modal retrieval performance. By simultaneously preserving both common and unique features of multimodal data and constructing a latent data consistency learning model, it can consider both common features among different modalities and unique features of each modality, avoiding the adverse effects on latent semantic space learning caused by ignoring certain underlying data features and potential semantic correlations. Furthermore, by constructing the latent semantic consistency learning model for multimodal data using an asymmetric paradigm, quantization errors can be reduced, thus mitigating the adverse effects of quantization errors on hash code learning and generation. Attached Figure Description

[0084] Figure 1 A flowchart of a cross-modal hash retrieval method for maintaining data and semantic consistency provided in an embodiment of the present invention.

[0085] Figure 2 This is a flowchart illustrating the specific steps of the cross-modal hash retrieval method for maintaining data and semantic consistency in this embodiment. Detailed Implementation

[0086] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0087] like Figure 1 and Figure 2 As shown, this embodiment provides a cross-modal hash retrieval method that maintains data and semantic consistency, including the following steps:

[0088] S101, Obtain the common feature representation and unique feature representation of multimodal data, and construct a potential data consistency learning model for multimodal data based on the common feature representation and unique feature representation;

[0089] S102, Construct a latent semantic consistency learning model for multimodal data in the form of an asymmetric paradigm based on the optimal feature representation of multimodal data;

[0090] S103, iteratively optimize the latent data consistency learning model and the latent semantic consistency learning model respectively, and generate the optimal hash code for multimodal data based on the optimized two learning models;

[0091] S104. Learn the hash function based on the optimal hash code, and complete the cross-modal hash retrieval of multimodal data based on the learned hash function.

[0092] It is understood that this embodiment learns the hash code of multimodal data through a two-step learning method of latent data consistency learning and latent semantic consistency learning, which can maximize the preservation of data and semantic consistency of multimodal data, thereby improving the cross-modal retrieval effect. This embodiment simultaneously retains the common features and unique features of multimodal data and constructs a latent data consistency learning model, which can take into account both the common features between different modal data and the unique features of each modal data, avoiding the adverse effects on latent semantic space learning caused by ignoring certain underlying features and potential semantic correlations of data. This embodiment constructs the latent semantic consistency learning model of multimodal data in the form of an asymmetric paradigm, which can reduce quantization error, thereby reducing the adverse effects of quantization error on hash code learning and generation.

[0093] In this embodiment, step S101 includes the following steps:

[0094] Kerneling is performed on multimodal data to obtain its nonlinear structure;

[0095] Multimodal data is mapped to a common latent semantic space through collaborative semantic label matrix decomposition to obtain common feature representations of multimodal data;

[0096] Each modality of multimodal data is projected into the latent semantic space unique to that modality by using unique semantic label matrix decomposition to obtain a unique feature representation for each modality.

[0097] A latent data consistency learning model for multimodal data is constructed based on common feature representations and unique feature representations to obtain the optimal feature representation of multimodal data.

[0098] In this embodiment, the function expression for kernelizing multimodal data is:

[0099] ,

[0100] In the above formula, k∈{1,2} represents the image and text modalities, respectively. For the data features of the k-th mode, For random selection A sample of anchor points, The number of anchor points is set. The Gaussian kernel function;

[0101] The functional expression for the co-semantic tag matrix decomposition is:

[0102] ,

[0103] In the above formula, The fundamental matrix for matrix decomposition. This is the projection auxiliary matrix of the matrix decomposition. For the label matrix, For the feature matrices of image and text modalities, These are the equilibrium parameters for the k-th mode;

[0104] The function expression for the unique semantic tag matrix decomposition is:

[0105] ,

[0106] In the above formula, Let be the fundamental matrix of the matrix decomposition of the k-th mode. Let be the projection auxiliary matrix of the matrix decomposition of the k-th mode. For the correlation matrix, For non-negative trade-off parameters, This is the projection auxiliary matrix for the matrix decomposition of the image modality. The projection auxiliary matrix for the matrix decomposition of the text modality;

[0107] The functional expression of the latent data consistency learning model is:

[0108] ,

[0109] In the above formula, This represents a regularization term to avoid overfitting. and The parameter represents a non-negative trade-off.

[0110] The functional expression for the optimal feature representation is:

[0111] .

[0112] Understandably, in order to more effectively extract common features from multimodal data and bridge inconsistencies between different modalities, this embodiment learns a common latent semantic space through co-semantic label consistency matrix decomposition, that is, mapping multimodal data into a common latent semantic space to preserve their common features. To better learn the underlying features of cross-modal data and maintain data consistency, this embodiment projects the data of each modality into its unique latent semantic space and explores the cross-correlation between them by using a correlation matrix, thereby preserving the unique features of various modalities as much as possible, making the overall reliability of the data higher. Finally, a latent data consistency learning model is constructed based on the common and unique features of multimodal data, and the optimal feature representation of multimodal data in the latent semantic space is obtained for use in subsequent learning steps.

[0113] In a specific application embodiment, the specific steps of potential data consistency learning (corresponding to step S101) are as follows (parameter definitions are detailed in Table 1):

[0114] Step 1: First, kernelize the multimodal data to obtain a more effective nonlinear structure of the data.

[0115] ;

[0116] Step 2: Collective Matrix Factorization (CMF) is a direct and effective method for discovering connections between multimodal data. To more effectively extract common features from multimodal data and bridge inconsistencies between different modalities, a common latent semantic space is learned through Co-Semantic Label Consistency Matrix Factorization. This maps multimodal data into a common latent semantic space to preserve their common features. The objective function of Co-Semantic Label Matrix Factorization is as follows:

[0117] ;

[0118] Step 3: The heterogeneity of multimodal data may lead to the loss of valuable modality-specific information. To better learn the underlying features of cross-modal data and maintain data consistency, the data of each modality is projected into its unique latent semantic space, and a correlation matrix is ​​used. To explore the cross-correlation between them, thereby preserving the unique features of each modality as much as possible, the objective function of the unique semantic label matrix decomposition is as follows:

[0119] ;

[0120] Step 4, combining the collaborative semantic label matrix decomposition and the unique semantic label decomposition process, the latent data consistency learning objective function can be expressed as:

[0121] ;

[0122] In the above formula, This represents a regularization term to avoid overfitting. and The parameter represents a non-negative trade-off.

[0123] Step 5, for the kernelized multimodal data decomposed through collaborative and unique semantic tag matrix decomposition, since Through the correlation matrix Convert to , can be adopted To represent a specific latent representation matrix of an image-text pair, the common feature representation and the unique feature representation are fused in the following manner to obtain the optimal feature representation, which is:

[0124] .

[0125] In this embodiment, step S102 includes the following steps:

[0126] The hash code is learned by minimizing the quantization loss between the hash code and the projection matrix after orthogonal transformation to obtain the first hash code learning representation, and the hash code is learned with Hamming affinity preservation as the objective to obtain the second hash code learning representation.

[0127] The second hash code learning representation is optimized by using an asymmetric similarity matrix. Based on the optimized second hash code learning representation and the optimal feature representation, a latent semantic consistency learning model for multimodal data is constructed.

[0128] In this embodiment, the functional expression for the first hash code learning representation is:

[0129] st ,

[0130] In the above formula, The unified hash code matrix of the training set, It is an orthogonal rotation matrix. It is the identity matrix. This represents the optimal feature representation.

[0131] The functional expression for the second hash code learning representation is:

[0132] st ,

[0133] In the above formula, To standardize the length of hash codes, The amount of data in the training set, This is a pairwise semantic similarity matrix;

[0134] The functional expression of the latent semantic consistency learning model is:

[0135] ,

[0136] In the above formula, For auxiliary real-valued matrix, The parameter represents a non-negative trade-off.

[0137] It is understandable that, in order to better preserve the original local structure of the data, this embodiment learns the hash code of the training set by minimizing the quantization loss between the hash code after orthogonal transformation and the projection matrix (i.e., the first hash code learning representation); at the same time, in order to make full use of the supervision information and maintain the data and semantic consistency between data points, the objective of Hamming affinity preservation is adopted to jointly learn the hash code (i.e., the second hash code learning representation), and the quantization error is reduced in the form of an asymmetric paradigm to obtain the objective function of the asymmetric similarity matrix learning hash code (i.e., the latent semantic consistency learning model), which can reduce quantization error, reduce training time, and improve learning efficiency.

[0138] In a specific application embodiment, the specific steps of latent semantic consistency learning (corresponding to step S102) are as follows (parameter definitions are detailed in Table 1):

[0139] Step 1: To better preserve the original local structure of the data, minimize the hash code after orthogonal transformation. and projection matrix The hash codes of the training set are used to learn the quantization loss between them.

[0140] st ,

[0141] Step 2: To fully utilize the supervisory information of semantic labels and retain it in the hash code, while maintaining data and semantic consistency between data points, a Hamming affinity-preserving objective is adopted to jointly learn the hash code.

[0142] st ,

[0143] Step 3: The formula in Step 2 was originally proposed in Kernel Supervised Hashing (KSH). Although it has been shown to improve the accuracy of hash codes and retrieval performance, KSH algorithm optimization is a symmetric binary inner product problem. Therefore, an auxiliary real-valued matrix is ​​used. Replace one of the symmetric bivariate inner products This reduces quantization error using an asymmetric paradigm. Steps 1 and 2 are combined to learn the hash code, resulting in the following objective function for learning the hash code from the asymmetric similarity matrix:

[0144] ;

[0145] It should be noted that, due to pairwise semantic similarity matrices yes The size significantly increases the time and space complexity of the optimization process; therefore, direct computation is avoided here. The result, but will The calculation is incorporated into the matrix. and During the update and optimization process, the algorithm complexity is reduced from... Simplified to This significantly reduced training time.

[0146] In step S103 of this embodiment, the latent data consistency learning model and the latent semantic consistency learning model are iteratively optimized, including the following steps:

[0147] Each variable in the latent data consistency learning model is updated sequentially. When updating one variable, the other variables are fixed. The updating stops when the objective function of the latent data consistency learning model is determined to be converged, so as to obtain the optimal feature representation when the objective function of the latent data consistency learning model converges.

[0148] The latent semantic consistency learning model is updated based on the optimal feature representation. Each variable in the latent semantic consistency learning model is updated sequentially, while other variables are fixed when updating one variable. The update stops when the objective function of the latent semantic consistency learning model is determined to have converged, so as to obtain the optimized latent semantic consistency learning model.

[0149] It should be noted that the optimization problem of the function representation of the latent data consistency learning model and the latent semantic consistency learning model is non-convex with respect to the overall matrix variables. However, when keeping other matrix variables constant, considering only the nine matrix variables (i.e., ...) When they are convex, this embodiment proposes an iterative algorithm to solve the optimization problem. By updating the two parts of latent data consistency learning and latent semantic consistency learning respectively, and using the iterative algorithm to converge these two parts to their local minimum, the data and semantic consistency are maximized.

[0150] In this embodiment, each variable in the potential data consistency learning model is updated sequentially, including the following steps:

[0151] renew ,fixed and ,get The closed-form solution is:

[0152] ,

[0153] In the above formula, The parameter represents a non-negative trade-off.

[0154] renew ,fixed and ,get The closed-form solution is:

[0155] ;

[0156] renew ,fixed and ,get The closed-form solution is:

[0157] ;

[0158] renew ,fixed and ,get The closed-form solution is:

[0159] ,

[0160] In the above formula, These are the equilibrium parameters for the first mode;

[0161] renew ,fixed and ,get The closed-form solution is:

[0162] ,

[0163] In the above formula, These are the equilibrium parameters for the second mode;

[0164] renew ,fixed and ,get The closed-form solution is:

[0165] ;

[0166] Update each variable in the latent semantic consistency learning model sequentially, including the following steps:

[0167] renew ,fixed Singular value decomposition is used to obtain The solution is:

[0168] ,

[0169] In the above formula, It is a unitary matrix;

[0170] renew ,fixed Through auxiliary real-valued matrix Alternative ,get The closed-form solution is:

[0171] ,

[0172] In the above formula, It is a symbolic function. A column vector consisting entirely of 1s. For non-negative trade-off parameters, For auxiliary matrix variables;

[0173] renew ,fixed ,get The closed-form solution is:

[0174] .

[0175] As can be understood, this embodiment first alternately updates each variable in the latent data consistency learning model. When the objective function value gradually approaches a certain fixed value, it indicates that the objective function has converged, thus obtaining the optimal feature representation of the multimodal data. Then, it alternately updates each variable in the latent semantic consistency learning model until the objective function converges. During this learning process, a hash code matrix can be obtained. This generates a hash code.

[0176] Step S104 includes the following steps:

[0177] Feature matrices from image and text modalities and the hash code of learning Come to train The objective function expression for the learning process of a binary classifier is:

[0178] ,

[0179] In the above formula, Let be the optimal projection matrix for the k-th modality, and let represent the linear mapping matrix from kernel features to hash codes. The parameter represents a non-negative trade-off.

[0180] get The optimal solution is:

[0181] ;

[0182] In the process of cross-modal hash retrieval of multimodal data based on the learned hash function, a mapping matrix is ​​used. The hash code is obtained, where the expression of the hash function is:

[0183] ,

[0184] In the above formula, This represents the search term for the k-th mode. Indicates kernelized feature extraction, It is a symbolic function that converts consecutive data into hash codes.

[0185] It should be noted that the definitions of the parameter symbols mentioned in the text are detailed in Table 1 below:

[0186]

[0187] Table 1

[0188] During training, by setting different parameter values ​​and analyzing the retrieval results under different values, the optimal parameter configuration can be determined as follows:

[0189] .

[0190] This embodiment also provides a cross-modal hash retrieval system that maintains data and semantic consistency, including a microprocessor and a memory interconnected thereto. The microprocessor is programmed or configured to execute the aforementioned cross-modal hash retrieval method that maintains data and semantic consistency. This embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by the microprocessor to execute the aforementioned cross-modal hash retrieval method that maintains data and semantic consistency. The system and medium of this invention, corresponding to the methods described above, also possess the advantages of the methods described above.

[0191] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. Computer-readable media include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0192] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A cross-modal hash retrieval method that maintains data and semantic consistency, characterized in that, Including the following steps: S101, Obtain the common feature representation and unique feature representation of the multimodal data, and construct a potential data consistency learning model for the multimodal data based on the common feature representation and unique feature representation; S102, Construct a latent semantic consistency learning model for multimodal data in the form of an asymmetric paradigm based on the optimal feature representation of multimodal data; S103, iteratively optimize the potential data consistency learning model and the potential semantic consistency learning model respectively, and generate the optimal hash code of the multimodal data based on the optimized two learning models; S104, perform hash function learning based on the optimal hash code, and complete cross-modal hash retrieval of multimodal data based on the learned hash function; Step S101 includes the following steps: Kerneling is performed on multimodal data to obtain its nonlinear structure; Multimodal data is mapped to a common latent semantic space through collaborative semantic label matrix decomposition to obtain common feature representations of multimodal data; Each modality of multimodal data is projected into the latent semantic space unique to that modality by using unique semantic label matrix decomposition to obtain a unique feature representation for each modality. Based on the common feature representation and the unique feature representation, a potential data consistency learning model for multimodal data is constructed to obtain the optimal feature representation of the multimodal data; Step S102 includes the following steps: The hash code is learned by minimizing the quantization loss between the hash code and the projection matrix after orthogonal transformation to obtain the first hash code learning representation, and the hash code is learned with Hamming affinity preservation as the objective to obtain the second hash code learning representation. The second hash code learning representation is optimized by using an asymmetric similarity matrix. Based on the optimized second hash code learning representation and the optimal feature representation, a latent semantic consistency learning model for multimodal data is constructed.

2. The cross-modal hash retrieval method for maintaining data and semantic consistency according to claim 1, characterized in that, The function expression for kernelizing multimodal data is: ， In the above formula, k∈{1,2} represents the image and text modalities, respectively. For the data features of the k-th mode, For random selection A sample of anchor points, The number of anchor points is set. The Gaussian kernel function; The functional expression for the collaborative semantic tag matrix decomposition is: ， In the above formula, The fundamental matrix for matrix decomposition. This is the projection auxiliary matrix of the matrix decomposition. For the label matrix, For the feature matrices of image and text modalities, These are the equilibrium parameters for the k-th mode; The functional expression for the unique semantic tag matrix decomposition is: ， In the above formula, Let be the fundamental matrix of the matrix decomposition of the k-th mode. Let be the projection auxiliary matrix of the matrix decomposition of the k-th mode. For the correlation matrix, For non-negative trade-off parameters, This is the projection auxiliary matrix for the matrix decomposition of the image modality. The projection auxiliary matrix for the matrix decomposition of the text modality; The functional expression of the latent data consistency learning model is: ， In the above formula, This represents a regularization term to avoid overfitting. and The parameter represents a non-negative trade-off. The functional expression representing the optimal feature is: 。 3. The cross-modal hash retrieval method for maintaining data and semantic consistency according to claim 1, characterized in that, The functional expression for the first hash code learning representation is: s.t. , In the above formula, The unified hash code matrix of the training set, It is an orthogonal rotation matrix. It is the identity matrix. This represents the optimal feature representation. The functional expression for the second hash code learning representation is: s.t. , In the above formula, To standardize the length of hash codes, The amount of data in the training set, This is a pairwise semantic similarity matrix; The functional expression of the latent semantic consistency learning model is: ， In the above formula, For auxiliary real-valued matrix, The parameter represents a non-negative trade-off.

4. The cross-modal hash retrieval method for maintaining data and semantic consistency according to claim 1, characterized in that, In step S103, the latent data consistency learning model and the latent semantic consistency learning model are iteratively optimized, including the following steps: Each variable in the latent data consistency learning model is updated sequentially, while other variables are fixed when updating one variable, and the updating is stopped when the objective function of the latent data consistency learning model is determined to be converged, so as to obtain the optimal feature representation when the objective function of the latent data consistency learning model converges. The latent semantic consistency learning model is updated based on the optimal feature representation. Each variable in the latent semantic consistency learning model is updated sequentially. When updating one variable, the other variables are fixed. The update stops when the objective function of the latent semantic consistency learning model is determined to have converged, so as to obtain the optimized latent semantic consistency learning model.

5. The cross-modal hash retrieval method for maintaining data and semantic consistency according to claim 4, characterized in that, The step of sequentially updating each variable in the potential data consistency learning model includes the following steps: renew ,fixed and ,get The closed-form solution is: ， In the above formula, The parameter represents a non-negative trade-off. renew ,fixed and ,get The closed-form solution is: ； renew ,fixed and ,get The closed-form solution is: ； renew ,fixed and ,get The closed-form solution is: ， In the above formula, These are the equilibrium parameters for the first mode; renew ,fixed and ,get The closed-form solution is: ， In the above formula, These are the equilibrium parameters for the second mode; renew ,fixed and ,get The closed-form solution is: ； The step of sequentially updating each variable in the latent semantic consistency learning model includes the following steps: renew ,fixed Singular value decomposition is used to obtain The solution is: ， In the above formula, It is a unitary matrix; renew ,fixed Through auxiliary real-valued matrix Alternative ,get The closed-form solution is: ， In the above formula, It is a symbolic function. A column vector consisting entirely of 1s. For non-negative trade-off parameters, For auxiliary matrix variables; renew ,fixed ,get The closed-form solution is: 。 6. The cross-modal hash retrieval method for maintaining data and semantic consistency according to claim 1, characterized in that, Step S104 includes the following steps: Feature matrices from image and text modalities and the hash code of learning Come to train The objective function expression for the learning process of a binary classifier is: ， In the above formula, Let be the optimal projection matrix for the k-th modality, and let represent the linear mapping matrix from kernel features to hash codes. The parameter represents a non-negative trade-off. get The optimal solution is: ； In the process of cross-modal hash retrieval of multimodal data based on the learned hash function, a mapping matrix is ​​used. The hash code is obtained, where the expression of the hash function is: ， In the above formula, This represents the search term for the k-th mode. This indicates the extraction of kernelized features.

7. A cross-modal hash retrieval system that maintains data and semantic consistency, comprising interconnected microprocessors and memory, characterized in that, The microprocessor is programmed or configured to perform the cross-modal hash retrieval method for maintaining data and semantic consistency as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to perform the cross-modal hash retrieval method for maintaining data and semantic consistency as described in any one of claims 1 to 6.

Images