Large-scale low-quality multimodal unsupervised hashing method and dynamic streaming data retrieval system

Abstract

The present invention belongs to the field of multimodal information processing, and discloses a large-scale low-quality multimodal unsupervised hashing method and a dynamic streaming data retrieval system. In order to fully mine the strong discriminative hash codes of large-scale low-quality multimodal data, a multi-layer hash code mapping mode of inner and outer layers is adopted to construct a sparse regularized multi-layer hash mapping model; in order to fuse the unique feature content and semantic consistency feature content between various modal data in low-quality multimodal data, the reduced modal spaces corresponding to different modal mappings are mapped, and each reduced modal space is modeled by inline coupling. The present invention further extends the above-mentioned application method to a dynamic multimodal streaming data environment, and the global loss function of the dynamic streaming data hash function solver constructed can gradually, orderly and stably mine the semantic information of dynamic streaming data, enhance the learning discriminative power of the low-quality dynamic streaming data hash function, and realize rapid retrieval of large-scale low-quality dynamic multimodal streaming data.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal information processing, and in particular relates to a large-scale low-quality multimodal unsupervised hashing method and a dynamic stream data retrieval system. Background Art

[0002] With the exponential growth of massive multimedia data, how to efficiently extract useful information from large-scale, heterogeneous, and multimodal multimedia data, and multimodal hash learning are increasingly gaining widespread attention in fields such as computer vision and information processing. However, in the face of large-scale low-quality environments, it is often difficult to obtain large-scale labeled data due to the cost and difficulty of data collection, and the complexity of heterogeneous scenes further increases the difficulty of data annotation, which makes multimodal data accompanied by incomplete labeling information and non-constant data distribution. Therefore, how to use the correlation between low-quality sample information and heterogeneous modalities in the absence of comprehensive annotations to mine useful knowledge contained in massive low-quality heterogeneous data and accurately and efficiently process valuable information in multimodal data has become an urgent issue in the field of multimodal information processing research. For this reason, unsupervised multimodal hash learning methods have received widespread attention in the current field of multimodal information processing technology.

[0003] In order to better capture multimodal heterogeneous semantic association information and reduce the heterogeneity of each modality, the literature (Liu X.L., Mu YD, Zhang DC, et al. Large-Scale Unsupervised Hashing with Shared Structure Learning) proposed large-scale unsupervised hashing (SSH) to decompose the feature space into a subspace shared by all hash functions and its complementary subspace. In order to embed supervised label information into the hash code to be learned, the literature (Zhan YW, Wang YX, Sun Y., et al. Discrete online cross-modal hashing) designed a discrete online cross-modal hashing scheme to combine similarity preservation and label embedding into a unified framework. Considering the ability to discretely solve binary constraint problems, the literature (Wang YX, Luo X., Xu X.S. Label embedding online hashing for cross-modal retrieval.) proposes a label embedding online cross-modal hashing method, which mines semantic structures through a label embedding framework that includes label similarity preservation and label reconstruction to generate more discriminative hash codes. However, most multimodal hashing models map multimodal data to a common representation space, which is not enough to eliminate the heterogeneity of each modality. Secondly, in order to achieve the mapping from high-dimensional hash codes to low-dimensional hash codes, the current methods are more likely to use a single-layer hash function mapping mode. If a one-step dimensionality reduction hash learning mode is used, it will lose some key data semantic information or discriminative information when applied to large-scale low-quality multimodal data sets. At the same time, when facing large-scale low-quality heterogeneous streaming media data, the amount of low-quality heterogeneous streaming media data changes dynamically and in real time, presenting a cumulative process. Therefore, it is impossible to directly use offline multimodal hashing methods for processing. At the same time, dynamic streaming data also has characteristics such as randomness and diversity. Traditional offline multimodal hashing algorithms cannot adapt to data changes in dynamic environments. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a large-scale low-quality multimodal unsupervised hashing method and a dynamic streaming data retrieval system, which solves the problem of how to efficiently mine multimodal deep-level semantically consistent content in a low-quality environment, while minimizing the interference of noise data and redundant data on multimodal hash learning, thereby realizing rapid retrieval of large-scale low-quality dynamic multimodal streaming data.

[0005] The large-scale low-quality multimodal unsupervised hashing method of the present invention comprises the following steps:

[0006] Step 1: Acquire multimodal data in a large-scale low-quality environment, extract sample feature information of the multimodal data, and perform sample centering and radial basis function (RBF) kernelization processing on the obtained sample features to obtain processed multimodal data features;

[0007] Step 2: Construct a sparse regularized hierarchical hash mapping model to mine the strong semantic discriminant hash codes corresponding to multimodal samples in a large-scale low-quality environment, and construct a multimodal matrix decomposition hash learning model to integrate the unique feature content and semantic consistency feature content between various modal data to obtain a multimodal unsupervised hash learning global loss function; solve the hash code of multimodal data and calculate the hash function corresponding to each modality;

[0008] Step 3: Use the hash function corresponding to the query sample modality to perform hash coding mapping on the query sample to obtain the query hash code corresponding to the query sample; perform Hamming distance calculation on the obtained query hash code and the hash code in the multimodal data in a large-scale low-quality environment to obtain the corresponding retrieval result.

[0009] Furthermore, step 1 is specifically as follows:

[0010] The multimodal data includes low-quality image samples and low-quality text samples. The feature information of the low-quality image samples is extracted using an image deep network model to obtain low-quality image features S1, and the feature information of the low-quality text samples is extracted using a text deep network model to obtain low-quality text features S2; the low-quality image features and the low-quality text features are respectively subjected to sample centering processing and RBF kernelization processing to obtain processed image features and text features

[0011] Furthermore, step 2 is specifically as follows:

[0012] Step 2-1: In order to fully explore the strong semantic discriminant hash codes corresponding to multimodal samples in large-scale low-quality environments, a sparse regularized multi-layer hash mapping model is constructed:

[0013]

[0014] Where B is the hash code to be learned, r is the hash code length, n is the size of the multimodal dataset, P1 is the outer hash mapping matrix of the image modality, R1 is the inner hash mapping matrix of the image modality, P2 is the outer hash mapping matrix of the text modality, R2 is the inner hash mapping matrix of the text modality, θ is the multi-layer hash mapping balance parameter, δ is the sparse regularization parameter, tr(.) represents the trace operation of the matrix, and ||.|| represents the L of the matrix. 21 norm;

[0015] Step 2-2: In order to integrate the unique feature content and semantic consistency feature content between various modal data, a multimodal matrix decomposition hash learning model is constructed:

[0016]

[0017] Where U1 is the conversion matrix of the image modality, E1 is the unique matrix of the image modality, U2 is the conversion matrix of the text modality, E2 is the unique matrix of the text modality, β is the balance parameter, a is the multimodal unique feature parameter, and μ is the regularization parameter;

[0018] Step 2-3: Combine steps 2-1 and 2-2 to construct a global loss function for multimodal unsupervised hash learning:

[0019]

[0020] Step 2-4, use the alternating iterative optimization solution method to solve the values ​​of parameters R1, R2, P1, P2, U1, U2, E1, E2, and B;

[0021] Step 2-5: Based on the parameters R1, R2, P1, and P2, further calculate the hash function H1 of the image modality, and its value is H1=P1R1; similarly, calculate the hash function H2 of the text modality, and its value is H2=P2R2.

[0022] Furthermore, steps 2-4 are specifically as follows:

[0023] Step 2-4-1, when updating parameter R1 and fixing other parameters, the value of R1 is calculated as:

[0024]

[0025] Where M is the multi-level transition diagonal matrix of the image modality, and the value of the i-th row and i-th column in the diagonal matrix M is i=1,...,k1, k1 is the row dimension of P1, ||.||2 is the 2-norm of the matrix;

[0026] Step 2-4-2, when the parameter R2 is updated and other parameters are fixed, the value of R2 is calculated as:

[0027]

[0028] Where N is the multi-level transition diagonal matrix of the text mode, and the value of the i-th row and i-th column in the diagonal matrix N is i=1,...,k2, k2 is the row dimension of P2.

[0029] Step 2-4-3, when updating parameter P1 and fixing other parameters, the value of P1 is calculated as:

[0030]

[0031] Step 2-4-4, when updating parameter P2 and fixing other parameters, the value of P2 is calculated as:

[0032]

[0033] Step 2-4-5, when updating parameter U1 and fixing other parameters, the value of U1 is calculated as:

[0034]

[0035] Step 2-4-6, when updating parameter U2, fix other parameters. The value of U2 is calculated as:

[0036]

[0037] Step 2-4-7, when updating parameter E1 and fixing other parameters, the value of E1 is calculated as:

[0038]

[0039] Step 2-4-8, when updating parameter E2 and fixing other parameters, the value of E2 is calculated as:

[0040]

[0041] Step 2-4-9, when updating parameter B and fixing other parameters, the value of B is calculated as:

[0042]

[0043] Where sgn(.) is the sign function.

[0044] Furthermore, in step 3, by using the hash function corresponding to the query sample modality, the query sample is converted and mapped to obtain the hash code corresponding to the query sample modality, specifically:

[0045] Assume that when the query sample is image modality data According to the obtained hash function H1 of the image modality, the hash code B of the image modality data to be retrieved is obtained I_query for:

[0046]

[0047] The obtained query hash code is used to calculate the Hamming distance with the hash code in the multimodal data set, and the calculated Hamming distances are sorted from small to large, and the first s multimodal data are selected as the retrieval result output, where s is an integer greater than or equal to 1;

[0048] Assume that when the query sample is text modal data According to the hash function H2 of the obtained text modality, the hash code B of the text modality data to be retrieved is obtained T_query for:

[0049]

[0050] The obtained query hash code is used to calculate the Hamming distance with the hash code in the multimodal data set, and the calculated Hamming distances are sorted from small to large. The first s multimodal data are selected as the retrieval result output, where s is an integer greater than or equal to 1.

[0051] The present invention also provides a large-scale low-quality multimodal dynamic streaming data retrieval system for implementing the above method, the system comprising a dynamic streaming data feature extractor module, a dynamic streaming data hash function solver module and a dynamic streaming data multimodal retriever module;

[0052] The dynamic stream data feature extractor module uses the corresponding modality feature extractor to extract the feature information of the corresponding modality;

[0053] The dynamic stream data hash function solver module adopts a one-step hash function solving process, and combines the modal feature information extracted by the dynamic stream data feature extractor module to expand the multimodal unsupervised hashing method in a large-scale low-quality environment, and obtains the corresponding dynamic stream data hash function solver global loss function; and adopts an alternating iterative optimization solving method to solve the parameters of the dynamic stream data hash function solver global loss function, and then obtains the dynamic stream data hash functions corresponding to different modalities;

[0054] The dynamic stream data multimodal retriever module further obtains the corresponding dynamic stream data query hash code through the dynamic stream data hash function corresponding to each modality, and performs Hamming distance calculation on the solved dynamic stream data query hash code and the hash code in the dynamic stream data multimodal data, sorts the solved Hamming distances from small to large, and selects the first s multimodal data as the retrieval result output.

[0055] The beneficial effects of the present invention are:

[0056] 1) In the scenario of large-scale low-quality unsupervised multimodal content retrieval, due to the lack of supervision information of multimodal data, it is impossible to directly use label information to guide the learning of hash codes; and various modal data often contain many redundant and damaged features. These noise points have a great impact on the learning of hash codes, which directly leads to a decrease in retrieval accuracy. In order to fully mine the highly discriminative hash codes of large-scale low-quality multimodal data, the method described in the present invention adopts a multi-layer hash code mapping mode of inner and outer layers to construct a sparse regularized multi-layer hash mapping model; at the same time, in order to integrate the unique feature content and semantic consistency feature content between various modal data in low-quality multimodal data, the reduced modal space corresponding to different modal mappings is adopted, and each reduced modal space is inline coupled to model, which can effectively mine the low-level data semantic characteristics of low-quality heterogeneous data and effectively reduce the influence of the diversity of topological distribution of low-quality heterogeneous data on the adaptability of multimodal hash learning models;

[0057] 2) In order to handle the changes of streaming media multimodal data in a dynamic environment, the present invention expands the online mode of the proposed large-scale low-quality multimodal unsupervised hashing method, so that it can capture more global semantic information in the process of streaming media data learning; at the same time, compared with other online incremental hashing methods, the present invention not only enhances the learning ability of the hash function of low-quality dynamic multimodal streaming data, but also can effectively solve the problems of missing and erroneous semantic label information and the influence of noise sample information on the system in a large-scale low-quality dynamic environment, and has strong anti-noise and anti-interference capabilities; the constructed dynamic streaming data hash function solver global loss function can gradually, orderly and stably mine the semantic information of dynamic streaming data, so as to achieve a balance between time and space overhead, which is not only scalable and adaptable, but also meets the growing needs of online learning and accelerates the retrieval rate of low-quality dynamic multimodal streaming data.

[0058] 3) The large-scale low-quality multimodal dynamic stream data retrieval system applied for in the present invention includes a dynamic stream data feature extractor module, a dynamic stream data hash function solver module and a dynamic stream data multimodal retriever module. Since the dynamic stream data feature extractor module utilizes the corresponding modal feature extractor to extract the feature information of the corresponding modality, the module has a wider scalability and can include multiple modalities such as image modality, text modality, voice modality, video modality, etc. The dynamic stream data hash function solver module ensures to the greatest extent that the newly-inflowing dynamic stream data can quickly learn the hash code distribution structure corresponding to the overall multimodal samples, and adopts a one-step hash function solving process to effectively avoid quantization errors and other problems, thereby improving the dynamic solution efficiency of the hash function of dynamic stream data. The dynamic stream data multimodal retriever module can realize fast and accurate retrieval of large-scale low-quality dynamic multimodal stream data. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a structural block diagram of the method described in the present invention.

[0060] Figure 2 It is a structural schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0061] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0062] like Figure 1 As shown, a large-scale low-quality multimodal unsupervised hashing method according to the present invention comprises the following steps:

[0063] Step 1, obtaining multimodal data in a large-scale low-quality environment, including low-quality image samples and low-quality text samples; further extracting feature information of the low-quality image samples to obtain low-quality image features, and feature information of the low-quality text samples to obtain low-quality text features; and performing sample centering processing and RBF kernelization processing on the low-quality image features and low-quality text features, respectively, to obtain processed image features and text features;

[0064] Step 2. In large-scale low-quality environmental applications, various modal data often contain many redundant and damaged features. These noise points have a great impact on the learning of hash codes, which directly leads to a decrease in retrieval accuracy. A sparse regularized hierarchical hash mapping model is constructed; in order to integrate the unique feature content and semantic consistency feature content between various modal data, the reduced modal spaces corresponding to different modal mappings are adopted, and each reduced modal space is inline coupled to model, and a multimodal matrix decomposition hash learning model is constructed; the sparse regularized hierarchical hash mapping model and the multimodal matrix decomposition hash learning model are integrated to obtain a multimodal unsupervised hash global quantization loss function, and the hash code is solved by an alternating iterative optimization method, and the hash function corresponding to each modality is calculated;

[0065] Step 3: Input the sample to be queried, perform feature extraction and RBF kernelization processing on it, and perform hash coding mapping on the query sample by using the hash function corresponding to the query sample modality to obtain the corresponding query hash code; perform Hamming distance calculation on the obtained query hash code and the hash code in the multimodal data in a large-scale low-quality environment to obtain the corresponding retrieval result.

[0066] The present invention is further described in detail by acquiring multimodal data in a large-scale low-quality environment, taking low-quality image modality samples and low-quality text modality samples as examples.

[0067] Step 1 is as follows: the present invention first constructs a convolutional neural network VGG199 model with 13 convolutional layers, 3 fully connected layers and 5 pooling layers, and trains the VGG19 model. Then, the trained VGG19 model is used to extract features of low-quality image modality samples in the low-quality multimodal dataset. d1=512 is the feature dimension of the image modality, and n=10000 is the size of the low-quality multimodal data set. The bag-of-words model (BOW) is used to model the low-quality multimodal text data. The BOW model is trained by converting words into vectors with semantic characteristics. The trained BOW model is then used to extract features from the text modality samples in the low-quality multimodal data set to obtain low-quality text features. d2=1386 is the feature dimension of the text mode. Sample centering and RBF kernelization are performed on low-quality image features and low-quality text features respectively. In the RBF kernelization process of the present invention, 1000 sample anchor points are randomly selected in the image mode and the text mode, and then the processed image features are obtained. and text features

[0068] In step 2, the hash code of the multimodal sample is obtained, and the hash function corresponding to each modality is calculated, including:

[0069] Step 2-1: In order to fully explore the strong semantic discriminant hash codes corresponding to multimodal samples in large-scale low-quality environments, and to integrate the unique feature content and semantic consistency feature content between various modal data, a multimodal unsupervised hash learning global loss function is constructed:

[0070]

[0071] Where B∈{-1,1} r×n is the hash code to be learned, r is the hash code length, and the present invention sets r=64; P1 is the outer hash mapping matrix of the image modality, R1 is the inner hash mapping matrix of the image modality, P2 is the outer hash mapping matrix of the text modality, R2 is the inner hash mapping matrix of the text modality, θ is the multi-layer hash mapping balance parameter, δ is the sparse regularization parameter, tr(.) represents the trace operation of the matrix, and ||.|| represents the L 21 norm; U1 is the conversion matrix of the image mode, E1 is the unique matrix of the image mode, U2 is the conversion matrix of the text mode, E2 is the unique matrix of the text mode, β is the balance parameter, a is the multimodal unique feature parameter, and μ is the regularization parameter. The present invention sets μ=1e -2 ,δ=1e -2 ,θ=1e 0 , β=1e-1 , a=1e -1 .

[0072] Step 2-2: Use the alternating iterative optimization method to solve the values ​​of parameters R1, R2, P1, P2, U1, U2, E1, E2, and B, namely:

[0073] Step 2-2-1, when updating parameter R1, fix other parameters. The value of R1 is calculated as:

[0074]

[0075] Where M is the multi-level transition diagonal matrix of the image modality, and the value of the i-th row and i-th column in the diagonal matrix M is i=1,...,k1, k1 is the row dimension of P1, ||.||2 is the 2-norm of the matrix.

[0076] Step 2-2-2, when updating parameter R2, fix other parameters. The value of R2 is calculated as:

[0077]

[0078] Where N is the multi-level transition diagonal matrix of the text mode, and the value of the i-th row and i-th column in the diagonal matrix N is i=1,...,k2, k2 is the row dimension of P2.

[0079] Step 2-2-3, when updating parameter P1, fix other parameters. The value of P1 is calculated as:

[0080]

[0081] Step 2-2-4, when updating parameter P2, fix other parameters. The value of P2 is calculated as:

[0082]

[0083] Step 2-2-5, when updating parameter U1, fix other parameters. The value of U1 is calculated as:

[0084]

[0085] Step 2-2-5, when updating parameter U2, fix other parameters. The value of U2 is calculated as:

[0086]

[0087] Step 2-2-6, when updating parameter E1, fix other parameters. The value of E1 is calculated as:

[0088]

[0089] Step 2-2-7, when updating parameter E2, fix other parameters. The value of E2 is calculated as:

[0090]

[0091] Step 2-2-8, when updating parameter B, fix other parameters. The value of B is calculated as:

[0092]

[0093] Where sgn(.) is the sign function.

[0094] Step 2-3, further solve the hash function H1 of the image mode through the obtained parameters R1, R2, P1, P2, and its value is H1=P1R1; similarly, solve the hash function H2 of the text mode, and its value is H2=P2R2.

[0095] In step 3, the obtained query hash code is used to calculate the Hamming distance with the hash code in the multimodal data set, the calculated Hamming distances are sorted from small to large, and the first 100 multimodal data are selected as the retrieval result output.

[0096] The present invention further proposes a large-scale low-quality multimodal dynamic stream data retrieval system, which includes a dynamic stream data feature extractor module, a dynamic stream data hash function solver module and a dynamic stream data multimodal retriever module.

[0097] The system of the present invention is directed to a dynamic stream data feature extractor module, which is not limited to image modalities and text modalities, and can include multiple modalities such as voice modalities and video modalities, wherein the dynamic stream data feature extractor module utilizes a corresponding modality feature extractor to extract feature information of the corresponding modality. In order to facilitate the explanation of the present system, the current system has a dynamic image stream data modality and a dynamic text stream data modality, and the corresponding modality feature extractors are an image deep network model and a text deep network model, respectively, which extract dynamic image stream data sample feature information and dynamic text stream data sample feature information, and perform sample centering processing and RBF kernelization processing on the feature information, thereby obtaining the processed image feature. and text features

[0098] The dynamic streaming data hash function solver module adopts a one-step hash function solving process to effectively avoid the quantization error caused by the two-step hash function solving process. At the same time, combined with the modal feature information extracted by the dynamic streaming data feature extractor module, the multimodal unsupervised hashing method in a large-scale low-quality environment is expanded to obtain the corresponding dynamic streaming data hash function solver global loss function:

[0099]

[0100] In the formula, H s is the hash code of the dynamic stream data to be learned, is the unique matrix of the dynamic streaming data image modality, is a unique matrix of the dynamic streaming data text mode. The present invention sets θ=1e 0 ,δ=1e -2 , β=1e -1 , a=1e -1 , μ=1e -2 .

[0101] The alternating iterative optimization solution method is used to solve the global loss function R1, R2, P1, P2, U1, U2 of the dynamic stream data hash function solver. H s ,get:

[0102]

[0103] The parameters R1, R2, P1, and P2 of the global loss function of the multimodal unsupervised hash learning of dynamic flow data are used to further solve the hash function of the dynamic flow data image mode. Its value is Similarly, the hash function for solving the text mode of dynamic streaming data Its value is

[0104] Dynamic stream data multimodal retriever module: Through the dynamic stream data hash function solver module, the dynamic stream data hash function corresponding to each modality is obtained, and the corresponding dynamic stream data query hash code is further obtained. That is, if the dynamic stream data to be queried is a dynamic stream data image modality data Hash function using dynamic streaming data text modality Obtain the hash code of the image modal data of the dynamic stream data to be retrieved for:

[0105]

[0106] If the dynamic stream data sample to be queried is dynamic stream data text mode data Hash function using dynamic streaming data text modality Obtain the hash code of the image modal data of the dynamic stream data to be retrieved for:

[0107]

[0108] The Hamming distance between the calculated dynamic stream data query hash code and the hash code in the dynamic stream data multimodal data set is calculated, the calculated Hamming distances are sorted from small to large, and the top 100 multimodal data are selected as the retrieval result output.

[0109] In summary, the present invention can effectively mine the deep semantic consistency content of low-quality heterogeneous data, effectively reducing the impact of the diversity of topological distribution of low-quality heterogeneous data on the adaptability of the multimodal hash learning model; and further extending the method to a dynamic multimodal streaming data environment, it can gradually, orderly and stably mine the semantic information of dynamic streaming data, which not only enhances the learning discrimination of the hash function of low-quality dynamic multimodal streaming data, but also strikes a balance between time and space overhead, accelerates the retrieval rate of low-quality dynamic multimodal streaming data, and realizes rapid retrieval of large-scale low-quality dynamic multimodal streaming data.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. Large-scale low-quality multimodal unsupervised hashing method, characterized by: The following steps are involved: Step 1: Acquire large-scale multimodal data in a low-quality environment, extract sample feature information of the multimodal data, and perform sample centering and RBF kernelization processing on the obtained sample features to obtain processed multimodal data features; Specifically: The multimodal data includes low-quality image samples and low-quality text samples. The feature information of the low-quality image samples is extracted using an image deep network model to obtain low-quality image features S1, and the feature information of the low-quality text samples is extracted using a text deep network model to obtain low-quality text features S2; the low-quality image features and the low-quality text features are respectively subjected to sample centering processing and RBF kernelization processing to obtain processed image features and text features Step 2: Construct a sparse regularized hierarchical hash mapping model to mine the strong semantic discriminant hash codes corresponding to multimodal samples in a large-scale low-quality environment, and construct a multimodal matrix decomposition hash learning model to integrate the unique feature content and semantic consistency feature content between various modal data to obtain the multimodal unsupervised hash learning global loss function; solve the hash code of multimodal data and calculate the hash function corresponding to each modality; specifically: Step 2-1: In order to fully explore the strong semantic discriminant hash codes corresponding to multimodal samples in large-scale low-quality environments, a sparse regularized multi-layer hash mapping model is constructed: Where B is the hash code to be learned, r is the hash code length, n is the size of the multimodal dataset, P1 is the outer hash mapping matrix of the image modality, R1 is the inner hash mapping matrix of the image modality, P2 is the outer hash mapping matrix of the text modality, R2 is the inner hash mapping matrix of the text modality, θ is the multi-layer hash mapping balance parameter, δ is the sparse regularization parameter, tr(.) represents the trace operation of the matrix, and ||.|| represents the L of the matrix. 21 norm; Step 2-2: In order to integrate the unique feature content and semantic consistency feature content between various modal data, a multimodal matrix decomposition hash learning model is constructed: Where U1 is the conversion matrix of the image modality, E1 is the unique matrix of the image modality, U2 is the conversion matrix of the text modality, E2 is the unique matrix of the text modality, β is the balance parameter, a is the multimodal unique feature parameter, and μ is the regularization parameter; Step 2-3: Combine steps 2-1 and 2-2 to construct a global loss function for multimodal unsupervised hash learning: Step 2-4, use the alternating iterative optimization solution method to solve the values ​​of parameters R1, R2, P1, P2, U1, U2, E1, E2, and B; Step 2-5: Based on the parameters R1, R2, P1, and P2, further solve the hash function H1 of the image mode, and its value is H1=P1R1; similarly, solve the hash function H2 of the text mode, and its value is H2=P2R2; Step 3: Use the hash function corresponding to the query sample modality to perform hash coding mapping on the query sample to obtain the query hash code corresponding to the query sample; The obtained query hash code is solved by Hamming distance with the hash code in multimodal data in a large-scale low-quality environment to obtain the corresponding retrieval results.

2. The large-scale low-quality multimodal unsupervised hashing method according to claim 1, characterized in that: Steps 2-4 are as follows: Step 2-4-1, when updating parameter R1 and fixing other parameters, the value of R1 is calculated as: Where M is the multi-level transition diagonal matrix of the image modality, and the value of the i-th row and i-th column in the diagonal matrix M is k1 is the row dimension of P1, ||.||2 is the 2-norm of the matrix; Step 2-4-2, when the parameter R2 is updated and other parameters are fixed, the value of R2 is calculated as: Where N is the multi-level transition diagonal matrix of the text mode, and the value of the i-th row and i-th column in the diagonal matrix N is k2 is the row dimension of P2; Step 2-4-3, when updating parameter P1 and fixing other parameters, the value of P1 is calculated as: Step 2-4-4, when updating parameter P2 and fixing other parameters, the value of P2 is calculated as: Step 2-4-5, when updating parameter U1 and fixing other parameters, the value of U1 is calculated as: Step 2-4-6, when updating parameter U2 and fixing other parameters, the value of U2 is calculated as: Step 2-4-7, when updating parameter E1 and fixing other parameters, the value of E1 is calculated as: Step 2-4-8, when updating parameter E2 and fixing other parameters, the value of E2 is calculated as: Step 2-4-9, when updating parameter B and fixing other parameters, the value of B is calculated as: Where sgn(.) is the sign function.

3. The large-scale low-quality multimodal unsupervised hashing method according to claim 2, characterized in that: In step 3, the query sample is converted and mapped by using the hash function corresponding to the query sample modality to obtain the hash code corresponding to the query sample modality, specifically: Assume that when the query sample is image modality data According to the obtained hash function H1 of the image modality, the hash code B of the image modality data to be retrieved is obtained I_query for: The obtained query hash code is used to calculate the Hamming distance with the hash code in the multimodal data set, and the calculated Hamming distances are sorted from small to large, and the first s multimodal data are selected as the retrieval result output, where s is an integer greater than or equal to 1; Assume that when the query sample is text modal data According to the hash function H2 of the obtained text modality, the hash code B of the text modality data to be retrieved is obtained T_query for: The obtained query hash code is used to calculate the Hamming distance with the hash code in the multimodal data set, and the calculated Hamming distances are sorted from small to large. The first s multimodal data are selected as the retrieval result output, where s is an integer greater than or equal to 1.

4. A large-scale low-quality multimodal dynamic stream data retrieval system, characterized in that: For implementing the method described in any one of claims 1 to 3, the system comprises a dynamic stream data feature extractor module, a dynamic stream data hash function solver module and a dynamic stream data multimodal retriever module; The dynamic stream data feature extractor module uses the corresponding modality feature extractor to extract the feature information of the corresponding modality; The dynamic stream data hash function solver module adopts a one-step hash function solving process, and combines the modal feature information extracted by the dynamic stream data feature extractor module to obtain the corresponding dynamic stream data hash function solver global loss function; and solves the parameters of the dynamic stream data hash function solver global loss function, and then obtains the dynamic stream data hash functions corresponding to different modalities; The dynamic stream data multimodal retriever module further obtains the corresponding dynamic stream data query hash code through the dynamic stream data hash function corresponding to each modality, and performs Hamming distance calculation on the solved dynamic stream data query hash code and the hash code in the dynamic stream data multimodal data, sorts the solved Hamming distances from small to large, and selects the first s multimodal data as the retrieval result output.

Images