Discrete semantic association hashing method for cross-modal retrieval of multimedia data

Abstract

The present invention discloses a discrete semantic association hashing method for cross-modal retrieval of multimedia data, including obtaining historical data and processing to obtain a data training set; initializing parameters and obtaining a set of data; updating an image linear mapping matrix; updating a text linear mapping matrix; updating intermediate matrix variables; updating a hash code matrix; updating a hash code auxiliary matrix; updating the error between the hash code matrix and the hash code auxiliary matrix; replacing a set of data from a training set; repeating the above steps until the set conditions are met, and obtaining a final hash code matrix, an image linear mapping matrix, and a text linear mapping matrix. The present invention has a linear time training complexity and can achieve a good retrieval accuracy; and the time complexity and space complexity of the method of the present invention are both more excellent, with high reliability and good practicality.

Description

Technical Field

[0001] The present invention belongs to the field of data mining, and in particular relates to a discrete semantic association hashing method for cross-modal retrieval of multimedia data. Background Art

[0002] With the rapid development of information technology, the rapid iteration of data storage technology and the increasing popularity of smart devices, multimedia data has shown a rapid growth trend. As the main way for computers and smart devices to describe real things, multimedia data has the characteristics of large total volume and diverse types. For example, multimedia data can be expressed in various forms such as text, images, videos, audio and 3D models that are more common in life. Since the representation of things on computers is no longer single, people are no longer satisfied with searching on a single data type, but are more accustomed to and expect to use data of one modality to retrieve data of multiple modalities. Just like the search engines used in daily life, text can be used to retrieve results of multiple modalities such as text, images, and videos. However, in the face of a large amount of multimedia data with diverse types, how to generate indexes for it to achieve accurate and efficient cross-modal retrieval on large-scale databases has become a fundamental problem for the effective use of multimedia big data resources.

[0003] Since the potential semantic information in the corresponding category labels of things can guide the encoding process of cross-modal retrieval hashing tasks, supervised methods usually achieve better retrieval accuracy than unsupervised methods and continue to attract the attention of researchers in recent years. Some supervised methods use category labels to construct pairwise similarity matrices to better maintain the semantic association between data, but this technical approach has the following problems:

[0004] Technical Problem 1: Directly constructing a similarity matrix for n data during training will result in O(n 2 )’s time and space complexity makes it difficult to apply this method to similarity retrieval tasks in large-scale dataset scenarios, which runs counter to the efficient retrieval and storage efficiency of cross-modal hash retrieval technology.

[0005] Technical Problem 2: Using only category labels to construct a similarity matrix and use it to guide the hash code learning process will result in the loss of some semantic information in the original labels. Summary of the invention

[0006] The purpose of the present invention is to provide a discrete semantic association hashing method for cross-modal retrieval of multimedia data, which has better time complexity and space complexity, high reliability and good practicality.

[0007] The discrete semantic association hashing method for cross-modal retrieval of multimedia data provided by the present invention comprises the following steps:

[0008] S1. Obtain historical image data and corresponding text data and process them to obtain a data training set;

[0009] S2. Initialize parameters and obtain a set of data from the training set;

[0010] S3. Update the image linear mapping matrix;

[0011] S4. Update the text linear mapping matrix;

[0012] S5. Update the intermediate matrix variables;

[0013] S6. Update the hash code matrix;

[0014] S7. Update the hash code auxiliary matrix;

[0015] S8. Update the error between the hash code matrix and the hash code auxiliary matrix;

[0016] S9. Replace a set of data from the training set;

[0017] S10. Repeat steps S3 to S9 until the set conditions are met, thereby obtaining the final hash code matrix, image linear mapping matrix and text linear mapping matrix.

[0018] The discrete semantic association hashing method for cross-modal retrieval of multimedia data further includes the following steps:

[0019] S11. Use the hash code matrix, image linear mapping matrix and text linear mapping matrix obtained in step S10 to perform cross-modal retrieval of multimedia data.

[0020] The acquisition of historical image data and corresponding text data and processing them in step S1 to obtain a data training set specifically includes the following steps:

[0021] A. Obtain historical image data and corresponding text data, and simultaneously obtain the image feature matrix X of the historical image data and the text feature matrix Y of the text data, and simultaneously obtain the label matrix L and the hash code length r;

[0022] B. De-average the matrices X and Y obtained in step A and embed them in a nonlinear space to obtain kernel features φ(X) and φ(Y);

[0023] C. The label matrix L obtained in step A is normalized using the following formula to obtain the normalized label matrix

[0024]

[0025] In the formula is the normalized label matrix The element in the dth row and the ith column in L di is the element in the dth row and ith column in the label matrix L; c is the label class number.

[0026] The initialization parameters described in step S2 specifically include initializing the image linear mapping matrix P x , initialize the text linear mapping matrix P y , initialize the intermediate matrix variable W, initialize the hash code matrix B, initialize the hash code auxiliary matrix A, and initialize the error E between the hash code matrix and the hash code auxiliary matrix.

[0027] The updating of the image linear mapping matrix in step S3 is specifically to update the image linear mapping matrix P using the following formula: x :

[0028]

[0029] Where B is the hash code matrix; X is the image feature matrix; I is the identity matrix; λ is the weight parameter; α is the balance parameter between modes.

[0030] The updating of the text linear mapping matrix in step S4 is specifically to update the text linear mapping matrix P using the following formula: y :

[0031]

[0032] Where B is the hash code matrix; Y is the text feature matrix; I is the unit matrix; λ is the weight parameter; α is the balance parameter between modes.

[0033] The updating of the intermediate matrix variable in step S5 is specifically to update the intermediate matrix variable W using the following formula:

[0034] W=(βBB T +μI) -1 (βrBSL T +μBL T )(LL T ) -1

[0035] Where β is the weight parameter; B is the hash code matrix; μ is the weight parameter; I is the unit matrix; r is the hash code length; S is the semantic similarity matrix and is the normalized label matrix, 1 n is a column vector consisting of 1s; L is the label matrix.

[0036] The updating of the hash code matrix in step S6 is specifically to update the hash code matrix B using the following formula: B = sgn(2αP x X+2(1-α)P y Y+2βrWLS+2μWL-βWLL T W T A+ρA-E) where sgn(AA) is the sign function, and if AA>0, sgn(AA)=1, if AA=0, sgn(AA)=0, if AA<0, sgn(AA)=-1; α is the balance parameter between modes; P x is the image linear mapping matrix; X is the image feature matrix; P y is the text linear mapping matrix; Y is the text feature matrix; β is the weight parameter; r is the hash code length; W is the intermediate matrix variable; L is the label matrix; S is the semantic similarity matrix; μ is the weight parameter; A is the hash code auxiliary matrix; ρ is the non-negative constraint coefficient; E is the error between the hash code matrix and the hash code auxiliary matrix.

[0037] The updating of the hash code auxiliary matrix in step S7 is specifically to update the hash code auxiliary matrix A using the following formula:

[0038] A=sgn(-βWLL T W T B+ρB+E)

[0039] Where sgn(AA) is the sign function; β is the weight parameter; W is the intermediate matrix variable; L is the label matrix; B is the hash code matrix; ρ is the non-negative constraint coefficient; E is the error between the hash code matrix and the hash code auxiliary matrix.

[0040] The step S8 of updating the error between the hash code matrix and the hash code auxiliary matrix specifically uses the following formula to update the error E between the hash code matrix and the hash code auxiliary matrix:

[0041] E'=E+ρ(BA)

[0042] Where E' is the error between the updated hash code matrix and the hash code auxiliary matrix; E is the error between the hash code matrix and the hash code auxiliary matrix before the update; ρ is a non-negative constraint coefficient; B is the hash code matrix; A is the hash code auxiliary matrix.

[0043] The discrete semantic association hashing method for cross-modal retrieval of multimedia data provided by the present invention constructs a pairwise similarity matrix between data in a non-explicit manner through category labels, can maintain the similarity between and within the multimedia data modalities, and can obtain linear training complexity, and can be applicable to cross-modal retrieval in large-scale data set scenarios; the method of the present invention simultaneously adopts a linear regression simplification method of hash codes and category labels for modeling, and improves the discriminability of the hash codes to be generated; in addition, the method proposes a discrete optimization strategy based on the augmented Lagrange multiplier method, which reduces the quantization loss in the hash code generation process; therefore, the method of the present invention has a linear time training complexity and can obtain a good retrieval accuracy; and the time complexity and space complexity of the method of the present invention are both better, with high reliability and good practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the framework of the method of the present invention.

[0045] Figure 2 The figure is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0046] like Figure 1 Shown is a schematic diagram of the framework of the method of the present invention:

[0047] The idea of ​​the method of the present invention is as follows:

[0048] Assume that the feature matrices used to describe paired image and text data are respectively recorded as and The corresponding label matrix is ​​recorded as Where n represents the total number of data, d1 and d2 represent the dimensions of image and text feature vectors, c represents the number of categories, and L ij =1 means that the jth image and text data pair belongs to category i, otherwise L ij =0, it does not belong to. The main goal of the method of the present invention is to learn to generate hash codes B∈{-1,+1} for a given training set object. r×n , where r is the length of the hash code, and a hash function is generated for the out-of-sample data that can be compared with the learned hash code, and the Hamming distance between the hash codes is ensured to be close to the corresponding metric distance in the original feature space, that is, when two objects are similar, the Hamming distance between the corresponding hash codes will be as small as possible, and vice versa.

[0049] Based on the given category labels, the cosine similarity between label vectors can be used to define the similarity matrix S that describes the semantic association between data objects. Obviously, when two data objects o i With o j If they belong to the same category, then there is Sij =1, when two data objects belong to different categories S ij = 0. In multi-category scenarios, when the two objects have more common categories, S ij Then it tends to 1 as much as possible, otherwise S ij It will tend to 0 as much as possible. For two data objects o i With o j , its semantic similarity is defined as follows:

[0050]

[0051] As mentioned above, when two data objects o i With o j The more similar the data objects are, the closer the Hamming distance between the corresponding hash codes will be to 0, otherwise the closer the Hamming distance between the corresponding hash codes will be to the length r of the hash codes. The relationship between the similarity used to describe the semantic association between data objects and the Hamming distance between the corresponding hash codes is obvious, but choosing the Hamming distance to model the problem will make the optimization process very complicated. Therefore, in the present invention, DCMH-SC will choose the inner product between hash codes to model the problem. The corresponding relationship between the inner product of the hash code and the Hamming distance of the hash code is as follows:

[0052]

[0053] Among them, b i With b j Respectively represent data objects o i With o j The learned hash code, Hamm(b i ,b j ) represents the hash code b i With b j Obviously, the Hamming distance between hash codes corresponds to the inner product one by one. If two objects are more similar, the inner product of their corresponding hash codes tends to r, otherwise it tends to -r. Therefore, we only need to slightly change the definition of the similarity matrix to make it correspond to the inner product of the hash code. The vectors in the label matrix are normalized and preprocessed as follows:

[0054]

[0055] The final definition of the semantic similarity matrix is ​​as follows:

[0056]

[0057] Among them, 1 n represents a column vector consisting of 1s, and S ij∈[-1,1]. The semantic similarity matrix defined above is reconstructed using the inner product of the hash code to be learned to achieve similarity preservation, and the Frobenius norm is selected as the loss function of this process, resulting in the following formula:

[0058]

[0059] However, due to the discreteness of hash codes, it is very complicated to directly solve the above symmetric matrix decomposition problem, and storing the semantic similarity matrix will also bring The space complexity of the level. In order to solve this problem, the method of the present invention chooses to convert it into an asymmetric matrix decomposition problem that is easier to solve by replacing the matrix variables. Based on the idea that data objects belonging to the same category will generate the same hash code, there must be a certain connection between the category label matrix L of the training data and its corresponding hash code matrix B. The method of the present invention still uses linear regression from L to B to represent this association. By replacing the latter B in the above formula with the corresponding linear association WL, it can not only simplify the subsequent optimization process, but also further ensure that data objects of the same category will learn to obtain the same hash code while preserving the paired semantic association of the objects, making the hash code more discriminative. The above process can be formulated as follows:

[0060]

[0061] stB∈{-1,1} r×n

[0062] Where μ is the weight parameter. As shown in the subsequent optimization process, the above formula avoids the difficult-to-solve discrete symmetric matrix decomposition problem, and the similarity matrix S does not need to be explicitly calculated and stored. It can be replaced by the preprocessed label matrix, thereby reducing the time and space complexity by Reduce to It can effectively reduce the training cost in large-scale data set scenarios.

[0063] In addition, the method of the present invention needs to learn a modality-specific hash function for each modality of the data to generate a unified and easily measurable hash code for the out-of-sample data. The method of the present invention selects a commonly used linear mapping matrix as the hash function of the image and the method of the present invention to reduce the complexity of the algorithm. In order to ensure the consistency between the hash code generated by the hash function and the hash code of the training data, and to minimize the error between the mapping of the hash code of the training data to the original data, the hash function learning can be expressed as follows:

[0064]

[0065] stB∈{-1,1} r×n

[0066] Among them, α is the balance parameter between the two modes, λ is the weight parameter, and They represent the unique linear mapping matrices of image and text respectively, which serve as the main components of the hash function in the retrieval stage, while R(·) represents the regularization term, which is used to prevent overfitting.

[0067] By combining the above two parts into a unified framework, the overall objective function of the method of the present invention can be expressed as follows:

[0068]

[0069] stB∈{-1,1} r×n

[0070] The optimization problem of the objective function of the method of the present invention is for all matrix variables P x , P y , W and B is a non-convex problem and is difficult to solve directly. Therefore, the method of the present invention chooses to adopt a method of fixing three of the matrix variables and solving the remaining single matrix variables, thereby converting the optimization problem into a convex problem. Therefore, each matrix variable can be solved iteratively to finally obtain the local optimal solution of the objective function. The specific optimization solution process is as follows:

[0071] (1) Update the matrix variable P x :By fixing the matrix variable P y , W and B, the overall objective function is rewritten as follows:

[0072]

[0073] So that the above formula is x The partial derivative of is 0, so we get P x The closed form solution of is as follows:

[0074]

[0075] (2) Update the matrix variable P y : With P x Similarly, we can get P y The closed form solution of is as follows:

[0076]

[0077] (3) Update the matrix variable W: by fixing the matrix variable P x , P y and B, the overall objective function can be rewritten as follows:

[0078]

[0079] The partial derivative of the above formula with respect to W is 0, so the closed solution of W is as follows:

[0080] W=(βBB T +μI) -1 (βrBSL T +μBL T )(LL T ) -1

[0081] It can be noted that in this step, there is a term BSL T contains the matrix variable S. If S is calculated in advance, it will result in However, in the optimization process, according to the formula The definition of can be obtained as follows:

[0082]

[0083] Obviously, the above formula reduces the training complexity to It will be more suitable for large-scale data set scenarios.

[0084] (4) Update the matrix variable B: By fixing the matrix variable P x , P y And W, the overall objective function can be rewritten as follows:

[0085]

[0086] stB∈{-1,1} r×n

[0087] Then rewrite the above formula as follows:

[0088]

[0089] stB∈{-1,1} r×n

[0090] Similarly, the discrete cyclic coordinate descent method can be used to iteratively solve each row of the hash code B for the above equation, but in the scenario of large-scale data sets, this still brings some time overhead. Therefore, based on the augmented Lagrange multiplier method, the method of the present invention introduces auxiliary matrix variables to transform the above equation into an equivalent but easier to solve problem. Specifically, by introducing the auxiliary variable A, B = A, so that the above equation can be rewritten as follows:

[0091]

[0092] stB∈{-1,1} r×n

[0093] where ρ is a non-negative constraint coefficient, represents the error between the hash code matrix B and the auxiliary matrix A. By fixing the matrix variables A and E, the above formula can be rewritten as follows:

[0094]

[0095] stB∈{-1,1} r×n

[0096] Obviously, the closed solution of B can be directly obtained as follows:

[0097] B=sgn(2αP x X+2(1-α)P y Y+2βrWLS+2μWL-βWLL T W T A+ρA-E)

[0098] It can be seen that without ignoring the discrete characteristics of the hash code, B can be solved in a single step, which will further improve the learning efficiency and retrieval accuracy of the algorithm. Similarly, the term WLS in the above formula can be replaced by the following formula to ensure better time complexity and space complexity:

[0099]

[0100] (5) Update the matrix variable A: By fixing the matrix variables B and E, we can rewrite:

[0101]

[0102] Then rewrite the above formula as:

[0103]

[0104] Similarly, the closed-form solution of A can be directly obtained as follows:

[0105] A=sgn(-βWLL T W T B+ρB+E)

[0106] (6) Update the matrix variable E: By fixing the matrix variables B and A, we can rewrite:

[0107]

[0108] The partial derivative of the above formula with respect to E is made 0, so that the closed solution of E is obtained as follows:

[0109] E=E+ρ(BA)

[0110] By iteratively using the above formula to solve each matrix variable, the value of the overall objective function is continuously reduced and converged to the local optimal solution, and finally the hash code of the training data and the hash function specific to each mode are learned.

[0111] Therefore, in summary, the specific implementation steps of the method of the present invention are as follows (eg Figure 2 shown):

[0112] S1. Obtain historical image data and corresponding text data and process them to obtain a data training set; specifically, the steps include:

[0113] A. Obtain historical image data and corresponding text data, and simultaneously obtain the image feature matrix X of the historical image data and the text feature matrix Y of the text data, and simultaneously obtain the label matrix L and the hash code length r;

[0114] B. De-average the matrices X and Y obtained in step A and embed them in a nonlinear space to obtain kernel features φ(X) and φ(Y);

[0115] C. The label matrix L obtained in step A is normalized using the following formula to obtain the normalized label matrix

[0116]

[0117] In the formula is the normalized label matrix The element in the dth row and the ith column in L di is the element in the dth row and the ith column in the label matrix L; c is the label class number;

[0118] S2. Initialize parameters and obtain a set of data from the training set; specifically, initialize the image linear mapping matrix P x , initialize the text linear mapping matrix P y , initialize the intermediate matrix variable W, initialize the hash code matrix B, initialize the hash code auxiliary matrix A, and initialize the error E between the hash code matrix and the hash code auxiliary matrix;

[0119] S3. Update the image linear mapping matrix; Specifically, the image linear mapping matrix P is updated using the following formula x :

[0120]

[0121] Where B is the hash code matrix; X is the image feature matrix; I is the identity matrix; λ is the weight parameter; α is the balance parameter between the modes;

[0122] S4. Update the text linear mapping matrix; Specifically, the text linear mapping matrix P is updated using the following formula y :

[0123]

[0124] Where B is the hash code matrix; Y is the text feature matrix; I is the unit matrix; λ is the weight parameter; α is the balance parameter between modes;

[0125] S5. Update the intermediate matrix variable; specifically, update the intermediate matrix variable W using the following formula:

[0126] W=(βBB T +μI) -1 (βrBSL T +μBL T )(LL T ) -1

[0127] Where β is the weight parameter; B is the hash code matrix; μ is the weight parameter; I is the unit matrix; r is the hash code length; S is the semantic similarity matrix and is the normalized label matrix, 1 n is a column vector consisting of 1; L is the label matrix;

[0128] S6. Update the hash code matrix; specifically, update the hash code matrix B using the following formula:

[0129] B=sgn(2αP x X+2(1-α)P y Y+2βrWLS+2μWL-βWLL T W T A+ρA-E)

[0130] Where sgn(AA) is a sign function, and if AA>0, sgn(AA)=1, if AA=0, sgn(AA)=0, if AA<0, sgn(AA)=-1; α is a balance parameter between modes; P x is the image linear mapping matrix; X is the image feature matrix; P y is the text linear mapping matrix; Y is the text feature matrix; β is the weight parameter; r is the hash code length; W is the intermediate matrix variable; L is the label matrix; S is the semantic similarity matrix; μ is the weight parameter; A is the hash code auxiliary matrix; ρ is the non-negative constraint coefficient; E is the error between the hash code matrix and the hash code auxiliary matrix;

[0131] S7. Update the hash code auxiliary matrix; specifically, update the hash code auxiliary matrix A using the following formula:

[0132] A=sgn(-βWLL T W T B+ρB+E)

[0133] Where sgn(AA) is the sign function; β is the weight parameter; W is the intermediate matrix variable; L is the label matrix; B is the hash code matrix; ρ is the non-negative constraint coefficient; E is the error between the hash code matrix and the hash code auxiliary matrix

[0134] S8. Update the error between the hash code matrix and the hash code auxiliary matrix; specifically, update the error E between the hash code matrix and the hash code auxiliary matrix using the following formula:

[0135] E'=E+ρ(BA)

[0136] Where E' is the error between the updated hash code matrix and the hash code auxiliary matrix; E is the error between the hash code matrix and the hash code auxiliary matrix before the update; ρ is a non-negative constraint coefficient; B is the hash code matrix; A is the hash code auxiliary matrix;

[0137] S9. Replace a set of data from the training set;

[0138] S10. Repeat steps S3 to S9 until the set conditions are met, thereby obtaining the final hash code matrix, image linear mapping matrix and text linear mapping matrix;

[0139] S11. Use the hash code matrix, image linear mapping matrix and text linear mapping matrix obtained in step S10 to perform cross-modal retrieval of multimedia data.

[0140] Finally, the discrete semantic association hashing method for cross-modal retrieval of multimedia data proposed in the present invention can also perform nonlinear expansion, out-of-sample data expansion and multimodal expansion:

[0141] Non-linear expansion:

[0142] The method of the present invention uses a linear mapping matrix to embed the data object from its original feature space into a common Hamming space. In order to make the algorithm more adaptable to actual scenarios, a kernel function can be used to capture the nonlinear structure in the original features of different modal data, and the kernel features can be used to replace the original features in the optimization process to obtain more accurate retrieval accuracy. In the experimental part, the method of the present invention uses the radial basis function (RBF) as the kernel mapping to obtain the kernel features of images and texts. Taking image data as an example, the value of the i-th dimension of the kernel feature of the object x to be generated with a dimension of p is calculated by the following formula:

[0143] φ i (x) = exp(-||xa i || 2 / (2σ 2 ))

[0144] In the formula represents p anchor points randomly selected from the training data, σ represents the kernel width; || || is the 1-norm;

[0145] Out-of-sample data expansion:

[0146] One of the main goals of the method of the present invention is to learn a modality-specific hash function for out-of-sample data to generate a unified hash code, thereby avoiding the time and space overhead caused by multiple training. Taking image data as an example, for a new query object Its corresponding hash code The trained mapping matrix P can be used directly x Directly generate. Therefore, the hash function for out-of-sample data expansion of the image can be obtained as follows:

[0147]

[0148] Similarly, the hash function for text data is as follows:

[0149]

[0150] Multimodal extensions:

[0151] Without loss of generality, the discussion of the present invention is limited to the two modal scenarios of image and text, but the method of the present invention is also applicable to scenarios with three or more modalities. The overall objective function of the method of the present invention can be expressed as follows:

[0152]

[0153] stB∈{-1,1} r×n

[0154] Where α m Used to balance the weights between the modes; similarly, the steps of the present invention are also applicable to the optimization solution of the objective function under multi-modal expansion.

Claims

1. A discrete semantic association hashing method for cross-modal retrieval of multimedia data, comprising the following steps: S1. Obtain historical image data and corresponding text data and process them to obtain a data training set; S2. Initialize parameters and obtain a set of data from the training set; S3. Update the image linear mapping matrix; S4. Update the text linear mapping matrix; S5. Update the intermediate matrix variables; S6. Update the hash code matrix; specifically, update the hash code matrix B using the following formula: B=sgn(2αPxX+2(1-α)PyY+2βrWLS+2μWL-βWLL T W T A+ρA-E); where sgn(AA) is the sign function, and if AA>0, then sgn(AA)=1, if AA=0, then sgn(AA)=0, if AA<0, then sgn(AA)=-1; α is the balance parameter between modes; P x is the image linear mapping matrix; X is the image feature matrix; P y is the text linear mapping matrix; Y is the text feature matrix; β is the weight parameter; r is the hash code length; W is the intermediate matrix variable; L is the label matrix; S is the semantic similarity matrix; μ is the weight parameter; A is the hash code auxiliary matrix; ρ is the non-negative constraint coefficient; E is the error between the hash code matrix and the hash code auxiliary matrix; S7. Update the hash code auxiliary matrix; S8. Update the error between the hash code matrix and the hash code auxiliary matrix; S9. Replace a set of data from the training set; S10. Repeat steps S3 to S9 until the set conditions are met, thereby obtaining the final hash code matrix, image linear mapping matrix and text linear mapping matrix; S11. Use the hash code matrix, image linear mapping matrix and text linear mapping matrix obtained in step S10 to perform cross-modal retrieval of multimedia data.

2. The discrete semantic association hashing method for cross-modal retrieval of multimedia data according to claim 1 is characterized in that The acquisition of historical image data and corresponding text data and processing them in step S1 to obtain a data training set specifically includes the following steps: A. Obtain historical image data and corresponding text data, and simultaneously obtain the image feature matrix X of the historical image data and the text feature matrix Y of the text data, and simultaneously obtain the label matrix L and the hash code length r; B. De-average the matrices X and Y obtained in step A and embed them in a nonlinear space to obtain kernel features φ(X) and φ(Y); C. The label matrix L obtained in step A is normalized using the following formula to obtain the normalized label matrix In the formula is the normalized label matrix The element in the dth row and the ith column in L di is the element in the dth row and the ith column in the label matrix L; c is the label class number.

3. The discrete semantic association hashing method for cross-modal retrieval of multimedia data according to claim 2 is characterized in that The initialization parameters described in step S2 specifically include initializing the image linear mapping matrix P x , initialize the text linear mapping matrix P y , initialize the intermediate matrix variable W, initialize the hash code matrix B, initialize the hash code auxiliary matrix A, and initialize the error E between the hash code matrix and the hash code auxiliary matrix.

4. The discrete semantic association hashing method for cross-modal retrieval of multimedia data according to claim 3 is characterized in that The updating of the image linear mapping matrix in step S3 is specifically to update the image linear mapping matrix P using the following formula: x : Where B is the hash code matrix; X is the image feature matrix; I is the identity matrix; λ is the weight parameter; α is the balance parameter between modes.

5. The discrete semantic association hashing method for cross-modal retrieval of multimedia data according to claim 4 is characterized in that The updating of the text linear mapping matrix in step S4 is specifically to update the text linear mapping matrix P using the following formula: y : Where B is the hash code matrix; Y is the text feature matrix; I is the unit matrix; λ is the weight parameter; α is the balance parameter between modes.

6. The discrete semantic association hashing method for cross-modal retrieval of multimedia data according to claim 5 is characterized in that The updating of the intermediate matrix variable in step S5 is specifically to update the intermediate matrix variable W using the following formula: W=(βBB T +μI) -1 (βrBSL T +μBL T )(LL T ) -1 Where β is the weight parameter; B is the hash code matrix; μ is the weight parameter; I is the unit matrix; r is the hash code length; S is the semantic similarity matrix and is the normalized label matrix, 1 n is a column vector consisting of 1s; L is the label matrix.

7. The discrete semantic association hashing method for cross-modal retrieval of multimedia data according to claim 1 is characterized in that The updating of the hash code auxiliary matrix in step S7 is specifically to update the hash code auxiliary matrix A using the following formula: A=sgn(-βWLL T W T B+ρB+E) Where sgn(AA) is the sign function; β is the weight parameter; W is the intermediate matrix variable; L is the label matrix; B is the hash code matrix; ρ is the non-negative constraint coefficient; E is the error between the hash code matrix and the hash code auxiliary matrix; The step S8 of updating the error between the hash code matrix and the hash code auxiliary matrix specifically uses the following formula to update the error E between the hash code matrix and the hash code auxiliary matrix: E'=E+ρ(BA) Where E' is the error between the updated hash code matrix and the hash code auxiliary matrix; E is the error between the hash code matrix and the hash code auxiliary matrix before the update; ρ is a non-negative constraint coefficient; B is the hash code matrix; A is the hash code auxiliary matrix.

Images