A cross-modal retrieval method based on online multi-hash code federated learning

Abstract

The application discloses a cross-modal retrieval method based on online multi-hash code joint learning. The method comprises the following steps: obtaining a cross-modal data set, and dividing a training set in the data set into t rounds of data blocks to simulate streaming data; a target function is constructed, the training data set divided into t rounds is used to simulate streaming data to train the target function, when the tth round of data block arrives, the different length hash codes trained by the first t-1 rounds of data blocks are saved unchanged, the different length hash codes of the tth round of data block are learned and saved, and the hash function is updated online according to real-time new data of the tth round of data block; cross-modal retrieval of a sample to be retrieved is performed; in the process of model training, only the optimization result of the previous round needs to be stored, so that different length hash codes are generated for real-time new data without retraining old data, the flexibility of hash code learning is increased, the memory consumption of the system is reduced, and the retrieval efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a cross-modal retrieval method in an online scene, in particular to a cross-modal retrieval method based on online multi-hash code joint learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] In the field of big data, multimedia data is growing explosively. Therefore, how to effectively retrieve the required data information has become a problem of great concern. Since the binary hash code is a relatively short bit number, it can directly perform XOR operation to calculate the similarity between data. Therefore, the cross-modal hash method encodes different modal data into compact binary hash codes, and then uses their distances in Hamming space to measure the similarity between modes. This method greatly reduces the storage requirement. However, most cross-modal hash methods are based on batch processing, and for the growing streaming multi-modal data, the batch-based hash method needs to be retrained for all data points to learn the hash code of new data points. When new streaming data arrives frequently, it will bring unacceptable computational and memory costs. Therefore, the batch-based hash method is not suitable for processing streaming data. The online hash method updates the hash function and hash code while the new streaming data appears, and also maintains the effectiveness of the binary code of the old streaming data. Therefore, the online hash method can quickly and effectively retrieve the streaming data. The existing online cross-modal hash method still has some limitations: (1) In order to obtain hash codes of different lengths, the model needs to be retrained. Obviously, this greatly reduces the flexibility of the hash method in practical applications. (2) For the learning of hash codes of different lengths, the semantic information contained in the longer length hash code is not fully utilized. (3) The consistency of multi-modal data and the modality-specific properties are not considered at the same time, which may affect the retrieval accuracy. (4) By constructing a similarity matrix with a size of one square of training data to save the supervision information, it causes a great time and space consumption for large data, especially for streaming data. SUMMARY

[0004] In view of the above challenges, the present application provides a cross-modal retrieval method based on online multi-hash code joint learning. By simultaneously learning hash codes of different lengths in a unified framework without retraining, the present application can increase the flexibility of hash codes while achieving better performance.

[0005] In order to achieve the purpose of the present application, the technical scheme of the cross-modal retrieval method based on online multi-hash code joint learning provided by the present application is as follows:

[0006] Step1, obtain a cross-modal dataset, divide the training set in the dataset into t rounds of data blocks to simulate streaming data; wherein each round of data block contains sample features of different modalities and corresponding semantic labels;

[0007] Step2, construct a target function learning, use the training dataset divided into t rounds to simulate streaming data to train the target function; when the tth round of data block arrives, save the different length hash codes trained by the first t-1 rounds of data blocks, learn the different length hash codes of the tth round of data block and store them in the retrieval library, and update the hash function online according to the real-time new data of the tth round of data block;

[0008] Step3, cross-modal retrieval of the sample to be retrieved: first obtain the hash code corresponding to the retrieval set, input the query sample, obtain the hash code of the query sample according to the hash function obtained in the last step, substitute the hash code of the query sample into the retrieval set for query, and return the required retrieval sample of another modality with close distance to the sample to be searched by calculating the Hamming distance between the query set and each sample in the retrieval set.

[0009] In the process of model training, only the optimization result of the previous round needs to be stored, so that different length hash codes are generated for real-time new data without retraining old data, the flexibility of hash code learning is increased, the memory consumption of the system is reduced, and the retrieval efficiency is improved.

[0010] Further, the sample features of different modalities of the online cross-modal retrieval dataset at least include: sample features of image text modalities and sample features of text modalities.

[0011] Further, in Step1, before the data enters the model training, a random block method should be used to process and divide the training set to simulate streaming data, so as to ensure the generalization ability of the model.

[0012] Further, the specific process of the target function obtained in Step2 includes: mapping the original sample to a high-dimensional feature space, and projecting the multi-modal data features to a discrete common representation as the uniform hash code of the training instance through a consistent and modality-specific projection matrix. The modality-specific projection matrix is subjected to sparse constraint based on l 2,1 norm, and the label matrix is embedded into the target function as supervision information. In addition, different length hash codes are semantically associated with the longest length hash code to fully explore the learned hash codes.

[0013] Furthermore, the objective function in Step 2 includes two parts: hash code learning and hash function learning. Hash code learning refers to learning by combining semantic labels and the semantic relationship between hash codes of different lengths and the longest hash code. Hash function learning refers to using a simple linear regression model as the hash function for each modality, and projecting the features of the training samples onto the hash code through a consistent and modality-specific projection matrix. By putting hash code learning and hash function learning into the same model, the semantic relationship between hash code and hash function is enhanced, thereby achieving high-precision online cross-modal retrieval.

[0014] Furthermore, the objective function in Step 2 includes two data components: the data newly arriving in round t and the old data blocks accumulated in the previous t-1 rounds.

[0015] Furthermore, the objective function in Step 2 is not optimal and needs to be optimized. Since the objective function is a non-convex problem, when other variables are fixed and a matrix variable is updated, the function becomes a convex problem, which facilitates updating the objective function. This alternating iterative algorithm is used to update the matrix variable until the objective function converges or reaches the maximum number of iterations, and finally the optimal hash code and hash function are obtained.

[0016] Furthermore, in Step 2, each data block includes two modalities: image and text, along with corresponding labels; during the t-th training round, the data blocks are labeled with class names. Pairs of data blocks Added to the training set; where and Let n represent the image feature matrix of dimension d1 and the text feature matrix of dimension d2, respectively. t c and c represent the new image-text data block size and the number of categories, respectively; in round t, the samples are divided into the old training samples accumulated from the previous t-1 rounds. and the new training sample X in the tth iteration (t) ; Include There are 10 sample pairs, and their corresponding label matrices are: Therefore, the total training dataset in round t is denoted as Include There are 1 data points, and the corresponding total label matrix is ​​denoted as .

[0017] Furthermore, the specific steps in Step 2 include:

[0018] Step 2.1, Multi-Hash Code Joint Learning

[0019] In the unified training model, multiple hash codes of different lengths are simultaneously learned, and the scalability of the hash length is increased; in addition, in order to explore and learn the hash code, the longest hash code is used for learning other hash codes of different lengths:

[0020]

[0021] wherein, denotes the Frobenius square of the matrix, and denote the kth hash code of the tth round data block and the first t-1 round data block (1≤k≤K), K is the number of hash codes, and denote the longest length hash code of the tth round data block and the first t-1 round data block, is a projection matrix, r k and r K denote the length of B k and B K , and α and γ are balance parameters;

[0022] 2.2, label embedding

[0023] In order to make full use of semantic label information, the supervised label information is regressed to the hash code, so that the learned hash code maintains the consistency of the semantic label; it is expressed by the following formula:

[0024]

[0025] wherein, is a projection matrix, and β is a balance parameter;

[0026] 2.3, hash function learning

[0027] The original multi-modal data nonlinear features are directly projected to discrete common representations as unified hash codes of training instances through a projection matrix; in order to make the learned hash code be able to retain the consistency and the modal-specific properties of the multi-modal data, the projection matrix is decomposed into a consistent and a modal-specific matrix; the inconsistency of the image and text modal is considered as a special kind of noise, so a sparse constraint based on l 2,1 norm is imposed on the modal-specific projection matrix; the hash function in the application is learned through a linear regression problem:

[0028]

[0029] wherein and are consistent and modal-specific projection matrices; μ, λ and γ represent the balance parameters for controlling the corresponding contribution terms; is the kernel feature matrix of image or text, which aims to map the original sample to a high-dimensional feature space, and is suitable for processing linearly inseparable problems; here, φ(.) represents the RBF kernel function, and m represents the number of anchor points; therefore, the kernel feature φ(x i ) is defined as follows:

[0030]

[0031] wherein represents m anchor points; is a Gaussian kernel parameter.

[0032] 2.4, objective function

[0033] After the above steps, the objective function is:

[0034]

[0035] 2.5, iterative update of the objective function: the objective function obtained in the last step is updated by iteration until the objective function converges or the maximum number of iterations is reached, and the hash function and the hash code of the training set are obtained.

[0036] Further, the optimization step of the objective function in Step 2 is as follows:

[0037] 2.5.1, update C k(t) : fix other variables, for C k(t) , the objective function is simplified as:

[0038]

[0039] Set , the closed solution of C k(t) is derived as follows:

[0040]

[0041] wherein and are represented as:

[0042]

[0043]

[0044]

[0045]

[0046] wherein and are obtained in the t-1th round, from and The representation of the observation is denoted as: and The former part is related to the newly arrived data in the t-th round, and the latter part is related to the old data accumulated in the previous t-1 rounds; therefore and The C is calculated in an incremental manner, k(t) updated by online learning, similarly, W k(t) , and R k(t) are updated by online learning.

[0047] 2.5.2, update Fixing other variables, for C k(t) , the objective function is simplified as:

[0048]

[0049] To solve the optimization problem based on the l 2,1 norm, the following weight matrix is introduced:

[0050]

[0051] where is the j-th diagonal element of D (i) , and is the j-th row of the matrix , therefore, the above objective function is rewritten as:

[0052]

[0053] The update of is denoted as:

[0054]

[0055] 2.5.3, update W k(t) : Similar to the update step of C k(t) , the closed-form solution of W k(t) is obtained as follows:

[0056]

[0057]

[0058]

[0059] 2.5.4, update R k(t) (1≤k≤K-1): Similar to the update step of C k(t) , the closed-form solution of R k(t) is obtained as follows:

[0060]

[0061]

[0062]

[0063] 2.5.5, updating B k(t) , fixing other variables, for B k(t) , the objective function is simplified as:

[0064]

[0065] From the above expression, it is observed that when k=K, i.e., only one length of hash code is learned, the constraint term is meaningless, thus B k(t) is updated case by case; when k=K, B k(t) is obtained by the following update:

[0066]

[0067] When 1≤k<K, i.e., multiple lengths of hash codes are learned, the solution of B k(t) is easily obtained:

[0068]

[0069] where sgn(·) is the sign function;

[0070] The objective function is optimized by the above steps until the objective function converges or the maximum number of iterations is reached, and the iteration is stopped.

[0071] The beneficial effects of the present application are:

[0072] The present application proposes a cross-modal retrieval method based on online multi-hash code joint learning, referred to as OMCJL. It uses an effective multi-modal mapping learning strategy, decomposes the projection matrix into consistent and modality-specific projection matrices, so that the resulting hash code retains the consistent and specific properties of multi-modal data. At the same time, sparse constraints are imposed on the modality-specific projection matrix. In addition, an online learning strategy for hash codes is proposed, which can simultaneously learn multiple hash codes of different lengths in one online model training, and the longest hash code is used for learning other hash codes of different lengths, which can further explore and learn hash codes. At the same time, in order to improve the retrieval performance, semantic labels are used to guide the learning of hash codes. The present application only needs to store the optimization results of the previous round during model training, thereby realizing the generation of hash codes of different lengths for real-time new data without retraining old data, increasing the flexibility of hash code learning, reducing the memory consumption of the system, and improving the retrieval efficiency. Attached Figure Description

[0073] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention.

[0074] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0075] The following descriptions are illustrative and are intended to further illustrate the technical solutions of the present invention in conjunction with the accompanying drawings.

[0076] Example 1, Figure 1 This is a flowchart of a cross-modal retrieval method based on online multi-hash code joint learning according to the present invention.

[0077] In this example, see Figure 1 The method of the present invention specifically includes the following processes:

[0078] 1. Obtain a cross-modal dataset and divide the training set into t rounds of data blocks to simulate streaming data. In this example, each round of data blocks includes two modalities: image and text, along with corresponding labels.

[0079] In this invention, during the t-th training round, the data is labeled with class tags. Pairs of data blocks It was added to the training set. and Let n represent the image feature matrix of dimension d1 and the text feature matrix of dimension d2, respectively. t Let c and denot be the new image-text data block size and the number of categories, respectively. In round t, the samples can be divided into the old training samples accumulated from the previous t-1 rounds. and the new training sample X in the tth iteration (t) . Include There are 10 sample pairs, and their corresponding label matrices are: Therefore, the total training dataset in round t is denoted as Include There are 1 data points, and the corresponding total label matrix is ​​denoted as .

[0080] 2. Construct a target function learning mechanism. Use a training dataset divided into t rounds to simulate streaming data to train the target function. When the data block of the t-th round arrives, keep the hash codes of different lengths obtained from the training of the data blocks of the previous t-1 rounds unchanged, learn the hash codes of different lengths of the data block of the t-th round and save them, and update the hash function online according to the real-time new data of the data block of the t-th round.

[0081] The objective function includes two parts of hash code learning and hash function learning. The hash code learning refers to learning in combination with semantic labels and semantic relationship between hash codes of different lengths and hash codes of the longest length. The hash function learning refers to adopting a simple linear regression model as a hash function for each modality, projecting features of training samples into hash codes through consistent and modality-specific projection matrices, and learning by putting the hash code learning and the hash function learning into the same model, so as to enhance the semantic connection between hash codes and hash functions. In addition, the objective function further includes two parts of newly arrived data of the tth round and accumulated old data blocks of the first t-1 rounds. Next, the learning model of each module will be described in detail.

[0082] 2.1, multi-hash code joint learning

[0083] Generally, before learning binary codes, a fixed hash length (for example, 16 bits or 32 bits) is defined in advance. When the hash code length changes, these models need to be retrained. Therefore, multiple hash codes of different lengths are simultaneously learned in the unified training model, which increases the scalability of the hash length. In addition, longer hash codes contain richer semantics, and in order to further explore and learn hash codes, the longest hash code is used for learning hash codes of other different lengths in the present application:

[0084]

[0085] wherein, represents the Frobenius square of a matrix, and respectively represent the kth hash code (1≤k≤K) of the tth round data block and the first t-1 round data block, K is the number of hash codes, and respectively represent the longest length hash code of the tth round data block and the first t-1 round data block, is a projection matrix, r k and r K respectively represent the length of B k and B K , and α and γ are balance parameters.

[0086] 2.2, label embedding

[0087] In order to make full use of semantic label information, the learned hash codes can maintain the consistency of semantic labels by further regressing the supervised label information to the hash codes. It can be represented by the following formula:

[0088]

[0089] wherein, is the projection matrix, and β is the balance parameter.

[0090] 2.3 Hash function learning

[0091] To avoid large quantization error, the present application directly projects the original multi-modal data nonlinear features to discrete common representation as uniform hash codes of training instances by projection matrix. In order to make the learned hash codes to preserve the consistency and the modal-specific properties of multi-modal data, the projection matrix is decomposed into consistent and modal-specific matrices. The inconsistency of image and text modalities can be considered as a special kind of noise, so in this instance, the modal-specific projection matrix is imposed with sparse constraint based on l 2,1 norm. The hash function in the present application is learned by linear regression problem:

[0092]

[0093] where and are the consistent and modal-specific projection matrices respectively. μ, λ and γ represent the trade-off parameters that control the respective contribution terms. is the kernel feature matrix of image or text, which aims to map the original samples to high-dimensional feature space, suitable for processing linearly inseparable problems. Here, φ(.) represents the RBF kernel function, and m represents the number of anchor points. Therefore, the kernel feature φ(x i ) is defined as follows:

[0094]

[0095] where represents m anchor points. is the Gaussian kernel parameter.

[0096] 2.4 Objective function

[0097] The objective function is as follows:

[0098]

[0099] Step 3, iterative update of the objective function: the objective function obtained in the last step is updated by iteration until the objective function converges or reaches the maximum number of iterations, and the hash function and the hash code of the training set are obtained.

[0100] The objective function is not optimal, and needs to be iteratively updated next. Obviously, the overall objective function is a non-convex optimization problem. Therefore, an effective alternating iterative algorithm is proposed to solve this problem.

[0101] The optimization steps of the objective function are as follows:

[0102] 3.1, Update C k(t) : Fixing other variables, for C k(t) , the objective function can be simplified as:

[0103]

[0104] Setting , the closed-form solution of C k(t) can be derived. As follows:

[0105]

[0106] where and can be represented as:

[0107]

[0108]

[0109]

[0110]

[0111] where and are obtained in the t-1th round, from and , it can be observed that: and The former part is related to the newly arrived data in the tth round, and the latter part is related to the old data accumulated in the first t-1 rounds. Therefore and can be calculated in an incremental manner, C k(t) can be updated by online learning. Similarly, W k(t) , and R k(t) can also be updated by online learning.

[0112] 3.2, Update Fixing other variables, for C k(t) , the objective function can be simplified as:

[0113]

[0114] To solve the optimization problem based on the l 2,1 norm, the following weight matrix is introduced:

[0115]

[0116] where is the jth diagonal element of D (i) , is the jth row of matrix Thus, the objective function can be rewritten as:

[0117]

[0118] The update of can be expressed as:

[0119]

[0120] 3.3, Update W k(t) : Similar to the update of C k(t) , the closed-form solution of W k(t) is obtained. As follows:

[0121]

[0122]

[0123]

[0124] 3.4, Update R k(t) (1≤k≤K-1): Similar to the update of C k(t) , the closed-form solution of R k(t) is obtained. As follows:

[0125]

[0126]

[0127]

[0128] 3.5, Update B k(t) : Fixing other variables, the objective function for B k(t) can be simplified as:

[0129]

[0130] From the above expression, we can observe that when k = K, i.e., only one length of hash code is learned, the constraint term is meaningless. Thus, the update of B k(t) can be discussed in two cases. When k = K, B k(t) can be obtained by the following update:

[0131]

[0132] When 1≤k<K, i.e., multiple lengths of hash code are learned, the solution of B k(t) can be easily obtained:

[0133]

[0134] where sgn(·) is a sign function.

[0135] The target function is optimized through the above steps until the target function converges or the maximum number of iterations is reached, and the iteration is stopped.

[0136] 4. Query, cross-modal retrieval of samples to be retrieved: first get the hash code corresponding to the retrieval set, input the query sample, and get the hash code of the query sample according to the hash function obtained in the last step. The hash code of the query sample is substituted into the retrieval set for query. The specific implementation steps are as follows:

[0137] The feature matrix corresponding to the given image and text query sample is x 1query and x 2que , combined with the projection matrix W1 and W2 obtained in the last step. The hash code corresponding to the query sample is obtained by formula and In this embodiment, we perform two main retrieval tasks: image query text and text query image.

[0138] Since the query task of the present application is carried out in a binary space, the Hamming distance between the query sample and each sample in the retrieval set is calculated, and the top 100 samples in the sorted order are taken as the query result.

[0139] In order to illustrate the effect of the present application, the technical scheme of the present application is further described by specific embodiments as follows:

[0140] 1. Simulation conditions

[0141] The present application uses Matlab 2020a software for experimental simulation. The experiment is carried out on the MIRFlickr dataset (containing two modalities of image and text). The experiment includes two query tasks: image query text (Img2Text) and text query image (Text2Img). In the experiment, the parameters μ, λ, α, β and γ are set to 1e2, 1e-6, 1e-1, 1e3 and 1e-6 respectively, and the MIRFlickr dataset is divided into 8 rounds of data blocks, the first 7 rounds of data blocks are 2000 samples in size, and the 8th round of data blocks is 1902 samples in size.

[0142] 2. Simulation content

[0143] The method of the present application is compared with existing online cross-modal hashing retrieval methods, and the comparison methods include online cross-modal hashing (OCMH), online collaborative matrix factorization hashing (OCMFH), online latent semantic hashing (OLSH), flexible online multi-modal hashing (FOMH) and discrete online cross-modal hashing (DOCH). Among them, OCMH and OCMFH are online unsupervised cross-modal hashing methods, and the remaining comparison methods are online supervised cross-modal hashing methods.

[0144] 3、Simulation results

[0145] In this simulation experiment, a widely used indicator is used to measure the performance of the OMCJL method proposed in the present application and other comparison methods. That is, the average value of the average precision (mAP). Given a query and a retrieval result list, its mAP is defined as:

[0146]

[0147] Where Q is a query instance, N is the number of relevant instances in the retrieval set, and R is the total number of retrieved data points. In the simulation experiment of the present application, R is set to 100. q (r) represents the top-r retrieval accuracy. If the qth query instance is related to the rth instance, then q (r) = 1, otherwise q (r) = 0. Generally, the larger the mAP value, the better the retrieval performance. The hash code length taken from the simulation experiment is 16 bits, 32 bits, 64 bits and 128 bits, and the corresponding mAP values of the OMCJL method proposed in the present application and other comparison methods are shown in Table 1.

[0148] Table 1 mAP values of all methods on the MIRFlickr dataset on image query text (Img2Text) and text query image (Text2Img) tasks

[0149]

[0150] From the results in Table 1, it can be seen that the mAP values of the OMCJL method proposed in the present application in the two query tasks of the MIRFlickr dataset are higher than those of other comparison methods. Further proving the superiority of the OMCJL method proposed in the present application in online cross-modal retrieval.

[0151] The above-described embodiments only express the specific implementation of the present application, and the description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A cross-modal retrieval method based on online multi-hash code joint learning, characterized in that: The specific steps of the method are as follows: Step 1: Obtain the cross-modal dataset and divide the training set in the dataset into t rounds of data blocks to simulate streaming data; each round of data block contains sample features of different modalities and their corresponding semantic labels; Step 2: Construct the objective function learning by using a training dataset divided into t rounds to simulate streaming data to train the objective function; When the data block of round t arrives, the hash codes of different lengths obtained from training the data blocks of the previous t-1 rounds are kept unchanged. The hash codes of different lengths of the data block of round t are learned and saved. The hash function is updated online according to the real-time new data of the data block of round t. Step 3: Perform cross-modal retrieval of the sample to be retrieved: First, obtain the hash code corresponding to the retrieval set. Input the query sample and obtain the hash code of the query sample according to the hash function obtained in the previous step. Substitute the hash code of the query sample into the retrieval set to perform the query. Obtain the query result by calculating the Hamming distance between the query set and each sample in the retrieval set. In Step 2, each data block includes two modalities: image and text, along with corresponding labels; during the t-th training round, the data blocks are labeled with class labels. Pairs of data blocks Added to the training set; where and Let represent the image feature matrix of dimension d1 and the text feature matrix of dimension d2, respectively. c and c represent the new image-text data block size and the number of categories, respectively; in round t, the samples are divided into the old training samples accumulated from the previous t-1 rounds. and the new training samples of the tth time ; Include There are 10 sample pairs, and their corresponding label matrices are: Therefore, the total training dataset in round t is denoted as ,Include The total label matrix corresponding to each data point is denoted as . ; The specific steps in Step 2 include: Step 2.1: Joint learning of multiple hash codes; In a unified training model, multiple hash codes of different lengths are learned simultaneously, increasing the scalability of hash lengths. Furthermore, to explore and learn hash codes, the longest hash code is used for learning other hash codes of different lengths. ； in, Denotes the Frobenius square of the matrix. and These represent the k-th hash codes of the data block in round t and the data blocks from the previous (t-1) rounds, respectively. K is the number of hash codes. and Let represent the longest hash codes of the data block in round t and the data blocks in the previous t-1 rounds, respectively. Let be the projection matrix. and They represent and Length, and For balance parameters; Step 2.2, Tag embedding; To fully utilize semantic label information, the supervised label information is regressed to hash codes, ensuring that the learned hash codes maintain consistency with the semantic labels; this is expressed by the following formula: ； in, Let be the projection matrix. For balance parameters; Step 2.3: Learning about hash functions; The nonlinear features of the original multimodal data are directly projected onto a discrete common representation as a unified hash code for training instances using a projection matrix. To preserve the consistency and modality-specific properties of the learned hash code, the projection matrix is ​​decomposed into consistent and modality-specific matrices. The inconsistency between image and text modalities is considered a special type of noise; therefore, a modality-specific projection matrix is ​​applied based on... Sparsity constraints on norms; hash functions are learned through linear regression problems. ； in and These are the consistent and mode-specific projection matrices, respectively; , and This represents the trade-off parameters that control the corresponding contribution items; It is the kernel feature matrix of an image or text, designed to map the original samples to a high-dimensional feature space, suitable for handling linearly inseparable problems; here, This represents the RBF kernel function, where m represents the number of anchor points; therefore, kernel characteristics... Defined as follows: ； in This represents m anchor points; These are Gaussian kernel parameters; Step 2.4, Objective Function; Based on the above steps, the objective function is: ； Step 2.5: Iteratively update the objective function: Iterate and update the objective function obtained in the previous step until the objective function converges or reaches the maximum number of iterations, and obtain the hash function and the hash code of the training set.

2. The cross-modal retrieval method based on online multi-hash code joint learning according to claim 1, characterized in that: In Step 1, in order to ensure the generalization ability of the model, the data is processed and the training set is divided using random block method before entering the model training to simulate streaming data.

3. The cross-modal retrieval method based on online multi-hash code joint learning according to claim 1, characterized in that, The sample features of the different modalities include at least: sample features of the image-text modality and sample features of the text modality.

4. The cross-modal retrieval method based on online multi-hash code joint learning according to claim 1, characterized in that: The specific process of obtaining the objective function in Step 2 includes: mapping the original samples to a high-dimensional feature space, and projecting the multimodal data features onto a discrete common representation as a unified hash code for training instances using a consistent and modality-specific projection matrix, and applying a method based on... The norm is used to impose sparsity constraints, and the label matrix is ​​embedded into the objective function as supervision information. In addition, semantic relationships are established between hash codes of different lengths and the longest hash code to fully explore the learned hash codes.

5. The cross-modal retrieval method based on online multi-hash code joint learning according to claim 1, characterized in that: The objective function in Step 2 includes two parts: hash code learning and hash function learning. Hash code learning refers to learning by combining semantic labels and the semantic relationship between hash codes of different lengths and the longest hash code. Hash function learning refers to using a simple linear regression model as the hash function for each modality, projecting the features of the training samples onto the hash code through a consistent and modality-specific projection matrix. By learning the hash code and hash function in the same model, the semantic relationship between the hash code and the hash function is enhanced, thereby achieving high-precision online cross-modal retrieval.

6. The cross-modal retrieval method based on online multi-hash code joint learning according to claim 1, characterized in that: The objective function in Step 2 includes two data components: the data newly arriving in round t and the old data blocks accumulated in the previous t-1 rounds.

7. The cross-modal retrieval method based on online multi-hash code joint learning according to claim 1, characterized in that: The optimization steps for the objective function in Step 2 are as follows: 2.5.1 Update : Fix other variables, for The objective function simplifies to: ； set up , deduced The closed solution is as follows: ； in , and Represented as: ； ； ； ； in , , and They were all obtained in round t-1, from , , and The representation observed is: , , and The first part relates to the newly arrived data in round t, and the second part relates to the old data accumulated in the previous rounds (t-1); therefore , , and Calculated incrementally Similarly, updates are made through online learning. , ,and Updated through online learning; 2.5.2 Update Fix other variables, for The objective function simplifies to: ； In order to solve based on The optimization problem of norms introduces the following weight matrix: ； in yes The diagonal element of the j-th row, It is a matrix The j-th row, therefore, the above objective function is rewritten as: ； set up , The update is represented as: ； 2.5.3, Update :and The update steps are similar, resulting in... The closed solution is as follows: ； ； ； 2.5.4 Update :and The update steps are similar, resulting in... The closed solution is as follows: ； ； ； 2.5.5, Update With other variables fixed, for The objective function simplifies to: ； From the above expression, we can observe that: when When only a hash code of a single length is learned, the constraint term... Meaningless, therefore The update is discussed in different cases; when hour, Obtained through the following updates: ； when When learning hash codes of multiple lengths, The solution is easy to obtain: ； in It is a symbolic function; The objective function is optimized through the above steps until it converges or the maximum number of iterations is reached, at which point the iteration stops.

Images