Self-paced Hash multi-modal learning method for noise tag

Abstract

The invention discloses a self-paced Hash multi-modal learning method for noise tags. The method comprises the following steps: S1, obtaining feature data sets and tag data sets of all modals; s2, initializing a hash center for each category; s3, each modal feature is projected to a public Hamming space by using a modal-specific sub-network, the loss of each sample is calculated, noise samples are distinguished by using a self-paced function, the loss of other samples is weighted, and finally all sub-networks and hash centers are optimized; s4, repeating the step S3 to optimize all the sub-networks and the Hash centers until convergence; s5, inputting a test set sample of each mode, and obtaining a hash code by using a sub-network specific to the mode; according to the method, the noise instance can be identified by evaluating the learning difficulty of each instance, and each instance in the Hamming space can be learned from easy to difficult, so that the model has higher robustness and generalization ability.

Description

Technical Field

[0001] The present invention relates to the field of multimodal learning in artificial intelligence to improve the accuracy of semantic information representation by jointly modeling modalities such as text and images. More specifically, it relates to a self-paced hashing multimodal learning method for noise labels. Background Art

[0002] With the explosive growth of multimodal data, multimodal learning has attracted extensive attention in the community. Multimodal learning can be used in many downstream tasks and aims to make full use of the complementary information between different modalities to improve model performance. For large-scale multimodal data, multimodal hash learning has become an effective solution due to its low storage cost and high retrieval efficiency. The basic idea of ​​multimodal hash learning is to learn discriminative hash codes to alleviate the heterogeneity gap between different modalities. Due to the complexity of collecting annotations, some unsupervised cross-modal retrieval methods have been proposed to eliminate the dependence on rich labels. However, they usually fail to achieve satisfactory retrieval performance due to the lack of guidance from supervised semantic information. Recently, many supervised multimodal hashing have been proposed and achieved satisfactory performance. Most of them implicitly assume that all collected labels are correctly labeled. However, in real-world scenarios, such an assumption is unrealistic because manual annotations or non-expert annotations inevitably bring some noisy labels. Due to the large heterogeneity gap, learning from multimodal instances with noisy labels is difficult. Therefore, alleviating the performance degradation caused by noisy labels on cross-modal retrieval is a challenging problem.

[0003] In order to mitigate the negative impact of noisy labels, many multimodal learning methods have been developed. For example, MRL proposed a robust loss function to increase the loss of clean samples, thereby guiding the model towards the correct optimization direction; ELRCMR used dynamic weights to prevent overfitting of noisy labels. However, most of them are real-valued representation methods, which may lead to inefficient computation and high storage costs. Although hash representation is more lightweight, unreliable labels may amplify quantization errors. To this end, some multimodal hash learning methods have been proposed. For example, NrDCMH adopts the difference between label similarity and feature similarity to detect noise and train instance pairs. Although these methods have made significant progress, most of them implicitly assume that the learning priority of each instance remains unchanged, which may bias the model towards difficult instances with noisy labels and lead to overfitting problems. Summary of the invention

[0004] In view of the above shortcomings in the prior art, the present invention proposes a self-paced hashing multimodal learning method for noisy labels. This method can not only identify noisy instances by evaluating the learning difficulty of each instance, but also learn each instance in the Hamming space from easy to difficult, so that the model has both stronger robustness and generalization ability.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A self-paced hashing multimodal learning method for noisy labels includes the following steps:

[0007] S1: Obtain feature datasets and label datasets of all modalities;

[0008] S2: Initialize a hash center for each category;

[0009] S3: Use a modality-specific sub-network to project each modality feature into a common Hamming space and calculate the loss of each sample. Then use the self-stepping function to distinguish the noise samples and weight the losses of other samples. Finally, optimize all sub-networks and hash centers.

[0010] S4: Repeat S3 to optimize all sub-networks and hash centers until convergence;

[0011] S5: Input the test set samples of each modality and obtain the hash code using the modality-specific sub-network.

[0012] Furthermore, the feature dataset and label dataset in step S1 are defined as:

[0013] D={D1,D2,…,D M}

[0014] Where M is the number of modes, and the data set of the mth mode is

[0015]

[0016] in is the i-th sample on the m-th mode, y i ∈{0, 1} K is its corresponding label, N is the number of instances, K is the number of categories, if the i-th instance belongs to the k-th category, then y i The kth element of is 1, which is y i，k =1, otherwise y i，k =0.

[0017] Furthermore, the hash center in S2 is defined as:

[0018] C={c1,c2,…,c K}

[0019] The hash center of the kth category is a normalized and discretized binary vector.

[0020] Furthermore, S3 assumes is the hash code of the i-th sample on the m-th mode. The loss consists of two items. The first item is the anti-noise self-paced loss:

[0021]

[0022] in is the weight factor, regularizer The loss weight of the i-th instance is γ is the learning progress parameter, is the hash center of the kth category. When the instance loss is greater than γ, the instance is considered to be noise and assigned a weight of 0. On the contrary, instances with smaller loss values ​​are considered easy and assigned larger weights, and instances with larger loss values ​​are considered difficult and assigned smaller weights. The second cross-modal contrast loss is:

[0023]

[0024] in is the weight factor, is the hash code of the i-th sample on the m-th modality. The total loss function is as follows:

[0025]

[0026] Among them, α is a hyperparameter that balances the two losses.

[0027] Further, in S4, the S3 operation is repeated to optimize all sub-networks and hash centers until convergence.

[0028] Furthermore, in S5, the optimized sub-network is used to obtain the binary representation of the test set and perform downstream tasks. Taking the image-to-text retrieval task as an example, the specific implementation process is as follows:

[0029] 1. A query sample given an image.

[0030] 2. Calculate the similarity between the sample and all test samples in the text modality.

[0031] 3. Sort by similarity, and the search sample ranked first is the most relevant search result.

[0032] 4. Sort by similarity and obtain the first K corresponding samples based on the sorting results, which are the candidate search items.

[0033] In summary, the self-paced hashing multimodal learning method for noise labels provided by the present invention has the following beneficial effects:

[0034] The present invention adopts a self-paced hashing multimodal learning method for noise labels to identify noise instances by evaluating the learning difficulty of each instance, and can also learn each instance in the Hamming space from easy to difficult, so that the model has both stronger robustness and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. It should be understood that the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A step diagram of a self-paced hashing multimodal learning method for noise labels provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0038] Specifically, the method includes the following steps:

[0039] S1: Obtain feature datasets and label datasets of all modalities;

[0040] Furthermore, the feature dataset and label dataset are defined as:

[0041] D={D1,D2,…,D M}

[0042] Where M is the number of modes, and the data set of the mth mode is

[0043]

[0044] in is the i-th sample on the m-th mode, y i ∈{0, 1} K is its corresponding label, N is the number of instances, K is the number of categories, if the i-th instance belongs to the k-th category, then the k-th element of yi is 1, that is, y i，k =1, otherwise y i，k =0.

[0045] S2: Initialize a hash center for each category;

[0046] Furthermore, the hash center in S2 is defined as:

[0047] C = {c1, c2, ..., c K}

[0048] The hash center of the kth category is a normalized and discretized binary vector.

[0049] S3: Use a modality-specific sub-network to project each modality feature into a common Hamming space and calculate the loss of each sample. Then use the self-stepping function to distinguish the noise samples while weighting the losses of other samples. Finally, optimize all sub-networks and hash centers.

[0050] Furthermore, S3 assumes is the hash code of the i-th sample on the m-th mode. The loss consists of two items. The first item is the anti-noise self-paced loss:

[0051]

[0052] in is the weight factor, regularizer The loss weight of the i-th instance is γ is the learning progress parameter, is the hash center of the kth category. When the instance loss is greater than γ, the instance is considered to be noise and assigned a weight of 0. On the contrary, instances with smaller loss values ​​are considered easy and assigned larger weights, and instances with larger loss values ​​are considered difficult and assigned smaller weights. The second cross-modal contrast loss is:

[0053]

[0054] in is the weight factor, is the hash code of the i-th sample on the m-th modality. The total loss function is as follows:

[0055]

[0056] Among them, α is a hyperparameter that balances the two losses.

[0057] S4: Repeat S3 to optimize all sub-networks and hash centers until convergence.

[0058] S5: Input the test set samples of each modality and obtain the hash code using the modality-specific sub-network.

[0059] Furthermore, in S5, the optimized subnetwork is used to obtain the binary representation of the test set and perform downstream tasks. Taking the image-to-text retrieval task as an example, the specific implementation process is as follows:

[0060] 1. A query sample given an image.

[0061] 2. Calculate the similarity between the sample and all test samples in the text modality.

[0062] 3. Sort by similarity, and the search sample ranked first is the most relevant search result.

[0063] 4. Sort by similarity and obtain the first K corresponding samples based on the sorting results, which are the candidate search items.

[0064] In the embodiments provided by the present invention, it should be understood that the disclosed methods can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0065] In the embodiments provided by the present invention, it should be understood that the disclosed methods can be implemented in other ways. For example, the method embodiments described above are only illustrative, for example, the division of the method step units is only a logical function division, and there may be other division methods in actual implementation, such as more steps and details, or some steps may be ignored or not executed.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0067] Table 1 MAP of each method for cross-modal retrieval task on INRIA-Websearch dataset

[0068]

Claims

1. A self-paced hashing multimodal learning method for noisy labels, characterized by: The method comprises: S1: Obtain feature datasets and label datasets of all modalities; S2: Initialize a hash center for each category; S3: Use a modality-specific sub-network to project each modality feature into a common Hamming space and calculate the loss of each sample. Then use the self-stepping function to distinguish the noise samples and weight the losses of other samples. Finally, optimize all sub-networks and hash centers. S4: Repeat S3 to optimize all sub-networks and hash centers until convergence; S5: Input the test set samples of each modality and obtain the hash code using the modality-specific sub-network.

2. The self-paced hashing multimodal learning method for noisy labels according to claim 1 is characterized in that: The total loss function of step S3 is as follows: Among them, α is a hyperparameter that balances the two losses. To resist the noise self-stepping loss, is the cross-modal contrast loss, and the first anti-noise self-paced loss is: in, y i ∈{0, 1} K is its corresponding label, K is the number of categories, if the i-th instance belongs to the k-th category, then y i The kth element of is 1, which is y i，k =1, otherwise y i，k =0, is a standardized and discretized binary vector, the hash center of the k categories, is the hash code of the i-th sample on the m-th mode, r∈(0,1] is the weight factor, and the regularizer The loss weight of the i-th instance is γ is the learning progress parameter, and the second cross-modal contrast loss calculation formula is: in is the weight factor, is the hash code of the i-th sample on the m-th mode.

3. The self-paced hashing multimodal learning method for noisy labels according to claim 1, characterized in that: The anti-noise self-stepping loss in step S3 is as follows: in, y i ∈{0, 1} K is its corresponding label, K is the number of categories, if the i-th instance belongs to the k-th category, then y i The kth element of is 1, which is y i，k =1, otherwise y i，k =0, is a standardized and discretized binary vector, the hash center of the k categories, is the hash code of the i-th sample on the m-th mode, r∈(0,1] is the weight factor, and the regularizer The loss weight of the i-th instance is γ is the learning progress parameter.

Images