Multi-modal target identification method based on anti-noise contrast loss

Abstract

The invention discloses a multi-modal target identification method based on anti-noise contrast loss, and the method comprises the following steps: S1, selecting different encoder architectures and decoder architectures for each modal according to the input multi-modal target information, and respectively taking the encoder architectures and the decoder architectures as a feature extraction network and a reconstruction network; s2, constructing a multi-modal target recognition network according to the number of input multi-modal target categories; s3, performing training according to the anti-noise comparison loss and the modal reconstruction loss, and updating the feature extraction network, the reconstruction network and the recognition network; s4, inputting modal data of a to-be-identified target into the feature extraction network, fusing the modal data, inputting the fused modal data into the identification network, finally obtaining dimension vectors with the same category number, and selecting the category with the maximum probability value as a prediction category of the to-be-identified target; according to the invention, the problem of low recognition accuracy in a traditional cross entropy-based multi-modal target recognition method in a noisy scene is solved, and an over-fitting phenomenon caused by tag noise is relieved.

Description

Technical Field

[0001] The present invention relates to the field of target recognition, and in particular to a multimodal target recognition method based on noise-resistant contrast loss. Background Art

[0002] In recent years, multimodal object recognition technology has been widely used in many fields such as computer vision, natural language processing and speech recognition. By fusing data from different modalities (such as images, text and audio), multimodal methods can capture information unique to each modality and achieve more comprehensive feature expression, thereby significantly improving recognition accuracy and robustness. Compared with single-modal recognition methods, multimodal recognition can make up for the possible information loss problem of a single modality and make the model more stable and accurate in complex environments. For example, the joint modeling of images and texts can enable the model to rely not only on visual features, but also understand semantic information, thereby enhancing the recognition effect.

[0003] In current multimodal object recognition methods, deep neural networks have become the main implementation tool. Relying on their powerful feature learning capabilities, they encode the input modal data into a unified feature space for prediction in the classification network. In practical applications, commonly used classification methods are usually based on cross-entropy loss, which is a loss function that measures the difference between the predicted distribution and the true label distribution, and can guide the model to effectively converge to the ideal classification boundary. However, cross-entropy loss is sensitive to label noise. In the case of inaccurate labeling or the presence of noisy labels, the model is easily disturbed by noise, resulting in overfitting of wrong labels, thereby misleading the classification boundary. This situation is particularly evident in data with noisy labels. The model may be biased towards noisy samples during training, resulting in a significant decrease in recognition accuracy and a weakening of generalization ability. As the proportion of noisy labels increases, the robustness problem of cross-entropy loss becomes more obvious, limiting its applicability in complex environments. Therefore, designing a multimodal recognition algorithm with noise-resistant contrast loss to improve the robustness of the model in noisy scenes and effectively reduce the overfitting phenomenon caused by label noise can provide a more adaptable and stable solution for the practical application of multimodal target recognition technology. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multimodal target recognition method based on noise-resistant contrast loss, which solves the problem of low recognition accuracy in traditional multimodal target recognition methods based on cross entropy in noisy scenarios and alleviates the overfitting phenomenon caused by label noise.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a multimodal target recognition method based on noise-resistant contrast loss, comprising the following steps:

[0006] S1. According to the input multimodal target information, different encoder architectures and decoder architectures are selected for each modality as feature extraction networks and reconstruction networks respectively;

[0007] S2. Construct a multimodal target recognition network according to the number of multimodal target categories input;

[0008] S3, training according to the noise-resistant contrast loss and modality reconstruction loss, and updating the feature extraction network, reconstruction network and recognition network;

[0009] S4, input each modal data of the target to be identified into the feature extraction network, and then input it into the recognition network after fusion, and finally obtain a dimensional vector with the same number of categories, and select the category with the largest probability value as the predicted category of the target to be identified;

[0010] Furthermore, step S1 includes the following sub-steps:

[0011] S11. For the input multimodal target information, based on the characteristics of each modality (e.g., image, text, etc.), select a suitable encoder architecture to convert each modality data into a low-dimensional feature representation. Specifically, for the image modality, a convolutional neural network (CNN) or a visual transformer (ViT) is usually selected as the encoder architecture, such as ResNet or Swin Transformer, to extract the spatial structural features of the image; for the text modality, a transformer architecture based on the self-attention mechanism, such as BERT or RoBERTa, is usually selected to capture the semantic information of the text.

[0012] S12. In order to retain the unique information within each modality, based on the low-dimensional feature space, a suitable decoder architecture is selected for each modality to reconstruct the extracted low-dimensional features back to the original data space. Specifically, for the reconstruction of the image modality, a deconvolution network (Deconvolution Network) or an upsampling network (such as U-Net) is usually selected as a decoder to restore the structural features of the image; for the text modality, a Transformer decoder or Seq2Seq structure can be used to regenerate the features into semantically consistent text. In this way, it can be ensured that the features of each modality retain their unique information, which helps the model not lose information details within the modality when fusing features.

[0013] Furthermore, in step S11, for each modality, a specific encoder network is used to extract its feature representation:

[0014] Z i =E i (X i )

[0015] Where X i represents the input data of the i-th mode, E i represents the data encoder of the i-th mode, Z i Represents the initial feature representation of the i-th modality. Since the feature dimensions output by the data encoders of different modalities may be inconsistent, the features are mapped through a two-layer fully connected network with a rectified linear unit (ReLU) to unify the feature dimensions. Specifically, the mapping process is:

[0016] F i =ReLU(W2·(ReLU(W1·Z i +b1))+b2)

[0017] Among them, F i represents the feature representation of the i-th mode, W1 and W2 represent the weight matrices of the mapping network, b1 and b2 are bias terms, and ReLU represents the rectified linear unit activation function operation.

[0018] The beneficial effects of the above further scheme are:

[0019] 1. Flexibly selecting encoders of different modalities can effectively improve the expressiveness of features and enable the model to better adapt to the heterogeneity of multimodal data.

[0020] 2. Selecting appropriate encoders based on the data characteristics of different modalities can reduce the training complexity of the model and improve the efficiency of feature extraction.

[0021] Furthermore, the reconstruction process formula of step S12 is as follows:

[0022]

[0023] in, represents the reconstructed i-th modal data, D i represents the decoder network selected for the i-th modality, E i represents the data encoder of the i-th mode, X i Represents the input data of the i-th mode.

[0024] The beneficial effects of the above further scheme are:

[0025] 1. Using the decoder to reconstruct intra-modal data can enhance the robustness of the model, allowing the feature extraction network to learn the feature distribution of different modalities more stably.

[0026] 2. The reconstruction process introduces intra-modal consistency constraints, making the features of each modality more identifiable and complementary, which helps to preserve the unique information within the modality.

[0027] Further, step S2 includes the following sub-steps:

[0028] S21. Fuse the outputs of the modal encoders to obtain a fused feature representation of the sample.

[0029] S22. According to the number of multimodal target categories, a softmax-based recognition network is constructed to map the features of each modality to the category probability space.

[0030] Furthermore, the fusion method in step S21 is:

[0031]

[0032] Among them, F i represents the feature representation of the i-th modality, M represents the number of modalities, and F represents the sample fusion feature representation. Represents the summation operation of each modal feature.

[0033] The beneficial effects of the above further scheme are:

[0034] 1. By fusing the feature representations of each modality, the final feature representation F can integrate the information of different modalities, allowing the model to capture more comprehensive sample features.

[0035] 2. It effectively reduces the deviation that may be caused by a single modality, further improves the recognition accuracy, and enhances the generalization ability of the model.

[0036] Furthermore, the output after softmax activation in step S22 is:

[0037] P(y|F)=softmax(H(F))

[0038] Where F represents the sample fusion feature representation, H represents the recognition network, which is used to map the modal features to the category space, softmax represents the activation function operation, P(y|F) represents the category prediction probability of the input sample, and the output dimension is equal to the number of target categories.

[0039] The beneficial effects of the above scheme are:

[0040] 1. The recognition network can map multimodal features into a unified category space to facilitate target recognition tasks.

[0041] 2. Using softmax as the output activation function can effectively convert it into a probability distribution, making the prediction results more intuitive and easy to explain.

[0042] Further, step S3 includes the following sub-steps:

[0043] S31. Use noise-resistant contrast loss to train the fused feature representation of the samples to reduce the prediction error of the target category and alleviate the overfitting phenomenon caused by noisy labels.

[0044] S32. Use modal reconstruction loss to train the feature extraction network and reconstruction network to maintain the integrity of the modal information and enhance the robustness of the model.

[0045] Furthermore, the anti-noise contrast loss function in step S31 is:

[0046]

[0047] in represents the noise-resistant contrast loss function, P(y c |F) represents the category prediction probability of the i-th mode, α represents a noise-sensitive parameter, and y c Represents the one-hot encoding vector of the true category label, and C represents the number of categories. c |F) finds the gradient, which is proportional to the probability P(y c |F) optimization will be performed in the following two intervals:

[0048]

[0049] Where P(y c |F) represents the class prediction probability of the input sample, α represents a noise-sensitive parameter, and e represents the natural base. For the first interval Probability P(y c |F) is optimized in the opposite direction to the cross entropy optimization direction. For the second interval Probability P(y c |F) is less than the gradient of the cross entropy in this interval.

[0050] The beneficial effects of the above further scheme are:

[0051] 1. Noise labels will cause the common problem of false negative sample pairs. For false negative sample pairs in interval 1, the probability of being predicted as the correct category will be low due to incorrect labeling. At this time, the reverse optimization strategy of interval 1 can enable the sample pairs falling in this interval to obtain the correct gradient descent direction, thereby improving the robustness of the model.

[0052] 2. For the false negative sample pairs in interval 2, since their gradient values ​​are smaller than the ordinary cross entropy loss, they will obtain a smoother gradient to reduce the overfitting degree of the model for the false negative sample pairs. Further, the reconstruction loss function in step S32 is used to minimize the error between the input features and the reconstructed features, and is defined as follows:

[0053]

[0054] in represents the reconstruction loss function, X i represents the input data of the i-th mode, represents the reconstruction result of the i-th mode, Represents the L2 norm, which is used to calculate the sum of squares of the reconstruction error.

[0055] The beneficial effects of the above further scheme are:

[0056] 1. Reconstruction loss can help the model retain more original modal information during feature extraction, thereby improving the recognition effect of the target.

[0057] 2. The modal reconstruction loss as a constraint term can reduce the information loss in the feature extraction process and enhance the robustness of the model under noisy data.

[0058] Further, step S4 includes the following sub-steps:

[0059] S41, first, input each modality data of the sample to be identified into the respective feature extraction network to obtain the feature representation of each modality. Then, according to the steps in step S21, it is fused and then passed into the recognition network to finally obtain an output vector with the same dimension as the number of categories.

[0060] S42. After obtaining the output vector of the recognition network, select the category with the largest probability value as the final predicted category of the target to be recognized.

[0061] Furthermore, the predicted probability of each category in step S41 can be expressed as:

[0062] P(y|F)=softmax(H(F))

[0063] Among them, P(y|F i ) represents the category prediction probability of sample fusion feature representation, H represents the recognition network, F represents the sample fusion feature representation, and softmax represents the activation function operation.

[0064] The beneficial effects of the above further scheme are:

[0065] 1. Through the combination of feature extraction network and recognition network, multimodal features can be effectively extracted and recognized, providing strong support for multimodal target recognition tasks.

[0066] 2. Use the softmax function to convert the output into a probability distribution, which helps to further analyze and understand the model’s prediction confidence for each category.

[0067] Furthermore, the final predicted category of the target to be identified in step S42 can be expressed as:

[0068]

[0069] in represents the final predicted category of the target to be identified, c represents the current predicted category, C represents the total number of categories, F represents the sample fusion feature representation, P(y=c|F) represents the probability that the sample fusion feature representation is predicted as category c, and argmax represents the selection of the category label corresponding to the maximum probability to ensure the best recognition result. The beneficial effects of the above further scheme are:

[0070] 1. By selecting the class with the highest probability as the final prediction, the recognition accuracy can be enhanced and the practicality of the unsupervised multimodal object recognition method can be improved.

[0071] In summary, the beneficial effects of the present invention are as follows: the present invention establishes a multimodal target information feature extraction network, a reconstruction network and a multimodal target recognition network, and uses anti-noise contrast loss and modal reconstruction loss to train the feature extraction network, the reconstruction network and the recognition network. Finally, the maximum output probability of the recognition network is used as the predicted category of the target to be recognized. This solves the problem of low recognition accuracy in the traditional cross-entropy-based multimodal target recognition method in noisy scenarios, and alleviates the overfitting phenomenon caused by label noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. It should be understood that the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0073] Figure 1 Flowchart of the present invention

[0074] Figure 2 This is the principle diagram of the anti-noise contrast loss gradient of the present invention DETAILED DESCRIPTION

[0075] The technical solution in the embodiment of the present invention will be described below in conjunction with the accompanying drawings in the embodiment of the present invention. Figure 1 As shown, a multimodal target recognition method based on noise-resistant contrast loss includes the following steps:

[0076] S1. According to the input multimodal target information, different encoder architectures and decoder architectures are selected for each modality as feature extraction networks and reconstruction networks respectively;

[0077] Step S1 includes the following sub-steps:

[0078] S11. For the input multimodal target information, according to the characteristics of the modality (such as image, text, audio, etc.), select the appropriate encoder architecture (such as Swin Transformer architecture for image modality and BERT architecture for text modality) to convert each modality data into a low-dimensional feature representation. For each modality, use a specific encoder network to extract its feature representation:

[0079] Z i =E i (X i )

[0080] Where X i represents the input data of the i-th mode, E i represents the data encoder of the i-th mode, Z i Represents the initial feature representation of the i-th modality. Since the feature dimensions output by the data encoders of different modalities may be inconsistent, the features are mapped through a two-layer fully connected network with a rectified linear unit (ReLU) to unify the feature dimensions. Specifically, the mapping process is:

[0081] F i =ReLU(W2·(ReLU(W1·Z i +b1))+b2)

[0082] Among them, F i represents the feature representation of the i-th mode, W1 and W2 represent the weight matrices of the mapping network, b1 and b2 are bias terms, and ReLU represents the rectified linear unit activation function operation.

[0083] S12. In order to retain the unique information within the modality, a corresponding decoder architecture is selected for each modality so that it can reconstruct the extracted low-dimensional features back to the original data space. The reconstruction process formula is as follows:

[0084]

[0085] in, represents the reconstructed i-th modal data, D i represents the decoder network selected for the i-th modality, E i represents the data encoder of the i-th mode, X i Represents the input data of the i-th mode.

[0086] Further, step S2 includes the following sub-steps:

[0087] S21. The outputs of the modal encoders are fused to obtain a fused feature representation of the sample. The fusion method is:

[0088]

[0089] Among them, F i represents the feature representation of the i-th modality, M represents the number of modalities, and F represents the sample fusion feature representation. Represents the summation operation of each modal feature.

[0090] S22. According to the number of multimodal target categories, a softmax-based recognition network is constructed to map the features of each modality to the category probability space. The output after softmax activation is:

[0091] P(y|F)=softmax(H(F))

[0092] Where F represents the sample fusion feature representation, H represents the recognition network, which is used to map the modal features to the category space, softmax represents the activation function operation, P(y|F) represents the category prediction probability of the input sample, and the output dimension is equal to the number of target categories.

[0093] Further, step S3 includes the following sub-steps:

[0094] S31. Use the noise-resistant contrast loss to train the final feature representation of the sample to reduce the prediction error of the target category and alleviate the overfitting phenomenon caused by the noise label. The noise-resistant contrast loss function is:

[0095]

[0096] in represents the noise-resistant contrast loss function, P(y c |F) represents the category prediction probability of the i-th mode, α represents a parameter sensitive to false negative samples, and y c Represents the one-hot encoding vector of the true category label, and C represents the number of categories. c |F) finds the gradient, which is proportional to the probability P(y c |F) optimization will be performed in the following two intervals:

[0097]

[0098] Where P(y c |F) represents the predicted probability of the input sample, α represents a noise-sensitive parameter, and e represents the natural base. like Figure 2 As shown, the probability P(y c |F) is optimized in the opposite direction to the cross entropy optimization direction; for the second interval like Figure 2 As shown inc |F) is less than the gradient of the cross entropy in this interval.

[0099] S32. Use the modal reconstruction loss to train the feature extraction network and the reconstruction network to maintain the integrity of the modal information and enhance the robustness of the model. The reconstruction loss function is used to minimize the error between the input features and the reconstructed features, and is defined as follows:

[0100]

[0101] in represents the reconstruction loss function, X i represents the input data of the i-th mode, represents the reconstruction result of the i-th mode, Represents the L2 norm, which is used to calculate the sum of squares of the reconstruction error.

[0102] Further, step S4 includes the following sub-steps:

[0103] S41. First, input each modality data of the sample to be identified into the respective feature extraction network to obtain the feature representation of each modality. Then, it is fused according to the method in step S21, and then passed into the recognition network to finally obtain an output vector with the same dimension as the number of categories. The prediction probability of each category can be expressed as:

[0104] P(y|F)=softmax(H(F))

[0105] Among them, P(y|F i ) represents the category prediction probability of sample fusion feature representation, H represents the recognition network, F represents the sample fusion feature representation, and softmax represents the activation function operation.

[0106] S42, after obtaining the output vector of the recognition network, select the category with the largest probability value as the final predicted category of the target to be recognized. The final predicted category of the target to be recognized can be expressed as:

[0107]

[0108] in represents the final predicted category of the target to be identified, c represents the current predicted category, C represents the total number of categories, F represents the sample fusion feature representation, P(y=c|F) represents the probability that the sample fusion feature representation is predicted to be category c, and argmax represents the selection of the category label corresponding to the maximum probability to ensure the best recognition result.

[0109] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A multimodal target recognition method based on noise-resistant contrast loss, characterized in that: The following steps are involved: S1. According to the input multimodal target information, different encoder architectures and decoder architectures are selected for each modality as feature extraction networks and reconstruction networks respectively; S2. Construct a multimodal target recognition network according to the number of multimodal target categories input; S3, training according to the noise-resistant contrast loss and modality reconstruction loss, and updating the feature extraction network, reconstruction network and recognition network; S4. Input the data of each modality of the target to be identified into the feature extraction network, and then input them into the recognition network after fusion, and finally obtain a vector with the same dimension as the number of categories, and select the category with the largest probability value as the predicted category of the target to be identified.

2. The multimodal target recognition method based on noise-resistant contrast loss according to claim 1, characterized in that: The step S2 comprises the following sub-steps: S21. The outputs of the modal encoders are fused to obtain a fused feature representation of the sample. The fusion method is: Among them, F i represents the feature representation of the i-th modality, M represents the number of modalities, and F represents the sample fusion feature representation. Represents the summation operation of each modal feature; S22. According to the number of multimodal target categories, a softmax-based recognition network is constructed to map the features of each modality to the category probability space. The output after softmax activation is: P(y|F)=softmax(H(F)) Where F represents the sample fusion feature representation, H represents the recognition network, which is used to map the modal features to the category space, softmax represents the activation function operation, P(y|F) represents the category prediction probability of the input sample, and the output dimension is equal to the number of target categories.

3. The multimodal target recognition method based on noise-resistant contrast loss according to claim 2, characterized in that: The step S3 comprises the following sub-steps: S31. Use the noise-resistant contrast loss to train the final feature representation of the sample to reduce the prediction error of the target category and alleviate the overfitting phenomenon caused by the noise label. The noise-resistant contrast loss function is: in represents the noise-resistant contrast loss function, P(y c |F) represents the category prediction probability of the i-th mode, α represents a noise-sensitive parameter, and y c Represents the unique hot encoding vector of the true category label, C represents the number of categories, and the probability P(y c |F) finds the gradient, which is proportional to the probability P(y c |F) optimization will be performed in the following two intervals: Where P(y c |F) represents the class prediction probability of the input sample, α represents a noise-sensitive parameter, and e represents the natural base; for the first interval Probability P(y c |F) is optimized in the opposite direction to the cross entropy optimization direction; for the second interval Probability P(y c |F) is smaller than the gradient of the cross entropy in this interval; for the false negative sample pairs in interval 1, the probability of being predicted as the correct category due to the wrong labeling will be low. At this time, the reverse optimization strategy of interval 1 can make the sample pairs in this interval obtain the correct gradient descent direction, thereby improving the robustness of the model; for the false negative sample pairs in interval 2, since their gradient value is smaller than the ordinary cross entropy loss, they will obtain a smoother gradient to reduce the overfitting degree of the model for the false negative sample pairs; S32. Use the modal reconstruction loss to train the feature extraction network and the reconstruction network to maintain the integrity of the modal information and enhance the robustness of the model. The reconstruction loss function is used to minimize the error between the input features and the reconstructed features, and is defined as follows: in represents the reconstruction loss function, X i represents the input data of the i-th mode, represents the reconstruction result of the i-th mode, Represents the L2 norm, which is used to calculate the sum of squares of the reconstruction error.

Images