Multi-modal target identification method based on modal contribution degree evaluation

Abstract

The invention discloses a multi-modal target identification method based on modal contribution degree evaluation. The method comprises the following steps: constructing and obtaining a trained multi-modal target identification network; preprocessing the multi-modal image by adopting a data enhancement technology to obtain image input features with the same feature dimension; respectively extracting shallow basic features in the image input features through a feature filtering module, and respectively extracting high-frequency features from the corresponding shallow basic features; respectively calculating the confidence coefficient and the energy value of the high-frequency feature through a modal contribution degree evaluation module; calculating a fusion weight by using the normalized confidence coefficient and the normalized energy value; and obtaining a final target identification result according to the fusion weight. Irrelevant information is effectively inhibited by fully utilizing information of different modes; the contribution degree of the modality is measured, so that the modality is subjected to sample-level dynamic fusion identification; therefore, the problem of unbalanced modal information contribution in the multi-modal target identification process can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal target recognition technology, and particularly relates to a multi-modal target recognition method based on modal contribution degree evaluation. BACKGROUND

[0002] The popular application of multi-sensor improves the efficiency of target recognition in various fields, thereby providing different dimensional information for target recognition. In recent years, designing a reliable multi-modal fusion framework has attracted widespread attention from the industry and academia. Multi-modal fusion technology has broad application prospects in military and civilian fields. Target recognition tasks have important research value. Some existing multi-modal fusion frameworks often use static fusion, ignoring the imbalance of information between modalities. For example, visible light images can provide clear texture information of the target, but when light and other environmental factors affect, the thermal radiation information in the infrared image can overcome these limitations and provide more stable and reliable target information. At present, deep learning methods using multiple modal data have made a series of breakthroughs.

[0003] However, such methods lack interpretability in the end-to-end feedforward propagation process, mainly due to the following reasons: 1) multi-modal recognition methods based on deep learning may perform well in specific scenarios, but lack the ability to adaptively adjust and effectively respond to environmental changes; 2) deep learning models are often data-driven, and their performance depends on a large amount of training data. However, in the industrial scene of target recognition, it is a great challenge to obtain and label a large amount of high-quality multi-modal data. The scarcity and imbalance of data lead to unstable performance of the model in actual application, and the distribution difference between different modal data may also make it difficult for the model to effectively learn the cross-modal feature correlation. 3) Existing multi-modal fusion methods usually lack a mechanism for dynamically weighting the importance of different modal information, which makes it possible for key information to be ignored or overwhelmed by secondary information in certain scenarios. For example, in complex sea conditions, changes in sea surface reflection and weather conditions may reduce the effective information in visible light images, while infrared images may provide more stable target information. Static fusion strategies cannot adapt to the dynamic changes in information weights, thereby limiting the effectiveness of multi-modal fusion. 4) Finally, the "black box" nature of deep learning models also leads to interpretability problems in actual application. The unexplainability of the decision-making process makes it difficult for users to understand the behavior and output of the model, especially in the case of incorrect recognition or abnormal situations. This unexplainability not only limits the debugging and optimization of the model, but also increases the risk of deployment in critical application scenarios. SUMMARY

[0004] In view of the problems in the prior art, the present application provides a multi-modal target recognition method based on modal contribution degree evaluation, aiming to solve the problem that the existing target recognition model based on traditional multi-modal fusion performs static reasoning on the feedforward process of different modalities, thereby ignoring the imbalance problem existing between different modalities.

[0005] A multi-modal target recognition method based on modal contribution degree evaluation, comprising the following steps:

[0006] Step 1: Construct and obtain a trained multi-modal target recognition network, which comprises a feature filtering module and a modal contribution degree evaluation module;

[0007] Step 2: Obtain the infrared image and the visible light image of the target;

[0008] Step 3: Preprocess the infrared image and the visible light image by using a data enhancement technique to obtain infrared image input features and visible light image input features with the same feature dimension;

[0009] Step 4: Extract shallow basic features from the infrared image input features and the visible light image input features respectively through the feature filtering module, and extract infrared high-frequency features and visible light high-frequency features from the corresponding shallow basic features respectively;

[0010] Step 5: Calculate the confidence and energy value of the infrared high-frequency features and the visible light high-frequency features respectively through the modal contribution degree evaluation module;

[0011] Step 6: Calculate the fusion weight using the normalized confidence and the normalized energy value;

[0012] Step 7: Obtain the final target recognition result according to the fusion weight.

[0013] Further, the feature filtering module comprises a long-distance feature extractor and a short-distance feature extractor, and step 4 is specifically:

[0014] Step 4.1: The long-distance feature extractor extracts shallow basic features from the infrared image input features or the visible light image input features by using a ResNet feature extraction network;

[0015] Step 4.2: The short-distance feature extractor extracts infrared high-frequency features and visible light high-frequency features from the corresponding shallow basic features by using a Dense-KAN module;

[0016] Further, the Resnet feature extraction network comprises a plurality of convolutional layers and pooling layers, which are used to gradually abstract image information.

[0017] Further, the Dense-KAN module is composed of a series of KAN layers connected densely, and each KAN layer is composed of a series of connected Kolmogorov-Arnold layers, and each Kolmogorov-Arnold layer contains a set of learnable one-dimensional activation functions.

[0018] Further, the modal contribution evaluation module includes a confidence estimation branch and a modal energy value evaluation module, and step 5 is specifically:

[0019] Step 5.1: Calculate the confidence of the infrared high-frequency feature or the visible light high-frequency feature through the confidence estimation branch.

[0020] Step 5.2: Calculate the energy value of the infrared high-frequency feature or the visible light high-frequency feature through the modal energy value evaluation module.

[0021] Further, the confidence estimation branch contains a fully connected layer, and outputs a single scalar between 0 and 1.

[0022] Further, the modal energy value evaluation module calculates the energy value of the infrared high-frequency feature or the visible light high-frequency feature is: ;

[0023] wherein, is a temperature parameter, is the logit of the classifier output corresponding to the kth class label, is the input feature of the mth modality, is the set of logits of the mth modality classifier output.

[0024] Further, step 6 is specifically: the fusion weight is:

[0025]

[0026] wherein, and are the normalized confidence of the infrared high-frequency feature and the normalized confidence of the visible light high-frequency feature, respectively, and are the normalized energy value of the infrared high-frequency feature and the normalized energy value of the visible light high-frequency feature, respectively.

[0027] Further, step 7 is specifically:

[0028] Step 7.1: Calculate the final confidence ;

[0029]

[0030] wherein, is a fusion weight, and respectively are the confidence of the infrared high-frequency feature and the confidence of the visible light high-frequency feature;

[0031] Step 7.2: Calculate the target recognition result

[0032]

[0033] Wherein, argmax represents the argument maximization function.

[0034] The beneficial effects of the present application: through the function of the feature filtering module to carry out accurate screening and nonlinear enhancement processing on the multi-modal original features, overcome the shortcomings when dealing with complex nonlinear features, fully excavate and utilize different modal information, effectively suppress irrelevant interference, optimize the quality from the feature input layer, enhance the capture and expression ability of the model to effective features, and enhance the robustness; with the help of the modal contribution evaluation module, the support value of each modal in the current sample recognition for classification decision is analyzed in real time, the modal contribution weight is quantified, and sample-level modal dynamic fusion recognition is realized; finally, with the synergistic mechanism of "feature filtering module purifies multi-modal effective information to lay a foundation, and modal contribution evaluation module accurately determines the value of the modal according to the information and dynamically fuses", the problem of uneven contribution of modal information in multi-modal target recognition is solved - breaking the traditional fixed fusion or single-modal dominant limitation, making each modal reasonably exert in the adaptive scene, fundamentally optimizing information distribution, and improving recognition accuracy and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flowchart of the present application;

[0036] Figure 2 is a processing process diagram of data in a certain module;

[0037] Figure 3 is a structural frame of a certain module in the present application. DETAILED DESCRIPTION

[0038] The present application will be described in detail below with reference to the accompanying drawings. The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application. The left, middle, right, up, down and other orientation terms in the embodiments of the present application are only relative concepts or are with reference to the normal use state of the product, and should not be considered as limiting.

[0039] ​A multi-modal target recognition method based on modal contribution degree evaluation, comprising the following steps:

[0040] Step 1: Construct and obtain a trained multi-modal target recognition network, which includes a feature filtering module and a modal contribution degree evaluation module;

[0041] Step 2: Obtain the infrared image and the visible light image of the target;

[0042] Step 3: Preprocess the infrared image and the visible light image by using a data enhancement technique to obtain infrared image input features and visible light image input features with the same feature dimension; the data enhancement technique includes horizontal flipping, vertical flipping and arbitrary angle rotation:

[0043]

[0044] wherein, and represent the original infrared image and the visible light image, and represent the infrared image and the visible light image processed by the data enhancement technique, and have the same feature dimension, respectively represent the channel number, length and width of the image;

[0045] Step 4: Extract shallow basic features from the infrared image input features and the visible light image input features by the feature filtering module, and extract infrared high frequency features and visible light high frequency features from the corresponding shallow basic features; the feature filtering module includes a long distance feature extractor and a short distance feature extractor, specifically:

[0046] Step 4.1: Considering that the existing multi-modal fusion recognition method has shortcomings in processing complex nonlinear features, it is difficult to fully utilize the information of different modalities, the long distance feature extractor adopts a ResNet feature extraction network to extract shallow basic features from the infrared image input features or the visible light image input features; the Resnet feature extraction network contains multiple convolution layers and pooling layers, which are used to gradually abstract image information; the long distance feature extractor can convert the input original image into a multi-scale, high-dimensional feature representation, providing rich semantic information for subsequent feature fusion and discrimination;

[0047] Step 4.2: The short-distance feature extractor extracts infrared high-frequency features and visible light high-frequency features from the corresponding shallow basic features using a Dense-KAN module. The Dense-KAN module is composed of a series of KAN layers connected in a dense manner. Each KAN layer is composed of a series of connected Kolmogorov-Arnold layers, and each Kolmogorov-Arnold layer contains a set of learnable one-dimensional activation functions. The short-distance feature extractor can effectively capture high-frequency features, alleviate the gradient vanishing problem, and improve the robustness and accuracy of multi-modal data fusion.

[0048] where a series of KAN layers can be represented as a nesting of i KAN layers, as follows:

[0049]

[0050] represents the i-th layer of the entire KAN network, each KAN layer has dimensional input and dimensional output, is the initial input feature vector of the KAN network, and Φ contains × learnable activation functions:

[0051]

[0052] where is the feature mapping between and , and to avoid the loss of key information during network training, a Dense-KAN module is designed to help each KAN layer better utilize the output of the previous layers, thereby further enriching the feature representation capability of the model, as follows:

[0053] For the i-th layer, its input is the concatenation of the outputs of all previous layers :

[0054]

[0055] where represents the concatenation of the outputs of each KAN layer; represents the output of the l-th KAN layer in the Dense-KAN module;

[0056] The output of the i-th layer is calculated by its Kolmogorov-Arnold module:

[0057] ​​

[0058] wherein, represents the non-linear transformation represented by the th Kolmogorov-Arnold module;

[0059] each layer of the KAN layer is densely connected, and the final output of each modality after the Dense-KAN module processing is:

[0060]

[0061] wherein, is the final output of the Dense-KAN module, wherein , and represent the infrared modality and the visible light modality, respectively;

[0062] Step 5: Calculate the confidence and energy value of the infrared high-frequency feature and the visible light high-frequency feature respectively through the modality contribution degree evaluation module; the modality contribution degree evaluation module includes a confidence estimation branch and a modality energy value evaluation module, specifically:

[0063] Step 5.1: Calculate the confidence of the infrared high-frequency feature or the visible light high-frequency feature through the confidence estimation branch; the confidence estimation branch contains a fully connected layer, and outputs a confidence estimation value between 0 and 1 ;

[0064] wherein, w is the weight vector of the fully connected layer in the confidence estimation branch, b is the bias term, is an activation function, and here a Sigmoid function is adopted;

[0065] Step 5.2: Calculate the energy value of the infrared high-frequency feature or the visible light high-frequency feature through the modality energy value evaluation module; the modality energy value evaluation module calculates the energy value of the infrared high-frequency feature or the visible light high-frequency feature is: ;

[0066] wherein, is a temperature parameter, is the logit of the classifier output corresponding to the kth class label, is the input feature of the mth modality, is the logit set of the mth modality classifier output, corresponding to the classification score of the single modality feature;

[0067] Step 6: Calculate the fusion weight using the normalized confidence and the normalized energy value; wherein the fusion weight is:

[0068]

[0069] wherein, and are the normalized confidence of the infrared high-frequency feature and the normalized confidence of the visible light high-frequency feature, respectively, and are the normalized energy value of the infrared high-frequency feature and the normalized energy value of the visible light high-frequency feature, respectively;

[0070] Step 7: obtaining the final target recognition result according to the fusion weight, specifically:

[0071] Step 7.1: calculating the last confidence ;

[0072]

[0073] wherein, is the fusion weight, and are the confidence of the infrared high-frequency feature and the confidence of the visible light high-frequency feature, respectively;

[0074] Step 7.2: calculating the target recognition result ;

[0075]

[0076] wherein, argmax represents the argument maximization function.

[0077] In addition, considering that this modal contribution is not enough to give an accurate estimate without additional regularization, a regularization method based on historical trajectory information is designed to improve the correlation between the contribution score of the modal and the corresponding loss. Specifically, a regularization method using historical trajectory information is designed, and the sample loss in the training stage is used as supervision information; thus, the historical loss function is used to regularize the fusion weight, and the m modal sample of a given sample is the average loss of training:

[0078]

[0079] wherein, is the single-peak classifier of the m modal at the iteration time t, is the current learnable parameter of the classifier; The classification loss function here refers to the cross-entropy loss function; after training Ts-1 epoch, sample T times and calculate the average training loss; and the fusion weight is determined by the following relationship:

[0080]

[0081] wherein, and are the fusion weights of the m-th modality, the smaller the average loss is, the higher the weight allocated to the corresponding modality in the fusion is, realizing the sample-level modality contribution adaptation;

[0082] The complete regularization is defined as follows:

[0083]

[0084] wherein, represents a weight relationship indication function, according to and the size relationship, the penalty direction of the regularization loss is controlled, and both represent the modality predefined confidence threshold, and the rationality of the weight difference is constrained;

[0085] Finally, the total loss function is defined as the sum of the cross-entropy loss of each modality and the regularization term:

[0086]

[0087] wherein, is a hyperparameter for controlling the regularization strength, and respectively represent the final prediction function after multi-modal fusion and the prediction function of a single modality, and are the cross-entropy loss and the regularization term respectively.

[0088] Compared with the traditional deep learning multi-modal fusion recognition method, the model of the present application increases the explainability, and provides a sample-level solution for the unbalanced multi-modal scene.

[0089] The above shows and describes the basic principles, main features and advantages of the present application. It should be understood by those skilled in the art that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-modal target identification method based on modal contribution degree evaluation, characterized in that: The method comprises the following steps: Step 1: constructing and obtaining a trained multi-modal target recognition network, the multi-modal target recognition network comprising a feature filtering module and a modal contribution degree evaluation module; Step 2: obtaining an infrared image and a visible light image of a target; Step 3: pre-processing the infrared image and the visible light image by using a data enhancement technology to obtain an infrared image input feature and a visible light image input feature with the same feature dimension; Step 4: extracting shallow basic features in the infrared image input feature and the visible light image input feature respectively by the feature filtering module, and extracting infrared high-frequency features and visible light high-frequency features from the corresponding shallow basic features respectively; Step 5: calculating confidence and energy values of the infrared high-frequency features and the visible light high-frequency features respectively by the modal contribution degree evaluation module; Step 6: calculating a fusion weight using the normalized confidence and the normalized energy value; Step 7: obtaining a final target recognition result according to the fusion weight.

2. The multi-modal target recognition method based on modal contribution degree evaluation according to claim 1, characterized in that: The feature filtering module comprises a long-distance feature extractor and a short-distance feature extractor, and step 4 is specifically as follows: Step 4.1: the long-distance feature extractor extracts shallow basic features from the infrared image input feature or the visible light image input feature by using a ResNet feature extraction network; Step 4.2: the short-distance feature extractor extracts the infrared high-frequency features and the visible light high-frequency features from the corresponding shallow basic features by using a Dense-KAN module.

3. The multi-modal target recognition method based on modal contribution degree evaluation according to claim 2, characterized in that: The ResNet feature extraction network comprises a plurality of convolutional layers and pooling layers, and is used for gradually abstracting image information.

4. The multi-modal target recognition method based on modal contribution degree evaluation according to claim 2, characterized in that: The Dense-KAN module is composed of a series of KAN layers connected in a dense manner, and each KAN layer is composed of a series of connected Kolmogorov-Arnold layers, and each Kolmogorov-Arnold layer comprises a group of learnable one-dimensional activation functions.

5. The method of claim 1, wherein: The modal contribution degree evaluation module comprises a confidence estimation branch and a modal energy value evaluation module, and step 5 is specifically as follows: Step 5.1: calculating the confidence of the infrared high-frequency features or the visible light high-frequency features by the confidence estimation branch; Step 5.2: calculating the energy value of the infrared high-frequency features or the visible light high-frequency features by the modal energy value evaluation module.

6. The multi-modal target recognition method based on modal contribution degree evaluation according to claim 5, characterized in that: The confidence estimation branch comprises a fully connected layer, and outputs a single scalar between 0 and 1.

7. The multi-modal target recognition method based on modal contribution degree evaluation according to claim 5, characterized in that: The modal energy value evaluation module calculates the energy value of the infrared high-frequency feature or the visible light high-frequency feature To: ; wherein, is a temperature parameter, is a logit of the classifier output corresponding to the k-th class label, is an input feature of the m-th modality, is a set of logits of the m-th modality classifier output.

8. The method of claim 1, wherein: Step 6 is specifically: fusion weights is: ； wherein, and are the normalized confidence of the infrared high frequency feature and the normalized confidence of the visible light high frequency feature, respectively, and are the normalized energy value of the infrared high frequency feature and the normalized energy value of the visible light high frequency feature, respectively.

9. The method of claim 1, wherein: Step 7 is specifically as follows: Step 7.1: Calculate final confidence ; ； wherein, is a fusion weight, and are the confidence of the infrared high-frequency feature and the confidence of the visible light high-frequency feature, respectively. Step 7.2: Calculate target recognition result ; ； Wherein, argmax represents the maximum function of the independent variable.

Images