An aerial target recognition method based on bootstrap multi-modal learning
By using a bootstrap multimodal learning method to generate noisy associated samples and perform reliability assessment during the training phase, and then performing weighted fusion during the testing phase, the problem of noise interference in multimodal recognition is solved, and the robustness and accuracy of aerial target recognition are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN UNIV
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-02
AI Technical Summary
Existing multimodal recognition methods are susceptible to noise interference during the testing phase, leading to fluctuations in recognition results and performance degradation, especially in complex environments where it is difficult to maintain robustness and accuracy.
A multimodal aerial target training dataset is constructed, and a bootstrap multimodal learning method is adopted. During the training phase, noisy association samples are generated through modal masks, and modal reliability assessment and classification capabilities are jointly learned. During the testing phase, reliability weighted fusion is performed to suppress the interference of noise association on decision-making.
The robustness and recognition rate of aerial target recognition are improved. By simulating noise correlation during the training phase and performing reliability assessment and weighted fusion during the testing phase, the impact of noise is reduced, thereby improving the stability and accuracy of the recognition results.
Smart Images
Figure CN122135248A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of aerial target recognition and multimodal intelligent perception, specifically to an aerial target recognition method based on bootstrapping multimodal learning. Background Technology
[0002] In recent years, with the rapid development of applications such as UAV monitoring, airborne early warning, and low-altitude security, the actual deployment scale of aerial target recognition in complex environments has been continuously expanding, placing higher demands on robustness and accuracy. Aerial targets often have characteristics such as small scale, high maneuverability, large attitude changes, and abundant background clutter. Single-modal sensing is prone to insufficient effective information and inadequate target representation under conditions of changing lighting, deteriorating weather, long-distance observation, and strong interference. To improve recognition performance, research and engineering practice have gradually adopted multimodal sensing schemes, integrating information from multiple sources such as visible light, infrared, and radar to enhance the completeness of target representation and improve recognition stability.
[0003] Existing multimodal recognition methods typically learn based on the assumption of consistent and correct association between modalities during the training phase, and directly perform feature fusion or decision fusion during the testing phase. However, in real-world deployment environments, test data is prone to cross-modal noise associations, such as temporal mismatches caused by asynchronous multi-sensor operation, spatial registration errors, erroneous responses caused by sensor interference, or miscorrespondences of different modalities to different targets, resulting in a lack of strict alignment of multimodal information for the same target. In such cases, if a fixed or equal-weighted fusion strategy is still adopted, the model is prone to introducing incorrectly associated modal information into the decision-making process, interfering with the fusion results and leading to false detections, false negatives, or category confusion, thus causing fluctuations in recognition results and performance degradation. Therefore, it is necessary to simulate and inject noisy associated samples in a controlled manner during the training phase, and to clearly distinguish between noisy and clean samples. Based on this, modal reliability assessment and robust fusion mechanisms should be learned to improve the robustness and recognition rate of the model when encountering noisy associations during the testing phase. Summary of the Invention
[0004] To address the aforementioned shortcomings in existing technologies, this invention provides an aerial target recognition method based on bootstrapping multimodal learning. This method solves the problems of traditional multimodal recognition models being susceptible to interference, experiencing fluctuations in recognition results, and degrading performance in noisy correlation scenarios during the testing phase, thereby improving the robustness and recognition rate of aerial target recognition.
[0005] To achieve the aforementioned objectives, the present invention employs the following technical solution: an aerial target recognition method based on bootstrapping multimodal learning, comprising the following steps: S1. Construct a training dataset containing multimodal aerial target samples and their category labels, and set up a model structure for feature extraction, target classification and modal reliability evaluation for each modality; S2. During the training process, some samples are subjected to bootstrapping enhancement. Based on the noise correlation constraints, modal perturbation and combination schemes are generated for the samples to form an enhanced training set for revealing supervision. S3. Extract the representation of each modality and obtain the classification output. At the same time, characterize the prediction differences between modalities and the uncertainty within modalities, and generate a reliability score that represents the credibility of the modality. S4. Jointly optimize the classification learning objective and the reliability constraint objective to suppress the interference of noise association on the learning process and update the model parameters; S5. During the testing phase, extract the features of each modality and obtain the corresponding reliability score. Perform reliability weighted fusion on the classification output of each modality and output the final aerial target recognition result. Further, step S1 includes the following sub-steps: S11. Construct a training dataset containing multimodal (taking visible light images, infrared images, and messages as examples) aerial target samples and their category labels, denoted as the training dataset. , Depend on The sample consists of n samples, where the nth sample is... Each sample is denoted as Its category label .
[0006] S12. Set up the model structure for feature extraction, target classification and modal reliability assessment of each modality, so as to form an overall model framework for trimodal aerial target recognition.
[0007] Furthermore, in step S11, It contains data in three modalities, namely visible light images. Infrared images and message text Each sample corresponds to a unique category label. This is used to characterize the category information of the aerial target.
[0008] The beneficial effects of the above-mentioned further solutions are as follows: 1. By organizing visible light images, infrared images, and message text into a unified multimodal training dataset with category labels, it is easier to perform consistent modeling and description of samples, modalities, and labels in the future.
[0009] Furthermore, in step S12, the visible light and infrared modes use ResNet18 pre-trained on ImageNet as the backbone network, denoted as follows: and The message modality uses a pre-trained model BERT as the backbone network, denoted as... Furthermore, separate single-modal target classifiers were set up. , and This is used to map each modal feature to a classification output and to set up a modal reliability evaluator. , and This is used to generate reliability scores for each mode.
[0010] The beneficial effects of the above-mentioned further solutions are as follows: 1. By employing a ResNet18 backbone network for image modality and a Bert backbone network for text modality, and configuring corresponding single-modality classifiers and reliability evaluators, independent representation, classification, and reliability evaluation of each modality are achieved, providing a foundation for subsequent reliability-weighted fusion.
[0011] Furthermore, step S2 includes the following sub-steps: S21. At the beginning of each round of training, from the training dataset... S22. A subset of bootstrap sample indices is obtained by sampling, and modal masks that satisfy noise association constraints are generated for the samples in the subset. S23. Based on the modal masks, the selected samples are subjected to shuffling and replacement of the same modality within the subset to construct noise-associative samples, and these samples are summarized together with their masks to form a bootstrap augmented dataset.
[0012] Furthermore, in step S21, in the first... At the start of each round of training, a clean training set is provided: in, Indicates the sample index. Indicates modal index, Represents the training dataset. This represents the first of three modalities of data. One sample, Indicates the first Category labeling for each sample, Indicates the size of the dataset. Indicates the number of modes. Indicates the first The first sample The input consists of several modalities. First, a subset of indices is sampled from the training dataset: in, Indicates a subset of the sampling index. Indicates the size of the dataset. Indicates the proportion of self-increased growth. This indicates the dataset size operator. This represents the floor operation. Then, for each sample in the sampled index subset... Generate a modal mask: in, Indicates the sample index. Indicates modal index, Indicates sample modal mask, Indicates the first The first sample A mask for each modality. This represents the number of modes. Among them, Noise association constraints should be met: in, Indicates the sample index. Indicates modal index, Indicates the number of modes. Indicates the first The first sample A mask for each modality. This represents the floor operation.
[0013] The beneficial effects of the above-mentioned further solutions are as follows: 1. By generating modal masks under constraints, controllable combinations of noise associations can be stably constructed to reduce the training-test discrepancy while ensuring that samples still retain at least one clean modal information.
[0014] Furthermore, in step S22, based on the modal mask... The samples are shuffled and replaced with identical modalities within a subset to obtain bootstrapping augmented samples. The rules are as follows in, and Both represent sample indices. Indicates modal index, Indicates the first A self-bootstrapping augmentation sample, Indicates the first The first sample The form after modal bootstrapping augmentation Represents the first in the original dataset One sample, Represents the first in the original dataset One sample, Indicates the first The first sample A mask for each modality. This represents a subset of the sampled indices. When hour, from Internal random sampling is used to achieve random shuffling and replacement of this mode. Sample preservation and order The final result is a bootstrap augmented training set that reveals supervisory information: in, Indicates the sample index. This represents the bootstrap augmented training set obtained, which reveals supervisory information. Indicates the first A self-bootstrapping augmentation sample, Indicates the first Category labeling for each sample, Indicates sample modal mask, Indicates the size of the dataset.
[0015] The beneficial effects of the above-mentioned further solutions are as follows: 1. By randomly shuffling and replacing samples of the same modality within a subset and explicitly retaining the mask label, noisy associated samples can be controlled and labeled during the training phase, providing direct supervision for subsequent reliability assessment and robust fusion.
[0016] Furthermore, step S3 includes the following sub-steps: S31. Extract modal representations using a single-modal backbone network, and obtain classification outputs for each modality using the corresponding single-modal target classifier. S32. Characterize inter-modal prediction differences based on Jeffreys divergence for each modality, and characterize intra-modal uncertainty based on prediction entropy for each modality. S33. Combine modal representations, inter-modal prediction differences, and intra-modal uncertainties into modal evidence, and output a reliability score characterizing the credibility of the modality using a modal reliability evaluator. S34. Generate fusion weights based on the reliability scores of each modality, and perform reliability-weighted fusion of the classification outputs of each modality to obtain the final classification output.
[0017] Furthermore, in step S31, given the bootstrap augmented training set... , for the The first sample Each modality is represented by a single-modal backbone network, from which modal characteristics are extracted and distributions are predicted. in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Features of each modality Indicates the first A modal encoder, Indicates the first The form after bootstrapping of individual samples Indicates the first Encoder parameters for each mode, Indicates the first The first sample The classification logits of each modality Indicates the first A classification head for each modality, Indicates the first Classification header parameters for each modality, Indicates the first The first sample The predicted distribution of each modality This represents the softmax operator function.
[0018] The beneficial effects of the above-mentioned further solutions are as follows: 1. The representations and outputs of each modality are obtained through a backbone network and a single-modal classifier, providing a unified input basis for subsequent modeling of prediction differences, uncertainties and reliability.
[0019] Furthermore, in step S32, the distribution is predicted based on each modality. Calculate the prediction difference between modes And calculate the intramodal uncertainty based on the prediction entropy of each mode. : in, Indicates the sample index. and Both represent modal indices. Indicates the first The first sample Jeffreys divergence for each mode, Indicates the number of modes. This represents the Kullback-Leibler divergence operator function. Indicates the first The first sample The predicted distribution of each modality, Indicates the first The first sample The predicted distribution of each modality, Indicates the first The first sample Intramodal uncertainty of a mode, Indicates the first The first sample The prediction entropy of each modality Indicates the number of categories in the dataset. Indicates the first The first sample The first mode The predicted probability of each category.
[0020] The beneficial effects of the above-mentioned further solutions are as follows: 1. By simultaneously characterizing cross-modal prediction differences and single-modal confidence levels, complementary clues can be provided for subsequent reliability assessments to more accurately distinguish between credible modes and noise-related modes.
[0021] Furthermore, in step S33, the modal representation, inter-modal prediction differences, and intra-modal uncertainties are concatenated to form a modal evidence vector: in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Modal evidence vectors for each modality Indicates the first The first sample Jeffreys divergence for each mode, Indicates the first The first sample The intra-modal uncertainty of each mode is then determined. The modal evidence vector is then input into the modal reliability estimator to obtain the modal reliability score. in, Indicates the sample index. Indicates modal index, Indicates the first The first sample The reliability of each mode This represents the sigmoid operator function. Indicates the first A reliability estimator for each mode. No. The first sample A modal evidence vector representing each modality. Indicates the first Reliability estimator parameters for each mode.
[0022] The beneficial effects of the above-mentioned further solutions are as follows: 1. By jointly inputting the quality signals from the characterization and prediction levels into the reliability evaluator, a score directly corresponding to modal credibility can be obtained, thus providing an interpretable and usable weighting basis for the fusion stage.
[0023] Furthermore, in step S34, the prediction distributions of each modality are weighted and fused based on the reliability scores of each modality to obtain the final prediction distribution: in, Indicates the sample index. Indicates modal index, No. The predicted distribution of each sample Indicates the number of modes. Indicates the first The first sample The reliability of each mode Indicates the first The first sample The classification logits of each modality, This represents the softmax operator function.
[0024] The beneficial effects of the above-mentioned further solutions are as follows: 1. By adaptively allocating the contribution of each modality using a reliability score, the impact of noise-related modalities on decision-making can be reduced and the stability of the final recognition output can be improved.
[0025] Further, step S4 includes the following sub-steps: S41, constructing a classification learning objective based on the fusion prediction distribution obtained in step S3, used to supervise the identification of aerial target categories. S42, constructing a reliability constraint objective using known noise labels from the bootstrapping augmentation process, used to supervise the consistency between the output of the modal reliability estimator and the modal cleanliness indication. S43, weightedly combining the classification learning objective and the reliability constraint objective to obtain the overall optimization objective and updating the model parameters end-to-end.
[0026] Further, in step S41, for the batch sampled from the bootstrap augmented training set, its classification learning objective is defined as: in, This represents the classification cross-entropy of this batch. This represents the set of batch indices sampled from the bootstrap augmented training set. This indicates the dataset size operator. Indicates the sample index. Indicates the first The predicted category for each sample, Indicates the first The predicted category for each sample is The probability of.
[0027] The beneficial effects of the above-mentioned further solutions are as follows: 1. By applying a supervised classification objective to the fusion prediction results, the model can directly optimize the final recognition performance and improve the ability to distinguish aerial target categories.
[0028] Further, in step S42, for the batch sampled from the bootstrap augmented training set, its reliability constraint objective is defined as: in, This indicates the reliability alignment loss for this batch. This represents the set of batch indices sampled from the bootstrap augmented training set. This indicates the dataset size operator. Indicates the sample index. Indicates modal index, Indicates the number of modes. Indicates the first The first sample A mask for each modality. Indicates the first The first sample Reliability of each mode.
[0029] The beneficial effects of the above-mentioned further solutions are as follows: 1. By applying consistency constraints between reliability scores and known noise masks, the reliability evaluator can learn the discrimination rules of high reliability for clean modes and low reliability for disturbed modes, thereby providing a reliable weight basis for robust fusion.
[0030] Furthermore, in step S43, the classification learning objective and the reliability constraint objective are weighted and jointly combined to obtain the total loss function: in, This indicates the overall optimization goal. This represents the classification cross-entropy of this batch. This represents the reliability alignment loss for this batch. By jointly optimizing the classification objective and the reliability constraint objective, the backbone network for each modality is optimized. Single-modal classifier and modal reliability evaluator parameters Perform an end-to-end update.
[0031] The beneficial effects of the above-mentioned further solutions are as follows: 1. By jointly optimizing the classification objective and the reliability constraint objective, the model can learn an available reliability assessment mechanism while improving recognition accuracy, thereby suppressing the interference of noise association on the learning process and enhancing overall robustness.
[0032] Furthermore, step S5 includes the following sub-steps: S51. During the testing phase, extract features for each modality and obtain classification outputs for each modality. Simultaneously, construct modal evidence and obtain corresponding reliability scores. S52. Perform reliability-weighted fusion of the classification outputs for each modality based on their reliability scores to obtain a fused output. S53. Determine the final predicted category based on the fused output and output the final aerial target recognition result.
[0033] Further, in step S51, given a test sample , for the First, extract the representation of each modality and obtain the predicted distribution: in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Feature representation of each modality Indicates the first A backbone network of multiple modalities, Indicates the first The first sample Input data for each modality, Indicates the first Encoder parameters for each mode, Indicates the first The first sample The classification logits of each modality Indicates the first A classification head for each modality, Indicates the first The classification head parameters for each modality are then used. The predicted distribution is then... Further calculation of the prediction differences between modes Intramodal uncertainty And construct modal evidence: in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Modal evidence vectors for each modality Indicates the first The first sample Features of each modality Indicates the first The first sample Intermodal prediction differences among modalities No. The first sample The intra-modal uncertainty of each mode. A reliability score is obtained using a modal reliability estimator. in, Indicates the sample index. Indicates modal index, Indicates the first The first sample The reliability of each mode This represents the sigmoid operator function. Indicates the first A reliability estimator for each mode. Indicates the first The first sample Modal evidence vectors for each modality Indicates the first Reliability estimator parameters for each mode.
[0034] The beneficial effects of the above-mentioned further solutions are as follows: 1. By simultaneously calculating the classification output and reliability score of each modality during the testing phase, the reliability of each modality can be dynamically evaluated without explicit noise labeling.
[0035] Further, in step S52, the modal classification outputs are reliably weighted and fused based on the modal reliability scores to obtain fused logits: in, Indicates the sample index. Indicates modal index, This represents the logits after weighted fusion of various modalities. Indicates the number of modes. Indicates the first The first sample The reliability of each mode Indicates the first The first sample Logits of each modality.
[0036] The beneficial effects of the above-mentioned further solutions are as follows: 1. Reliability-weighted fusion can reduce the contribution of noisy correlated modes to the fusion results, thereby suppressing the interference of erroneous mode information on decision-making.
[0037] Further, in step S53, the final predicted category is output based on the fused logits: in, Indicates the final predicted category. Indicates the current category label. Indicates the total number of categories. Indicates the first The sample at the th Logits on each category.
[0038] The beneficial effects of the above-mentioned further solutions are as follows: 1. By determining the final category based on the maximum response of the fused output, multimodal information can be uniformly converged into a single stable decision under reliability constraints, thereby improving the consistency and accuracy of aerial target recognition results.
[0039] In summary, the beneficial effects of this invention are as follows: By constructing a multimodal aerial target training dataset and establishing an overall framework that includes feature extraction, classification, and reliability assessment, this invention controls the injection and labeling of noisy associated samples in a bootstrap manner during the training phase, jointly learning classification capabilities and modal reliability assessment capabilities. During the testing phase, the outputs of each modality are weighted and fused based on the reliability score to suppress the interference of noise association on the recognition decision. This solves the problem of unstable recognition results and decreased accuracy caused by cross-modal noise association during the testing phase, thereby improving the robustness and recognition rate of aerial target recognition. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. It should be understood that the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 The flowchart of this invention Detailed Implementation The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Figure 1 As shown, an aerial target recognition method based on bootstrapping multimodal learning includes the following steps: S1. Construct a training dataset containing multimodal aerial target samples and their category labels, and set up a model structure for feature extraction, target classification and modal reliability evaluation for each modality; Step S1 includes the following sub-steps: S11. Construct a training dataset containing multimodal (taking visible light images, infrared images, and messages as examples) aerial target samples and their category labels, denoted as the training dataset. , Depend on The sample consists of n samples, where the nth sample is... Each sample is denoted as Its category label . It contains data in three modalities, namely visible light images. Infrared images and message text Each sample corresponds to a unique category label. This is used to characterize the category information of the aerial target.
[0042] S12. Define the model structure for feature extraction, target classification, and modal reliability assessment for each modality, thus forming an overall model framework for trimodal aerial target recognition. The visible light and infrared modalities use a ResNet18 pre-trained on ImageNet as the backbone network, denoted as follows: and The message modality uses a pre-trained model BERT as the backbone network, denoted as... Furthermore, separate single-modal target classifiers were set up. , and This is used to map each modal feature to a classification output and to set up a modal reliability evaluator. , and This is used to generate reliability scores for each mode.
[0043] S2. During the training process, some samples are subjected to bootstrapping enhancement. Based on the noise correlation constraints, modal perturbation and combination schemes are generated for the samples to form an enhanced training set for revealing supervision. Step S2 includes the following sub-steps: S21, in the At the start of each round of training, a clean training set is provided: in, Indicates the sample index. Indicates modal index, Represents the training dataset. This represents the first of three modalities of data. One sample, Indicates the first Category labeling for each sample, Indicates the size of the dataset. Indicates the number of modes. Indicates the first The first sample The input consists of several modalities. First, a subset of indices is sampled from the training dataset: in, Indicates a subset of the sampling index. Indicates the size of the dataset. Indicates the proportion of self-increased growth. This indicates the dataset size operator. This represents the floor operation. Then, for each sample in the sampled index subset... Generate a modal mask: in, Indicates the sample index. Indicates modal index, Indicates sample modal mask, Indicates the first The first sample A mask for each modality. This represents the number of modes. Among them, Noise association constraints should be met: in, Indicates the sample index. Indicates modal index, Indicates the number of modes. Indicates the first The first sample A mask for each modality. This represents the floor operation. S22, based on the modal mask. The samples are shuffled and replaced with identical modalities within a subset to obtain bootstrapping augmented samples. The rules are as follows in, and Both represent sample indices. Indicates modal index, Indicates the first A self-bootstrapping augmentation sample, Indicates the first The first sample The form after modal bootstrapping augmentation Represents the first in the original dataset One sample, Represents the first in the original dataset One sample, Indicates the first The first sample A mask for each modality. This represents a subset of the sampled indices. When hour, from Internal random sampling is used to achieve random shuffling and replacement of this mode. Sample preservation and order The final result is a bootstrap augmented training set that reveals supervisory information: in, Indicates the sample index. This represents the bootstrap augmented training set obtained, which reveals supervisory information. Indicates the first A self-bootstrapping augmentation sample, Indicates the first Category labeling for each sample, Indicates sample modal mask, Indicates the size of the dataset.
[0044] S3. Extract the representation of each modality and obtain the classification output. At the same time, characterize the prediction differences between modalities and the uncertainty within modalities, and generate a reliability score that represents the credibility of the modality. Step S3 includes the following sub-steps: S31. Given a bootstrap augmented training set , for the The first sample Each modality is represented by a single-modal backbone network, from which modal characteristics are extracted and distributions are predicted. in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Features of each modality Indicates the first A modal encoder, Indicates the first The form after bootstrapping of individual samples Indicates the first Encoder parameters for each mode, Indicates the first The first sample The classification logits of each modality Indicates the first A classification head for each modality, Indicates the first Classification header parameters for each modality, Indicates the first The first sample The predicted distribution of each modality, This represents the softmax operator function.
[0045] S32, Based on the prediction distribution of each modality Calculate the prediction difference between modes And calculate the intramodal uncertainty based on the prediction entropy of each mode. : in, Indicates the sample index. and Both represent modal indices. Indicates the first The first sample Jeffreys divergence for each mode, Indicates the number of modes. This represents the Kullback-Leibler divergence operator function. Indicates the first The first sample The predicted distribution of each modality, Indicates the first The first sample The predicted distribution of each modality, Indicates the first The first sample Intramodal uncertainty of a mode, Indicates the first The first sample The prediction entropy of each modality Indicates the number of categories in the dataset. Indicates the first The first sample The first mode The predicted probability of each category.
[0046] S33. Concatenate modal representations, intermodal prediction differences, and intramodal uncertainties to form a modal evidence vector: in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Modal evidence vectors for each modality Indicates the first The first sample Jeffreys divergence for each mode, Indicates the first The first sample The intra-modal uncertainty of each mode is then determined. The modal evidence vector is then input into the modal reliability estimator to obtain the modal reliability score. in, Indicates the sample index. Indicates modal index, Indicates the first The first sample The reliability of each mode This represents the sigmoid operator function. Indicates the first A reliability estimator for each mode. No. The first sample A modal evidence vector representing each modality. Indicates the first Reliability estimator parameters for each mode.
[0047] S34. Based on the reliability scores of each modality, the prediction distributions of each modality are weighted and fused to obtain the final prediction distribution: in, Indicates the sample index. Indicates modal index, No. The predicted distribution of each sample Indicates the number of modes. Indicates the first The first sample The reliability of each mode Indicates the first The first sample The classification logits of each modality This represents the softmax operator function.
[0048] S4. Jointly optimize the classification learning objective and the reliability constraint objective to suppress the interference of noise association on the learning process and update the model parameters; Step S4 includes the following sub-steps: Further, step S4 includes the following sub-steps: S41, For a batch sampled from the bootstrap augmented training set, its classification learning objective is defined as: in, This represents the classification cross-entropy of this batch. This represents the set of batch indices sampled from the bootstrap augmented training set. This indicates the dataset size operator. Indicates the sample index. Indicates the first The predicted category for each sample, Indicates the first The predicted category for each sample is The probability of.
[0049] S42. For batches sampled from the bootstrap augmented training set, the reliability constraint objective is defined as: in, This indicates the reliability alignment loss for this batch. This represents the set of batch indices sampled from the bootstrap augmented training set. This indicates the dataset size operator. Indicates the sample index. Indicates modal index, Indicates the number of modes. Indicates the first The first sample A mask for each modality. Indicates the first The first sample Reliability of each mode.
[0050] S43. The total loss function is obtained by weightedly combining the classification learning objective and the reliability constraint objective: in, This indicates the overall optimization goal. This represents the classification cross-entropy of this batch. This represents the reliability alignment loss for this batch. By jointly optimizing the classification objective and the reliability constraint objective, the backbone network for each modality is optimized. Single-modal classifier and modal reliability evaluator parameters Perform an end-to-end update.
[0051] S5. During the testing phase, extract the features of each modality and obtain the corresponding reliability score. Perform reliability weighted fusion on the classification output of each modality and output the final aerial target recognition result. Step S5 includes the following sub-steps: S51, Given a test sample , for the First, extract the representation of each modality and obtain the predicted distribution: in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Feature representation of each modality Indicates the first A backbone network of multiple modalities, Indicates the first The first sample Input data for each modality, Indicates the first Encoder parameters for each mode, Indicates the first The first sample The classification logits of each modality Indicates the first A classification head for each modality, Indicates the first The classification head parameters for each modality are then used. The predicted distribution is then... Further calculation of the prediction differences between modes Intramodal uncertainty And construct modal evidence: in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Modal evidence vectors for each modality Indicates the first The first sample Features of each modality Indicates the first The first sample Intermodal prediction differences among modalities No. The first sample The intra-modal uncertainty of each mode. A reliability score is obtained using a modal reliability estimator. in, Indicates the sample index. Indicates modal index, Indicates the first The first sample The reliability of each mode This represents the sigmoid operator function. Indicates the first A reliability estimator for each mode. Indicates the first The first sample Modal evidence vectors for each modality Indicates the first Reliability estimator parameters for each mode.
[0052] S52. Based on the reliability scores of each modality, perform reliability-weighted fusion of the classification outputs of each modality to obtain the fused logits: in, Indicates the sample index. Indicates modal index, This represents the logits after weighted fusion of various modalities. Indicates the number of modes. Indicates the first The first sample The reliability of each mode Indicates the first The first sample Logits of each modality.
[0053] S53. Based on the fused logits, output the final predicted category: in, Indicates the final predicted category. Indicates the current category label. Indicates the total number of categories. Indicates the first The sample at the th Logits across each category. Finally, utilizing... This serves as the final result of aerial target identification.
Claims
1. An aerial target recognition method based on bootstrapping multimodal learning, characterized in that, Includes the following steps: S1. Construct a training dataset containing multimodal aerial target samples and their category labels, and set up a model structure for feature extraction, target classification and modal reliability evaluation for each modality; S2. During the training process, some samples are subjected to bootstrapping enhancement. Based on the noise correlation constraints, modal perturbation and combination schemes are generated for the samples to form an enhanced training set for revealing supervision. S3. Extract the representation of each modality and obtain the classification output. At the same time, characterize the prediction differences between modalities and the uncertainty within modalities, and generate a reliability score that represents the credibility of the modality. S4. Jointly optimize the classification learning objective and the reliability constraint objective to suppress the interference of noise association on the learning process and update the model parameters; S5. During the testing phase, extract the features of each modality and obtain the corresponding reliability score. Perform reliability weighted fusion on the classification outputs of each modality and output the final aerial target recognition result.
2. The aerial target recognition method based on bootstrapping multimodal learning according to claim 1, characterized in that, The process of generating the modality mask in step S2 is as follows: In the At the start of each round of training, a clean training set is provided: in, Indicates the sample index. Indicates modal index, Represents the training dataset. This represents the first data containing three modalities. One sample, Indicates the first Category labeling for each sample, Indicates the size of the dataset. Indicates the number of modes. Indicates the first The first sample The input consists of several modalities. First, a subset of indices is sampled from the training dataset: in, Indicates a subset of the sampling index. Indicates the size of the dataset. Indicates the proportion of self-increased growth. This indicates the dataset size operator. This represents the floor operation. Then, for each sample in the sampled index subset... Generate a modal mask: in, Indicates the sample index. Indicates modal index, Indicates sample modal mask, Indicates the first The first sample A mask for each modality. This represents the number of modes. Among them, Noise association constraints should be met: in, Indicates the sample index. Indicates modal index, Indicates the number of modes. Indicates the first The first sample A mask for each modality. This represents the floor operation.
3. The aerial target recognition method based on bootstrapping multimodal learning according to claim 1, characterized in that, The process of generating the bootstrap augmented training set in step S2 is as follows: Based on modal mask The samples are shuffled and replaced with identical modalities within a subset to obtain bootstrapping augmented samples. The rules are as follows: in, and Both represent sample indices. Indicates modal index, Indicates the first A self-bootstrapping augmentation sample, Indicates the first The first sample The form after modal bootstrapping augmentation Represents the first in the original dataset One sample, Represents the first in the original dataset One sample, Indicates the first The first sample A mask for each modality. This represents a subset of the sampled indices. When hour, from Internal random sampling is used to achieve random shuffling and replacement of this mode. Sample preservation and order The final result is a bootstrap augmented training set that reveals supervisory information: in, Indicates the sample index. This represents the bootstrap augmented training set obtained, which reveals supervisory information. Indicates the first A self-bootstrapping augmentation sample, Indicates the first Category labeling for each sample, Indicates sample modal mask, Indicates the size of the dataset.
4. The aerial target recognition method based on bootstrapping multimodal learning according to claim 1, characterized in that, The process of extracting modal representations and predicting distributions in step S3 is as follows: Given a bootstrap augmented training set , for the The first sample Each modality is represented by a single-modal backbone network, from which modal characteristics are extracted and distributions are predicted. in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Features of each modality Indicates the first A modal encoder, Indicates the first The form after bootstrapping of individual samples Indicates the first Encoder parameters for each mode, Indicates the first The first sample The classification logits of each modality, Indicates the first A classification head for each modality, Indicates the first Classification header parameters for each modality, Indicates the first The first sample The predicted distribution of each modality, This represents the softmax operator function.
5. The aerial target recognition method based on bootstrapping multimodal learning according to claim 1, characterized in that, The process of calculating the inter-modal prediction difference and intra-modal uncertainty in step S3 is as follows: Based on the predicted distribution of each modality Calculate the prediction difference between modes And calculate the intramodal uncertainty based on the prediction entropy of each mode. : in, Indicates the sample index. and Both represent modal indices. Indicates the first The first sample Jeffreys divergence for each mode, Indicates the number of modes. This represents the Kullback-Leibler divergence operator function. Indicates the first The first sample The predicted distribution of each modality, Indicates the first The first sample The predicted distribution of each modality, Indicates the first The first sample Intramodal uncertainty of a mode, Indicates the first The first sample The prediction entropy of each modality Indicates the number of categories in the dataset. Indicates the first The first sample The first mode The predicted probability of each category.
6. The aerial target recognition method based on bootstrapping multimodal learning according to claim 1, characterized in that, The process of generating the modal reliability score in step S3 is as follows: Modal representations, inter-modal prediction discrepancies, and intra-modal uncertainties are concatenated to form a modal evidence vector: in, Indicates the sample index. Indicates modal index, Indicates the first The first sample Modal evidence vectors for each modality Indicates the first The first sample Jeffreys divergence for each mode, Indicates the first The first sample The intra-modal uncertainty of each mode is then determined. The modal evidence vector is then input into the modal reliability estimator to obtain the modal reliability score. in, Indicates the sample index. Indicates modal index, Indicates the first The first sample The reliability of each mode This represents the sigmoid operator function. Indicates the first A reliability estimator for each mode. No. The first sample A modal evidence vector representing each modality. Indicates the first Reliability estimator parameters for each mode.
7. The aerial target recognition method based on bootstrapping multimodal learning according to claim 1, characterized in that, The process of obtaining the final predicted distribution in step S3 is as follows: The final prediction distribution is obtained by weighting and fusing the prediction distributions of each modality based on their reliability scores. in, Indicates the sample index. Indicates modal index, No. The predicted distribution of each sample Indicates the number of modes. Indicates the first The first sample The reliability of each mode Indicates the first The first sample The classification logits of each modality, This represents the softmax operator function.
8. The aerial target recognition method based on bootstrapping multimodal learning according to claim 1, characterized in that, The process of calculating the classification learning objective in step S4 is as follows: For batches sampled from the bootstrap augmented training set, the classification learning objective is defined as: in, This represents the classification cross-entropy of this batch. This represents the set of batch indices sampled from the bootstrap augmented training set. This indicates the dataset size operator. Indicates the sample index. Indicates the first The predicted category for each sample, Indicates the first The predicted category for each sample is The probability of.
9. The aerial target recognition method based on bootstrapping multimodal learning according to claim 1, characterized in that, The process of calculating the reliability constraint target in step S4 is as follows: For batches sampled from the bootstrap augmented training set, the reliability constraint objective is defined as: in, This indicates the reliability alignment loss for this batch. This represents the set of batch indices sampled from the bootstrap augmented training set. This indicates the dataset size operator. Indicates the sample index. Indicates modal index, Indicates the number of modes. Indicates the first The first sample A mask for each modality. Indicates the first The first sample Reliability of each mode.
10. The aerial target recognition method based on bootstrapping multimodal learning according to claim 1, characterized in that, The process of joint optimization and updating model parameters in step S4 is as follows: The total loss function is obtained by weighting and combining the classification learning objective and the reliability constraint objective: in, This indicates the overall optimization goal. This represents the classification cross-entropy of this batch. This represents the reliability alignment loss for this batch. By jointly optimizing the classification objective and the reliability constraint objective, the backbone network for each modality is optimized. Single-modal classifier and modal reliability evaluator parameters Perform an end-to-end update.