A multi-modal meta-learning remote sensing reconnaissance target recognition method guided by a detection mechanism

Abstract

The application discloses a kind of detection mechanism guided multi-modal meta-learning remote sensing reconnaissance target identification method, belong to computer vision and remote sensing image processing field.The core of method is: detection mechanism guided multi-modal feature extraction and fusion: for the electromagnetic scattering characteristics of SAR image, geometric texture characteristics of visible light image and thermal radiation characteristics of infrared image, special feature extraction network is designed respectively, and cross-modal fusion module based on attention mechanism is introduced, to realize the feature enhancement expression of physical information complementation;Meta-learning training paradigm: construct a meta-learner based on optimization, train on a large number of multi-modal remote sensing task sets, so that the model obtains the ability to quickly learn new target categories from a small number of samples.The method combines physical detection mechanism with data-driven meta-learning, effectively solves the problem of low recognition accuracy and poor generalization ability of traditional methods in complex reconnaissance scenarios such as sample scarcity, target type variability and SAR image interpretation difficulty.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and remote sensing image processing, and particularly relates to a multi-modal meta-learning remote sensing reconnaissance target recognition method guided by a detection mechanism. BACKGROUND

[0002] Remote sensing reconnaissance target recognition is a core link in modern military perception, disaster monitoring and emergency response. With the development of remote sensing technology, the fusion recognition of heterogeneous data (i.e. multi-modal remote sensing data) obtained by using synthetic aperture radar, visible light, infrared and other sensors has become a key way to improve target detection and recognition performance. SAR (synthetic aperture radar) has all-weather and all-day working ability, and its image reflects the geometric structure and dielectric properties of electromagnetic scattering mechanism of ground objects. Visible light images have high resolution and contain rich geometric texture and detail information. Infrared images can effectively capture the thermal radiation characteristics of targets to achieve night detection and camouflage recognition. These three modalities have natural complementarity in information.

[0003] However, existing multi-modal remote sensing target recognition methods still face the following challenges when dealing with complex reconnaissance scenarios: (1) recognition difficulties under sample scarcity conditions; (2) lack of multi-modal feature fusion mechanism; (3) insufficient model generalization and rapid adaptation ability; (4) difficulty in interpreting SAR images.

[0004] To solve the above problems of sample scarcity, lack of multi-modal fusion mechanism, insufficient model generalization ability and difficulty in interpreting SAR images, the application provides a multi-modal meta-learning remote sensing reconnaissance target recognition method guided by a detection mechanism. SUMMARY

[0005] The purpose of the application is to provide a multi-modal meta-learning remote sensing reconnaissance target recognition method guided by a detection mechanism, which aims to combine physical detection mechanism and meta-learning paradigm to solve the problem of rapid and accurate identification of unknown or rare targets under sample scarcity conditions.

[0006] To achieve the above purpose, the application provides a multi-modal meta-learning remote sensing reconnaissance target recognition method guided by a detection mechanism, comprising the following steps:

[0007] S1, obtaining SAR images, visible light images and infrared images of the same reconnaissance area, and performing registration and standardization processing to form a multi-modal image pair;

[0008] S2, inputting the registered multi-modal image pair into three parallel feature extraction networks for feature extraction, and fusing the extracted multi-modal features;

[0009] S3, construct a meta-training set containing a large number of multi-classification tasks, and train the feature extraction network and the classifier using a model-independent meta-learning framework;

[0010] S4, when facing new target categories, quickly fine-tune the model using a small number of labeled samples to quickly adapt to new tasks;

[0011] S5, input the target to be recognized into the adapted model, output the recognition result and confidence, and complete the target recognition in the small sample scene.

[0012] Further, the SAR image, visible light image and infrared image of the same reconnaissance area in S1 are acquired and registered and standardized, to form a multi-modal image pair, which specifically includes the following contents:

[0013] S1.1, multi-modal remote sensing data acquisition: through coordinated deployment or by using a platform with multi-sensor load, the synthetic aperture radar (SAR) image, visible light image and infrared image of the same geographical reconnaissance area are synchronously or quasi-synchronously acquired to ensure that the data of each modality have high consistency in time and space, thereby constructing the original multi-modal remote sensing data triplets , The constituent elements of the above are as follows:

[0014] ;

[0015] Among them represents the synthetic aperture radar (SAR) image, represents the visible light image, represents the infrared image;

[0016] S1.2, spatial registration: the acquired multi-modal images are spatially registered to accurately align the pixels in geometric position, so as to eliminate the spatial inconsistency caused by imaging angle, platform attitude and resolution difference, which specifically includes:

[0017] S1.2.1, control point extraction and matching: extracting significant feature points (such as SIFT or ORB feature points) on the visible light image as a reference control point set , then extracting corresponding feature point sets and on the SAR image and the infrared image respectively

[0018] S1.2.2, transformation model solving: based on the matched feature point pairs and , the spatial transformation models and The model is usually a projection transformation or an affine transformation, and its mathematical form is:

[0019] ;

[0020] wherein, is the source image coordinate, is the coordinate transformed to the reference image coordinate system, is the transformation matrix;

[0021] S1.2.3, image resampling: using the solved transformation matrix , resample and to align with the visible light image in space, to obtain the registered images and ;

[0022] S1.3, image standardization: the registered images of each modality are respectively standardized to eliminate the influence of sensor difference and imaging environment, and the pixel value is normalized to a unified numerical range, which is convenient for subsequent neural network processing, specifically including:

[0023] S1.3.1, SAR image standardization: according to the characteristics of SAR image, such as coherent speckle noise and large dynamic range, first, radiation calibration is performed, then logarithmic transformation is used to convert multiplicative noise into additive noise, and contrast enhancement is performed, and finally Z-Score standardization is performed:

[0024] ;

[0025] wherein, is a small constant to prevent the logarithm from being zero, and are the mean and standard deviation of the image block respectively;

[0026] S1.3.2, visible light image standardization: after color balance and white balance correction of , pixel value normalization is performed:

[0027] ;

[0028] wherein and are the mean and standard deviation of the visible light image block respectively:

[0029] or normalized to the interval [0, 1]:

[0030] ;

[0031] S1.3.3, infrared image standardization: Blackbody-based non-uniformity correction and atmospheric correction are performed to eliminate the influence of environmental radiation, and then minimum-maximum value normalization is performed to map the temperature value or the gray value to a specific range:

[0032] ;

[0033] wherein, and are preset lower and upper limits of the normalization range, or are the minimum and maximum values of pixel statistics of the image .

[0034] Further, the S2 inputs the registered multi-modal image pair into three parallel feature extraction networks for feature extraction, respectively, and fuses the extracted multi-modal features, and specifically includes the following contents:

[0035] S2.1, mechanism-guided feature extraction: the standardized multi-modal image pair output by S1 is input into three parallel structure-specific feature extraction branches, specifically:

[0036] S2.1.1, SAR feature extraction branch: the backbone network of the branch adopts a deep convolutional neural network (such as a ResNet variant) with a large receptive field and residual connection to adapt to the electromagnetic scattering characteristics of the target and the complex speckle noise in the SAR image, and the network parameters of the branch are set as , and the process of extracting features is:

[0037] ;

[0038] wherein, is the extracted SAR modality high-level feature map, which focuses on representing the physical information such as the strong scattering center distribution and material properties of the target;

[0039] S2.1.2, visible light feature extraction branch: the backbone network of the branch adopts a deep convolutional neural network (such as VGG or CNN with attention mechanism) focusing on texture and edge perception to capture the geometric structure and surface texture characteristics of the target in the visible light image, and the network parameters of the branch are set as , and the process of extracting features is:

[0040] ;

[0041] wherein, is the extracted visible light modality high-level feature map;

[0042] ​S2.1.3, infrared feature extraction branch: the backbone network of this branch adopts a deep convolutional neural network capable of enhancing the contrast of thermal contours, focusing on the thermal radiation distribution and contour characteristics of the target in the infrared image. The network parameters of this branch are set as The process of extracting features is as follows:

[0043] ;

[0044] wherein, is the extracted infrared modal high-level feature map;

[0045] S2.2, feature fusion based on attention mechanism: the feature maps extracted by the above three branches , , are spliced in the channel dimension to obtain the preliminary fused features ;

[0046] ;

[0047] wherein, [;] represents the splicing operation along the channel dimension;

[0048] Subsequently, the is input into a cross-modal channel attention module to adaptively learn and weight the importance of different modal feature channels. The specific steps of this module are as follows:

[0049] S2.2.1, global context information acquisition: global average pooling is performed on each channel of to generate a channel statistical descriptor The calculation formula of the cth element of

[0050] ;

[0051] S2.2.2, attention weight calculation: the descriptor z is input into a gating mechanism composed of two fully connected layers to capture the nonlinear dependence between channels and output the attention weight of each channel :

[0052] ;

[0053] wherein, , is the weight matrix of the fully connected layer, is the scaling ratio, is the ReLU activation function, is the Sigmoid activation function, which is used to map the weight to 0~1;

[0054] S2.2.3, Feature re-weighting: multiply the learned attention weight s with the original concatenated feature Channel-wise multiplication to get the enhanced fused feature :

[0055] ;

[0056] Finally, output , which will be used as the input of the subsequent meta-learning classifier.

[0057] Further, the S3 constructs a meta-training set containing a large number of multi-classification tasks, and trains the feature extraction network and the classifier using a model-independent meta-learning framework, which includes the following contents:

[0058] S3.1, Meta-training task set construction: from a basic dataset containing a large number of categories, construct a meta-training set according to the model-independent meta-learning paradigm . The set is composed of a large number of meta-tasks , each task simulates an N-way K-shot (N-class K-sample) classification problem, specifically:

[0059] S3.1.1, randomly sample N categories from the basic category set;

[0060] S3.1.2, from the N categories, randomly sample K labeled sample images from each category to form the support set of the task, where i is the index of the meta-task, j is the serial number of the sample in the set, is the jth multi-modal image sample, is the corresponding category label;

[0061] S3.1.3, from the same N categories, sample a number of samples from each category to form the query set of the task, where Q is the number of samples, i is the index of the meta-task, j is the serial number of the sample in the set, is the jth multi-modal image sample, is the corresponding category label;

[0062] S3.2, model initialization: initialize the parameters of the entire model, including the parameters of the three feature extraction branch networks defined in S2 , , , the parameters of the feature fusion module , and the parameters of a basic classifier g(·) , all trainable parameters are collectively referred to as ;

[0063] S3.3, Meta-learning training loop: Adopting the model-agnostic meta-learning framework, the model parameters Θ are updated through inner loop update and outer loop update Optimization is performed as follows:

[0064] S3.3.1, Inner loop update: For each meta-task in the current batch , the task support set is input into the model, and first the fusion features are obtained through the feature extraction and fusion network , and then the prediction probability and loss are calculated through the base classifier, and the loss function usually adopts cross-entropy loss:

[0065] ;

[0066] On the task , one or more steps of gradient descent are performed on the model parameters Θ to obtain task-specific adapted parameters :

[0067] ;

[0068] Wherein, a is the inner loop learning rate, which is a hyperparameter;

[0069] S3.3.2, Outer loop update: Using the adapted parameters obtained in the last step , the loss is calculated on the query set of the corresponding task , and this loss is used to evaluate the generalization performance of the adapted model on new samples;

[0070] After all the tasks in the current batch are completed, the loss of all tasks on the query set is aggregated to calculate the meta-loss:

[0071] ;

[0072] The most critical step is to calculate the gradient of the meta-loss with respect to the original parameters Θ through backpropagation, and use this gradient to update the initial parameters Θ of the model:

[0073] ;

[0074] Wherein, β is the outer loop learning rate (meta-learning rate), and this update step optimizes the generalization potential of the initial parameters Θ of the model, so that the model can perform well on new, unseen tasks after experiencing a small number of gradient steps;

[0075] Through repeated iteration of S3.3, the model parameters The trained set of excellent initialization points can quickly adapt to new tasks and achieve high-precision target recognition when encountering small sample tasks containing new categories, only a few inner loop updates based on the set of initial parameters are needed.

[0076] Further, when facing new target categories in S4, a small amount of labeled samples are used to quickly fine-tune the model to quickly adapt to new tasks, which includes the following contents:

[0077] S4.1, new task support set construction: in the deployment stage, when M new target categories that have not appeared in meta-training need to be recognized, and only annotated samples per category, these few labeled samples are constructed as the support set of the new task:

[0078] ;

[0079] Among them, is the label of the new category;

[0080] S4.2, model rapid self-adaptation: load the model initial parameters with good generalization potential obtained by S3 training , which is used as a starting point, only using the support set of the new task, G-step gradient descent is performed to quickly fine-tune the model, which includes:

[0081] S4.2.1, forward propagation and loss calculation: input the model , calculate the loss of the model on the current support set through feature extraction, fusion and classifier:

[0082] ;

[0083] Among them, g is the current gradient step number, and the initial step g=0 model parameter is Θ;

[0084] S4.2.2, parameter update: calculate the gradient according to the loss and update the model parameters, this update process can be for all parameters, or to improve efficiency and prevent overfitting, only update the parameters of the basic classifier and the parameters of the feature fusion module :

[0085] ;

[0086] Among them, is the learning rate in the adaptation stage, which can be the same as or different from the inner loop learning rate in meta-training. The parameters of the feature extraction branch are usually frozen in this process; ​

[0087] After the G-step update, the model parameters optimized for the new M classes are obtained , and the model has quickly migrated its recognition ability to the new target classes.

[0088] Further, in S5, the target to be identified is input into the adapted model, and the recognition result and confidence are output, completing target recognition in a small sample scenario, which includes the following contents:

[0089] S5.1, forward inference of the target to be identified: input the target image to be identified (i.e., query sample from the new class) to the model adapted quickly in S4;

[0090] S5.2, feature extraction and classification: the model first performs the same preprocessing as in training on , and then obtains its enhanced fusion feature representation through a multi-stream feature extraction network and a cross-modal fusion module, and then inputs the feature into the classifier adapted for the new task to calculate the logits (scores) of the target belonging to each new class:

[0091] ;

[0092] wherein is an M-dimensional vector;

[0093] S5.3, recognition result and confidence output: the target recognition module performs Softmax normalization on the logits vector z to obtain the probability distribution of the target belonging to each new class:

[0094] ;

[0095] The final system output is:

[0096] Recognition result: take the class with the maximum probability as the final recognition label:

[0097] ;

[0098] Confidence: take the maximum probability value as the confidence score of this recognition:

[0099] .

[0100] Beneficial Effects: This invention effectively mines and fuses the electromagnetic scattering characteristics of SAR images, the geometric texture characteristics of visible light images, and the thermal radiation characteristics of infrared images by designing a mechanism-guided multi-stream feature extraction network and a cross-modal attention fusion module. Simultaneously, by combining a model-independent meta-learning training paradigm, the model gains the ability to rapidly learn new targets from small samples across numerous multimodal remote sensing tasks. Ultimately, it achieves the fusion of physical detection mechanisms and data-driven meta-learning, significantly improving recognition accuracy and generalization ability in complex reconnaissance scenarios with scarce samples and diverse target types. Attached Figure Description

[0101] Figure 1 This is a flowchart illustrating the present invention;

[0102] Figure 2 This is a schematic diagram illustrating the process of registering and processing three types of images to construct multimodal image pairs in an embodiment of the present invention;

[0103] Figure 3 This is a schematic diagram illustrating the process of feature extraction and fusion of multimodal image pairs in an embodiment of the present invention;

[0104] Figure 4 This is a schematic diagram illustrating the process of training a task set using a meta-learning framework in an embodiment of the present invention;

[0105] Figure 5 This is a schematic diagram illustrating the process of the model fine-tuning and identifying targets when faced with new target categories in an embodiment of the present invention. Detailed Implementation

[0106] The invention will now be further described with reference to the accompanying drawings. Example

[0107] like Figure 1 As shown, a multimodal meta-learning remote sensing reconnaissance target recognition method guided by detection mechanism includes the following steps:

[0108] S1. Acquire SAR, visible light, and infrared images of the same reconnaissance area, and perform registration and standardization processing to form a multimodal image pair;

[0109] S2. Input the registered multimodal image into three parallel feature extraction networks to extract features, and then fuse the extracted multimodal features.

[0110] S3. Construct a meta-training set containing a large number of multi-classification tasks, and use a model-independent meta-learning framework to train the feature extraction network and classifier.

[0111] S4. When faced with a new target category, use a small number of labeled samples to quickly fine-tune the model so that it can quickly adapt to the new task.

[0112] S5, input the target to be identified into the adapted model, output the recognition result and confidence, and complete the target recognition in a small sample scenario.

[0113] Further, as shown in Figure 2 S1, the SAR image, visible light image and infrared image of the same reconnaissance area are acquired, registered and standardized, to form a multi-modal image pair, and the specific steps are as follows:

[0114] S1.1, multi-modal remote sensing data acquisition: through coordinated deployment or by using a platform with multi-sensor load, the synthetic aperture radar (SAR) image, visible light image and infrared image of the same geographical reconnaissance area are synchronously or quasi-synchronously acquired, to ensure that the data of each mode has high consistency in time and space, thereby constructing the original multi-modal remote sensing data triple , The constituent elements of the triple are as follows:

[0115] ;

[0116] Among them represents the synthetic aperture radar (SAR) image, represents the visible light image, represents the infrared image;

[0117] S1.2, spatial registration: the acquired multi-modal images are spatially registered to accurately align the pixels in geometric position, so as to eliminate the spatial inconsistency caused by imaging angle, platform attitude and resolution difference, specifically including:

[0118] S1.2.1, control point extraction and matching: extracting significant feature points (such as SIFT or ORB feature points) on the visible light image as a reference control point set , then extracting corresponding feature point sets and on the SAR image and the infrared image respectively;

[0119] S1.2.2, transformation model solving: based on the matched feature point pairs and , the spatial transformation models and are estimated by using the random sample consensus algorithm, which is usually a projection transformation or an affine transformation, and its mathematical form is:

[0120] ;

[0121] Among them, For source image coordinates, For coordinates transformed to the reference image coordinate system, is a transformation matrix;

[0122] S1.2.3, image resampling: using the solved transformation matrix , and are resampled to be spatially aligned with the visible light image , obtaining the registered images and ;

[0123] S1.3, image standardization: the registered images of each modality are respectively standardized to eliminate the influence of sensor differences and imaging environment, and the pixel values are normalized to a unified numerical range, which is convenient for subsequent neural network processing, specifically including:

[0124] S1.3.1, SAR image standardization: in view of the characteristics of SAR image such as coherent speckle noise and large dynamic range, first, radiation calibration is performed, then logarithmic transformation is used to convert multiplicative noise into additive noise, and contrast enhancement is performed, and finally Z-Score standardization is performed:

[0125] ;

[0126] wherein, is a small constant to prevent the logarithm from being zero, and are the mean and standard deviation of the image block respectively;

[0127] S1.3.2, visible light image standardization: after color balance and white balance correction are performed on , pixel value normalization is performed:

[0128] ;

[0129] wherein and are the mean and standard deviation of the visible light image block respectively:

[0130] or normalized to the interval [0, 1]:

[0131] ;

[0132] S1.3.3, infrared image standardization: after non-uniformity correction and atmospheric correction based on blackbody are performed on , the influence of environmental radiation is eliminated, and then minimum-maximum value standardization is performed to map the temperature value or gray value to a specific range:

[0133] ;​

[0134] wherein, and are preset normalization range lower and upper limits, or are pixel statistical minimum and maximum values of the image .

[0135] Further, as shown in Figure 3 S2, the registered multi-modal image pairs are respectively input into three parallel feature extraction networks for feature extraction, and the extracted multi-modal features are fused, and the specific steps are:

[0136] S2.1, mechanism-guided feature extraction: the standardized multi-modal image pairs output by S1 are respectively input into three parallel structure-specific feature extraction branches, specifically:

[0137] S2.1.1, SAR feature extraction branch: the backbone network of this branch adopts a deep convolutional neural network with a large receptive field and residual connection (such as a ResNet variant) to adapt to the electromagnetic scattering characteristics of the target and the complex speckle noise in the SAR image. Let the network parameters of this branch be , and the process of extracting features is:

[0138] ;

[0139] wherein, is the extracted SAR modality high-level feature map, which focuses on representing the physical information of the target such as the strong scattering center distribution and material properties;

[0140] S2.1.2, visible light feature extraction branch: the backbone network of this branch adopts a deep convolutional neural network (such as VGG or CNN with attention mechanism) that focuses on texture and edge perception to capture the geometric structure and surface texture characteristics of the target in the visible light image. Let the network parameters of this branch be , and the process of extracting features is:

[0141] ;

[0142] wherein, is the extracted visible light modality high-level feature map;

[0143] S2.1.3, infrared feature extraction branch: the backbone network of this branch adopts a deep convolutional neural network that can enhance the contrast of thermal contours to focus on the thermal radiation distribution and contour characteristics of the target in the infrared image. Let the network parameters of this branch be , and the process of extracting features is:

[0144] ; ​

[0145] wherein, is the extracted infrared modal high-level feature map;

[0146] S2.2, feature fusion based on attention mechanism: the feature maps extracted by the above three branches , , are spliced in the channel dimension to obtain the preliminary fused feature ;

[0147] ;

[0148] wherein, [;] represents the splicing operation along the channel dimension;

[0149] Subsequently, is input into a cross-modal channel attention module to adaptively learn and weight the importance of different modal feature channels, and the specific steps of the module are as follows:

[0150] S2.2.1, global context information acquisition: global average pooling is performed on each channel of to generate a channel statistical descriptor , and the calculation formula of the cth element of

[0151] ;

[0152] S2.2.2, attention weight calculation: the descriptor z is input into a gating mechanism composed of two fully connected layers to capture the nonlinear dependence between channels, and the attention weight of each channel is output:

[0153] ;

[0154] wherein, , is the weight matrix of the fully connected layer, is the scaling ratio, is the ReLU activation function, is the Sigmoid activation function, which is used to map the weight to 0~1;

[0155] S2.2.3, feature reweighting: the learned attention weight s is multiplied with the original spliced feature channel by channel to obtain the enhanced fused feature :

[0156] ;

[0157] Finally, the output , which will be used as the input of the subsequent meta-learning classifier.

[0158] Further, as shown in Figure 4 , the S3 is constructed to include a large number of meta-training sets of multi-classification tasks, and a model-independent meta-learning framework is used to train the feature extraction network and the classifier, and the specific steps are as follows:

[0159] S3.1, meta-training task set construction: from the basic data set containing a large number of categories, a meta-training set is constructed according to the model-independent meta-learning paradigm , which is composed of a large number of meta-tasks , each task simulates an N-way K-shot (N-class K-sample) classification problem, specifically:

[0160] S3.1.1, randomly sample N categories from the basic category set;

[0161] S3.1.2, randomly sample K labeled sample images from each of the N categories to form the support set of the task , where i is the index of the meta-task, j is the serial number of the sample in the set, is the jth multi-modal image sample, is the corresponding category label;

[0162] S3.1.3, sample a number of samples from the same N categories, and form the query set of the task , where Q is the number of samples, i is the index of the meta-task, j is the serial number of the sample in the set, is the jth multi-modal image sample, is the corresponding category label;

[0163] S2.2, model initialization: initialize the parameters of the entire model, including the parameters of the three feature extraction branch networks defined in S2 , , , the parameters of the feature fusion module , and the parameters of a basic classifier g(·) . All trainable parameters are collectively referred to as ;

[0164] S3.3, meta-learning training loop: using a model-independent meta-learning framework, the model parameters are optimized through inner loop update and outer loop update, specifically as follows:

[0165] S3.3.1, inner loop update: for each meta-task in the current batch, The input model first obtains fused features through a feature extraction and fusion network. Then, the predicted probability is calculated using the base classifier. And loss, the loss function is usually cross-entropy loss:

[0166] ;

[0167] In the mission Above, perform one or more steps of gradient descent on the model parameters Θ to obtain the task-specific adapted parameters. :

[0168] ;

[0169] Where α is the inner loop learning rate, which is a hyperparameter;

[0170] S3.3.2, Outer Loop Update: Use the adapted parameters obtained in the previous step. In the query set of the corresponding task Calculate loss This loss is used to evaluate the generalization performance of the adapted model on new samples;

[0171] After all tasks in the current batch are completed, aggregate the losses of all tasks on the query set and calculate the meta-loss:

[0172] ;

[0173] The most crucial step is to calculate the meta-loss through backpropagation. The gradient relative to the original parameters Θ, and using this gradient to update the initial parameters Θ of the model:

[0174] ;

[0175] Where β is the outer loop learning rate (meta-learning rate), this update step optimizes the "generalization potential" of the model's initial parameter Θ, so that the model can perform well on new and unseen tasks after experiencing a small number of gradient steps;

[0176] Through repeated iterations of S3.3, the model parameters The initial parameters are trained to form a set of excellent initialization points. When encountering a small sample task containing new categories, the system can quickly adapt to the new task by performing a few inner loop updates based on these initial parameters, thereby achieving high-precision target recognition.

[0177] Furthermore, such as Figure 5 As shown, in step S4, when faced with a new target category, the model is quickly fine-tuned using a small number of labeled samples to enable it to adapt quickly to the new task. The specific steps are as follows:

[0178] S4.1, new task support set construction: in the deployment stage, when M new target categories that have never appeared in meta-training need to be recognized, and there are only annotated samples for each category, these few annotated samples are constructed as the support set of the new task:

[0179] ;

[0180] wherein, is the label of the new category;

[0181] S4.2, model rapid adaptation: load the model initial parameters with good generalization potential obtained through S3 training , which is used as a starting point, only using the support set of the new task , G-step gradient descent is performed to rapidly fine-tune the model:

[0182] S4.2.1, forward propagation and loss calculation: input the model , calculate the loss of the model on the current support set through feature extraction, fusion and classifier: :

[0183] ;

[0184] wherein g is the current gradient step number, and the model parameters are Θ when the initial step g = 0;

[0185] S4.2.2, parameter update: calculate the gradient according to the loss and update the model parameters. This update process can be for all parameters, or to improve efficiency and prevent overfitting, only update the parameters of the basic classifier and the parameters of the feature fusion module :

[0186] ;

[0187] wherein, is the learning rate of the adaptation stage, which can be the same as or different from the inner loop learning rate in meta-training . The parameters of the feature extraction branch are usually frozen in this process;

[0188] After G-step update, the model parameters optimized for the new M categories are obtained, and the model has rapidly migrated its recognition ability to the new target categories.

[0189] Further, as shown in Figure 5 , the model after adaptation is input with the target to be recognized in S5, and the recognition result and confidence are output, completing target recognition in a small sample scenario, and the specific steps are:

[0190] S5.1, Forward inference of the target to be identified: input the target image to be identified (i.e. query sample ) from the new category to the model after the fast adaptation of S4;

[0191] S5.2, Feature extraction and classification: the model first performs the same preprocessing as in training on , and then obtains its enhanced fusion feature representation through the multi-stream feature extraction network and the cross-modal fusion module, and then inputs the feature into the classifier adapted to the new task to calculate the logits (scores) of the target belonging to each new category:

[0192] ;

[0193] wherein, is an M-dimensional vector;

[0194] S5.3, Recognition result and confidence output: the target recognition module performs Softmax normalization on the logits vector z to obtain the probability distribution of the target belonging to each new category:

[0195] ;

[0196] The final system output is:

[0197] Recognition result: take the class with the maximum probability as the final recognition label:

[0198]

[0199] Confidence: take the maximum probability value as the confidence score of this recognition:

[0200] .

Claims

1. A multimodal meta-learning remote sensing reconnaissance target recognition method guided by detection mechanism, characterized in that, Includes the following steps: S1. Acquire SAR, visible light, and infrared images of the same reconnaissance area, and perform registration and standardization processing to form a multimodal image pair; S2. Input the registered multimodal image into three parallel feature extraction networks to extract features, and then fuse the extracted multimodal features. S3. Construct a meta-training set containing a large number of multi-classification tasks, and use a model-independent meta-learning framework to train the feature extraction network and classifier. S4. When faced with a new target category, use a small number of labeled samples to quickly fine-tune the model so that it can quickly adapt to the new task. S5. Input the target to be identified into the adapted model, and output the recognition result and confidence score to complete the target recognition in a small sample scenario. This includes the following steps: S5.1 Forward reasoning of the target to be identified: The image of the target to be identified from the new category, i.e., the query sample, is then used to identify the target image to be identified. Input into the model after S4 fast adaptive processing; S5.2 Feature Extraction and Classification: The model first performs feature extraction and classification... Perform the same preprocessing as during training, and then obtain its enhanced fused feature representation through a multi-stream feature extraction network and a cross-modal fusion module. Then, this feature is input into a classifier that has been adapted to the new task. Calculate the logits of each new category: ； in, It is an M-dimensional vector; S5.3 Recognition Results and Confidence Output: The target recognition module performs Softmax normalization on the logits vector z to obtain the probability distribution of the target belonging to each new category: ； The final system output is: Recognition result: The category with the highest probability is selected as the final recognition label. ； Confidence score: The highest probability value is used as the confidence score for this identification. 。 2. The multimodal meta-learning remote sensing reconnaissance target identification method guided by detection mechanism according to claim 1, characterized in that, S1 includes the following steps: S1.1 Multimodal Remote Sensing Data Acquisition: By coordinating deployment or utilizing platforms equipped with multi-sensor payloads, synthetic aperture radar (SAR), visible light, and infrared (IR) images of the same geographic reconnaissance area are acquired synchronously or quasi-synchronously, ensuring high temporal and spatial consistency among the modal data, thereby constructing the original multimodal remote sensing data triplet. , The constituent elements are as follows: ； in This represents a synthetic aperture radar (SAR) image. Represents a visible light image. Represents an infrared image; S1.2 Spatial Registration: Spatial registration is performed on the acquired multimodal images to ensure that the pixels are precisely aligned in geometric positions, thereby eliminating spatial inconsistencies caused by differences in imaging viewpoint, platform pose, and resolution; specifically including: S1.2.1 Control Point Extraction and Matching: In visible light images Extract significant feature points as a reference control point set. Subsequently, in SAR images and infrared images Extract the corresponding feature point set. and ; S1.2.2, Solving the transformation model: Based on matching feature point pairs and The spatial transformation model is estimated using the random sampling consensus algorithm. and This model is a projection transformation or affine transformation, and its mathematical form is: ； in, The coordinates of the source image. The coordinates are after transformation to the reference image coordinate system. The transformation matrix; S1.2.3, Image resampling: using the solved transformation matrix ,right and Resample to match the visible light image. Spatial alignment yields the registered image. and ; S1.3 Image Standardization: Standardization is performed on each registered modal image to eliminate the influence of sensor differences and the imaging environment, normalizing pixel values ​​to a uniform numerical range to facilitate subsequent neural network processing. Specifically, this includes: S1.3.1 SAR Image Standardization: Considering the characteristics of speckle noise and large dynamic range in SAR images, radiometric calibration is first performed. Then, logarithmic transformation is used to convert multiplicative noise into additive noise, followed by contrast enhancement. Finally, Z-score standardization is performed. ； in, To prevent small constants with a logarithm of zero, and These are the mean and standard deviation of the image patch, respectively; S1.3.2, Visible light image standardization: [This refers to the standardization of visible light images.] After performing color balance and white balance correction, pixel value normalization is performed: ； in and These are the mean and standard deviation of the visible light image patch, respectively; Or normalize it to the [0,1] interval: ； S1.3.3, Infrared Image Standardization: [This section appears to be incomplete and requires further context.] Blackbody-based non-uniformity correction and atmospheric correction are performed to eliminate the influence of environmental radiation. Then, minimum-maximum normalization is applied to map temperature or grayscale values ​​to a specific range. ； in, and The lower and upper limits of the preset normalization range, or the image. The minimum and maximum values ​​of pixel statistics.

3. The multimodal meta-learning remote sensing reconnaissance target identification method guided by detection mechanism according to claim 2, characterized in that, S2 includes the following steps: S2.1 Feature extraction guided by detection mechanism: The normalized multimodal image output by S1 is compared with... Three parallel, structure-specific feature extraction branches are input respectively, as follows: S2.1.1 SAR Feature Extraction Branch: The backbone network of this branch adopts a deep convolutional neural network with a large receptive field and residual connections to adapt to the electromagnetic scattering characteristics of targets and complex speckle noise in SAR images. Let the network parameters of this branch be... The process of extracting features is as follows: ； in, The extracted SAR mode high-level feature map focuses on characterizing physical information such as the distribution of strong scattering centers and material properties of the target; S2.1.2 Visible Light Feature Extraction Branch: The backbone network of this branch uses a deep convolutional neural network that focuses on texture and edge perception to capture the geometric structure and surface texture characteristics of targets in visible light images. Let the network parameters of this branch be... The process of extracting features is as follows: ； in, High-level feature map of the extracted visible light modes; S2.1.3 Infrared Feature Extraction Branch: The backbone network of this branch uses a deep convolutional neural network that enhances thermal contour contrast to focus on the thermal radiation distribution and contour characteristics of the target in the infrared image. Let the network parameters of this branch be... The process of extracting features is as follows: ； in, High-level feature maps of the extracted infrared modes; S2.2 Feature Fusion Based on Attention Mechanism: The feature maps extracted from the above three branches are... , , By performing channel-dimensional splicing, preliminary fused features are obtained. ; ； Where [;] represents a splicing operation along the channel dimension; Then, Input a cross-modal channel attention module to adaptively learn and weight the importance of different modal feature channels. The specific steps of this module are as follows: S2.2.1, Obtaining Global Context Information: Global average pooling is performed on each channel to generate a channel statistics descriptor. The formula for calculating the c-th element is: ； S2.2.2 Attention Weight Calculation: The descriptor z is input into a gating mechanism consisting of two fully connected layers to capture the non-linear dependencies between channels and output the attention weights of each channel. : ； in, , This is the weight matrix of the fully connected layer. For scaling ratio, It is the ReLU activation function. This is the Sigmoid activation function, used to map weights to the range of 0 to 1; S2.2.3 Feature Reweighting: The learned attention weights s are combined with the original concatenated features. Channel-by-channel multiplication yields the enhanced fusion features. : ； Finally, the enhanced unified feature representation is output. This feature will be used as input to the subsequent meta-learning classifier.

4. The multimodal meta-learning remote sensing reconnaissance target identification method guided by detection mechanism according to claim 3, characterized in that, S3 includes the following steps: S3.1 Construction of the Meta-Training Task Set: From the basic dataset containing a large number of categories, construct the meta-training set according to the model-independent meta-learning paradigm. This set consists of a large number of meta-tasks The structure is as follows: Each task simulates an N-class K-sample classification problem, specifically: S3.1.1 Randomly sample N categories from the basic category set; S3.1.2 From these N categories, randomly sample K labeled sample images from each category to form the support set for this task. In the formula, i is the index of the meta-task, and j is the sequence number of the sample in the set. For the j-th multimodal image sample, for Corresponding category tags; S3.1.

3. From the same N categories, sample several more samples from each category to form the query set for this task. In the formula, Q is the number of samples, i is the index of the meta-task, and j is the sequence number of the sample in the set. For the j-th multimodal image sample, for Corresponding category tags; S3.2 Model Initialization: Initialize the parameters of the entire model, including the parameters of the three feature extraction branch networks defined in S2. , , Feature fusion module parameters And the parameters of a base classifier g(·) All trainable parameters are collectively referred to as model parameters. ; S3.3 Meta-learning Training Loop: Employing a model-independent meta-learning framework, the model parameters are updated through inner and outer loops. The optimizations are as follows: S3.3.1 Inner Loop Update: For each metatask in the current batch , will support the set of tasks The input model first obtains fused features through a feature extraction and fusion network. Then, the predicted probability is calculated using the base classifier. And loss, the loss function is usually cross-entropy loss: ； In the mission Above, regarding model parameters Perform one or more steps of gradient descent to obtain task-specific adapted parameters. : ； Where α is the inner loop learning rate, which is a hyperparameter; S3.3.2, Outer Loop Update: Use the adapted parameters obtained in the previous step. In the query set of the corresponding task Calculate loss This loss is used to evaluate the generalization performance of the adapted model on new samples; After all tasks in the current batch are completed, aggregate the losses of all tasks on the query set and calculate the meta-loss: ； Meta-loss is calculated using backpropagation. Relative to model parameters The gradient is used to update the model parameters. : ； Where β is the outer loop learning rate, i.e., the meta-learning rate, and this update step optimizes the model parameters. Its generalization potential enables the model to perform well on new and unseen tasks after a small number of gradient steps. Through repeated iterations of S3.3, the model parameters They were trained to become a set of excellent initialization points.

5. The multimodal meta-learning remote sensing reconnaissance target identification method guided by detection mechanism according to claim 4, characterized in that, S4 includes the following steps: S4.1 New Task Support Set Construction: During the deployment phase, when it is necessary to identify M entirely new target categories that have not appeared in the meta-training, and each category has only one... When there are a few labeled samples, these few labeled samples are used to construct the support set for the new task: ； in, Labels for the new category; S4.2, Fast Model Adaptation: Loading model parameters trained in S3 that have good generalization potential. Starting from this point, only the support set for the new task will be used. Perform multi-step gradient descent to quickly fine-tune the model, specifically including: S4.2.1 Forward Propagation and Loss Calculation: Input the model, and through feature extraction, fusion, and classifiers, calculate the model's loss on the current support set. : ； Where g is the current gradient step number, and the model parameters are initially set to g=0. ; S4.2.2 Parameter Update: Calculate the gradient based on the loss and update the model parameters. This update process can be applied to all parameters, or, to improve efficiency and prevent overfitting, only the parameters of the base classifier can be updated. Parameters of the feature fusion module : ； in, To adapt the learning rate to the stage, it can be compared with the inner loop learning rate during meta-training. The parameters of the feature extraction branches, whether the parameters are the same or different. It is usually frozen during this process. After multiple updates, the optimized model parameters for the new M categories are obtained. The model has quickly transferred its recognition capabilities to new target categories.

Images