Detection mechanism guided multi-mode element learning remote sensing reconnaissance target identification method

Abstract

The invention discloses a multi-mode element learning remote sensing reconnaissance target identification method guided by a detection mechanism, and belongs to the field of computer vision and remote sensing image processing. The core of the method is as follows: multi-modal feature extraction and fusion guided by a detection mechanism: aiming at the electromagnetic scattering characteristic of an SAR image, the geometric texture characteristic of a visible light image and the thermal radiation characteristic of an infrared image, respectively designing a special feature extraction network, and introducing a cross-modal fusion module based on an attention mechanism, feature enhancement expression of physical information complementation is realized; according to the meta learning training normal form, an optimization-based meta learning device is constructed, and training is performed on a large number of multi-modal remote sensing task sets, so that the model obtains the capability of quickly learning new target categories from a small number of samples. According to the method, a physical detection mechanism and data-driven meta learning are combined, and the problems that a traditional method is low in recognition precision and poor in generalization ability in a complex reconnaissance scene with scarce samples, variable target types and difficulty in SAR image interpretation are effectively solved.

Description

Technical Field

[0001] This invention belongs to the technical field of computer vision and remote sensing image processing, specifically relating to a multimodal meta-learning remote sensing reconnaissance target recognition method guided by detection mechanism. Background Technology

[0002] Remote sensing reconnaissance target identification is a core component of modern military perception, disaster monitoring, and emergency response. With the development of remote sensing technology, the fusion and identification of heterogeneous data (i.e., multimodal remote sensing data) acquired by multiple sensors such as synthetic aperture radar (SAR), visible light, and infrared has become a key approach to improving target detection and identification performance. SAR (Synthetic Aperture Radar) has all-weather, all-day operating capabilities, and its images reflect the geometric structure and dielectric properties of ground targets, as well as electromagnetic scattering mechanisms. Visible light images have high resolution and contain rich geometric textures and detailed information; infrared images can effectively capture the thermal radiation characteristics of targets, enabling nighttime detection and camouflage identification. These three modalities are naturally complementary in terms of information.

[0003] However, existing multimodal remote sensing target recognition methods, especially when dealing with complex reconnaissance scenarios, still face the following severe challenges: (1) the recognition dilemma under the condition of scarce samples; (2) the lack of mechanism for multimodal feature fusion; (3) insufficient model generalization and rapid adaptation capabilities; and (4) difficulties in SAR image interpretation.

[0004] To address the aforementioned issues of scarce samples, lack of multimodal fusion mechanisms, insufficient model generalization ability, and difficulties in SAR image interpretation, this invention proposes a multimodal meta-learning remote sensing reconnaissance target recognition method guided by detection mechanisms. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal meta-learning remote sensing reconnaissance target identification method guided by detection mechanism, which aims to combine physical detection mechanism with meta-learning paradigm to solve the problem of rapid and accurate identification of unknown or rare targets under conditions of scarce samples.

[0006] To achieve the above objectives, the present invention provides a multimodal meta-learning remote sensing reconnaissance target identification method guided by detection mechanism, comprising the following steps: S1. Acquire SAR, visible light, and infrared images of the same reconnaissance area, and perform registration and standardization processing to form a multimodal image pair; S2. Input the registered multimodal image into three parallel feature extraction networks to extract features, and then fuse the extracted multimodal features. S3. Construct a meta-training set containing a large number of multi-classification tasks, and use a model-independent meta-learning framework to train the feature extraction network and classifier. S4. When faced with a new target category, use a small number of labeled samples to quickly fine-tune the model so that it can quickly adapt to the new task. S5. Input the target to be identified into the adapted model, output the recognition result and confidence score, and complete the target recognition in a small sample scenario.

[0007] Furthermore, in step S1, SAR images, visible light images, and infrared images of the same reconnaissance area are acquired and registered and normalized to form a multimodal image pair, specifically including the following: S1.1 Multimodal Remote Sensing Data Acquisition: By coordinating deployment or utilizing platforms equipped with multi-sensor payloads, synthetic aperture radar (SAR), visible light, and infrared images of the same geographic reconnaissance area are acquired synchronously or quasi-synchronously, ensuring high temporal and spatial consistency among the modal data, thereby constructing the original multimodal remote sensing data triplet. , The constituent elements are as follows: ; in This represents a synthetic aperture radar (SAR) image. Represents a visible light image. Represents an infrared image; S1.2 Spatial Registration: Spatial registration is performed on the acquired multimodal images to ensure that the pixels are precisely aligned in geometric positions, thereby eliminating spatial inconsistencies caused by differences in imaging viewpoint, platform pose, and resolution. Specifically, this includes: S1.2.1 Control Point Extraction and Matching: In visible light images Extract salient feature points (such as SIFT or ORB feature points) as a reference control point set. Subsequently, in SAR images and infrared images Extract the corresponding feature point set. and ; S1.2.2, Solving the transformation model: Based on matching feature point pairs and The spatial transformation model is estimated using the random sampling consensus algorithm. and This model is typically a projection transformation or an affine transformation, and its mathematical form is: ; in, The coordinates of the source image. The coordinates are after transformation to the reference image coordinate system. The transformation matrix; S1.2.3, Image resampling: using the solved transformation matrix ,right and Resample to match the visible light image. Spatial alignment yields the registered image. and ; S1.3 Image Standardization: Standardization is performed on each registered modal image to eliminate the influence of sensor differences and the imaging environment, normalizing pixel values ​​to a uniform numerical range to facilitate subsequent neural network processing. This specifically includes: S1.3.1 SAR Image Standardization: Considering the characteristics of speckle noise and large dynamic range in SAR images, radiometric calibration is first performed. Then, logarithmic transformation is used to convert multiplicative noise into additive noise, followed by contrast enhancement. Finally, Z-score standardization is performed. ; in, To prevent small constants with a logarithm of zero, and These are the mean and standard deviation of the image patch, respectively; S1.3.2, Visible light image standardization: [This refers to the standardization of visible light images.] After performing color balance and white balance correction, pixel value normalization is performed: ; in and These are the mean and standard deviation of the visible light image patch, respectively: Or normalize it to the [0,1] interval: ; S1.3.3, Infrared Image Standardization: [This section appears to be incomplete and requires further context.] Blackbody-based non-uniformity correction and atmospheric correction are performed to eliminate the influence of environmental radiation. Then, minimum-maximum normalization is applied to map temperature or grayscale values ​​to a specific range. ; in, and The lower and upper limits of the preset normalization range, or the image. The minimum and maximum values ​​of pixel statistics.

[0008] Furthermore, in step S2, the registered multimodal image is input into three parallel feature extraction networks for feature extraction, and the extracted multimodal features are fused, specifically including the following: S2.1 Feature extraction guided by detection mechanism: The normalized multimodal image output by S1 is compared with... Three parallel, structure-specific feature extraction branches are input respectively, as follows: S2.1.1 SAR Feature Extraction Branch: The backbone network of this branch adopts a deep convolutional neural network (such as a ResNet variant) with a large receptive field and residual connections to adapt to the electromagnetic scattering characteristics of targets and complex speckle noise in SAR images. Let the network parameters of this branch be... The process of extracting features is as follows: ; in, The extracted SAR mode high-level feature map focuses on characterizing the target's physical information such as the distribution of strong scattering centers and material properties; S2.1.2 Visible Light Feature Extraction Branch: The backbone network of this branch uses a deep convolutional neural network (such as VGG or CNN with attention mechanism) that focuses on texture and edge perception to capture the geometric structure and surface texture characteristics of the target in the visible light image. Let the network parameters of this branch be... The process of extracting features is as follows: ; in, High-level feature map of the extracted visible light modes; S2.1.3 Infrared Feature Extraction Branch: The backbone network of this branch uses a deep convolutional neural network that enhances thermal contour contrast to focus on the thermal radiation distribution and contour characteristics of the target in the infrared image. Let the network parameters of this branch be... The process of extracting features is as follows: ; in, High-level feature maps of the extracted infrared modes; S2.2 Feature Fusion Based on Attention Mechanism: The feature maps extracted from the above three branches are... , , By performing channel-dimensional splicing, preliminary fused features are obtained. ; ; Where [;] represents a splicing operation along the channel dimension; Then Input a cross-modal channel attention module to adaptively learn and weight the importance of different modal feature channels. The specific steps of this module are as follows: S2.2.1, Obtaining Global Context Information: Global average pooling is performed on each channel to generate a channel statistics descriptor. The formula for calculating the c-th element is: ; S2.2.2 Attention Weight Calculation: The descriptor z is input into a gating mechanism consisting of two fully connected layers to capture the non-linear dependencies between channels and output the attention weights of each channel. : ; in, , This is the weight matrix of the fully connected layer. For scaling ratio, It is the ReLU activation function. This is the Sigmoid activation function, used to map weights to the range of 0 to 1; S2.2.3 Feature Reweighting: The learned attention weights s are combined with the original concatenated features. Channel-by-channel multiplication yields the enhanced fusion features. : ; Finally, output This feature will be used as input to the subsequent meta-learning classifier.

[0009] Furthermore, in S3, a meta-training set containing a large number of multi-classification tasks is constructed, and a model-independent meta-learning framework is used to train the feature extraction network and the classifier, specifically including the following: S3.1 Construction of the Meta-Training Task Set: From the basic dataset containing a large number of categories, construct the meta-training set according to the model-independent meta-learning paradigm. This set consists of a large number of meta-tasks. The structure is as follows: Each task simulates an N-way K-shot (N classes, K samples) classification problem, specifically: S3.1.1 Randomly sample N categories from the basic category set; S3.1.2 From these N categories, randomly sample K labeled sample images from each category to form the support set for this task. In the formula, i is the index of the meta-task, and j is the sequence number of the sample in the set. For the j-th multimodal image sample, for Corresponding category tags; S3.1.3. From the same N categories, sample several more samples from each category to form the query set for this task. In the formula, Q is the number of samples, i is the index of the meta-task, and j is the sequence number of the sample in the set. For the j-th multimodal image sample, for Corresponding category tags; S3.2 Model Initialization: Initialize the parameters of the entire model, including the parameters of the three feature extraction branch networks defined in S2. , , Feature fusion module parameters And the parameters of a base classifier g(·) All trainable parameters are collectively referred to as ; S3.3 Meta-learning Training Loop: Employing a model-independent meta-learning framework, the model parameters are updated through inner and outer loops. The optimizations are as follows: S3.3.1 Inner Loop Update: For each metatask in the current batch , will support the set of tasks The input model first obtains fused features through a feature extraction and fusion network. Then, the predicted probability is calculated using the base classifier. And loss, the loss function is usually cross-entropy loss: ; In the mission Above, perform one or more steps of gradient descent on the model parameters Θ to obtain the task-specific adapted parameters. : ; Where α is the inner loop learning rate, which is a hyperparameter; S3.3.2, Outer Loop Update: Use the adapted parameters obtained in the previous step. In the query set of the corresponding task Calculate loss This loss is used to evaluate the generalization performance of the adapted model on new samples; After all tasks in the current batch are completed, aggregate the losses of all tasks on the query set and calculate the meta-loss: ; The most crucial step is to calculate the meta-loss through backpropagation. The gradient relative to the original parameters Θ, and using this gradient to update the initial parameters Θ of the model: ; Where β is the outer loop learning rate (meta-learning rate), this update step optimizes the generalization potential of the model's initial parameter Θ, so that the model can perform well on new and unseen tasks after experiencing a small number of gradient steps. Through repeated iterations of S3.3, the model parameters The initial parameters are trained to form a set of excellent initialization points. When encountering a small sample task containing new categories, the initial parameters can be updated in a few inner loops based on this set of initial parameters to quickly adapt to the new task and achieve high-precision target recognition.

[0010] Furthermore, in S4, when faced with a new target category, the model is quickly fine-tuned using a small number of labeled samples to enable it to adapt quickly to the new task. This specifically includes the following: S4.1 New Task Support Set Construction: During the deployment phase, when it is necessary to identify M entirely new target categories that have not appeared in the meta-training, and each category has only one... When there are a few labeled samples, these few labeled samples are used to construct the support set for the new task: ; in, Labels for the new category; S4.2, Fast Model Adaptation: Loading initial parameters of the model trained in S3, which have good generalization potential. Starting from this point, only the support set for the new task will be used. Perform G-step gradient descent to quickly fine-tune the model, specifically including: S4.2.1 Forward Propagation and Loss Calculation: Input the model, and through feature extraction, fusion, and classifiers, calculate the model's loss on the current support set. : ; Where g is the current gradient step number, and the model parameters are Θ when g=0 in the initial step; S4.2.2 Parameter Update: Calculate the gradient based on the loss and update the model parameters. This update process can be applied to all parameters, or, to improve efficiency and prevent overfitting, only the parameters of the base classifier can be updated. Parameters of the feature fusion module : ; in, To adapt the learning rate to the stage, it can be compared with the inner loop learning rate during meta-training. Same or different. Parameters of the feature extraction branch. It is usually frozen during this process; After G-step updates, we obtain the optimized model parameters for the new M categories. The model has quickly transferred its recognition capabilities to new target categories.

[0011] Furthermore, in step S5, the target to be identified is input into the adapted model, and the identification result and confidence score are output to complete the target identification in a small sample scenario. Specifically, this includes the following: S5.1 Forward reasoning of the target to be identified: The image of the target to be identified from the new category (i.e., the query sample) is used to identify the target image to be identified. The input is fed into the model after S4 fast adaptation; S5.2 Feature Extraction and Classification: The model first performs feature extraction and classification... Perform the same preprocessing as during training, and then obtain its enhanced fused feature representation through a multi-stream feature extraction network and a cross-modal fusion module. Then, this feature is input into a classifier that has been adapted to the new task. Calculate the logits (scores) of each new category: ; in, It is an M-dimensional vector; S5.3 Recognition Results and Confidence Output: The target recognition module performs Softmax normalization on the logits vector z to obtain the probability distribution of the target belonging to each new category: ; The final system output is: Recognition result: The category with the highest probability is selected as the final recognition label. ; Confidence score: The highest probability value is used as the confidence score for this identification. .

[0012] Beneficial Effects: This invention effectively mines and fuses the electromagnetic scattering characteristics of SAR images, the geometric texture characteristics of visible light images, and the thermal radiation characteristics of infrared images by designing a mechanism-guided multi-stream feature extraction network and a cross-modal attention fusion module. Simultaneously, by combining a model-independent meta-learning training paradigm, the model gains the ability to rapidly learn new targets from small samples across numerous multimodal remote sensing tasks. Ultimately, it achieves the fusion of physical detection mechanisms and data-driven meta-learning, significantly improving recognition accuracy and generalization ability in complex reconnaissance scenarios with scarce samples and diverse target types. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the present invention; Figure 2 This is a schematic diagram illustrating the process of registering and processing three types of images to construct multimodal image pairs in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of feature extraction and fusion of multimodal image pairs in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the process of training a task set using a meta-learning framework in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the process of the model fine-tuning and identifying targets when faced with new target categories in an embodiment of the present invention. Detailed Implementation

[0014] The invention will now be further described with reference to the accompanying drawings. Example

[0015] like Figure 1 As shown, a multimodal meta-learning remote sensing reconnaissance target recognition method guided by detection mechanism includes the following steps: S1. Acquire SAR, visible light, and infrared images of the same reconnaissance area, and perform registration and standardization processing to form a multimodal image pair; S2. Input the registered multimodal image into three parallel feature extraction networks to extract features, and then fuse the extracted multimodal features. S3. Construct a meta-training set containing a large number of multi-classification tasks, and use a model-independent meta-learning framework to train the feature extraction network and classifier. S4. When faced with a new target category, use a small number of labeled samples to quickly fine-tune the model so that it can quickly adapt to the new task. S5. Input the target to be identified into the adapted model, output the recognition result and confidence score, and complete the target recognition in a small sample scenario.

[0016] Furthermore, such as Figure 2 As shown, in step S1, SAR images, visible light images, and infrared images of the same reconnaissance area are acquired, and registration and normalization processes are performed to form a multimodal image pair. The specific steps are as follows: S1.1 Multimodal Remote Sensing Data Acquisition: By coordinating deployment or utilizing platforms equipped with multi-sensor payloads, synthetic aperture radar (SAR), visible light, and infrared images of the same geographic reconnaissance area are acquired synchronously or quasi-synchronously, ensuring high temporal and spatial consistency among the modal data, thereby constructing the original multimodal remote sensing data triplet. , The constituent elements are as follows: ; in This represents a synthetic aperture radar (SAR) image. Represents a visible light image. Represents an infrared image; S1.2 Spatial Registration: Spatial registration is performed on the acquired multimodal images to ensure that the pixels are precisely aligned in geometric positions, thereby eliminating spatial inconsistencies caused by differences in imaging viewpoint, platform pose, and resolution. Specifically, this includes: S1.2.1 Control Point Extraction and Matching: In visible light images Extract salient feature points (such as SIFT or ORB feature points) as a reference control point set. Subsequently, in SAR images and infrared images Extract the corresponding feature point set. and ; S1.2.2, Solving the transformation model: Based on matching feature point pairs and The spatial transformation model is estimated using the random sampling consensus algorithm. and This model is typically a projection transformation or an affine transformation, and its mathematical form is: ; in, The coordinates of the source image. The coordinates are after transformation to the reference image coordinate system. The transformation matrix; S1.2.3, Image resampling: using the solved transformation matrix ,right and Resample to match the visible light image. Spatial alignment yields the registered image. and ; S1.3 Image Standardization: Standardization is performed on each registered modal image to eliminate the influence of sensor differences and the imaging environment, normalizing pixel values ​​to a uniform numerical range to facilitate subsequent neural network processing. This specifically includes: S1.3.1 SAR Image Standardization: Considering the characteristics of speckle noise and large dynamic range in SAR images, radiometric calibration is first performed. Then, logarithmic transformation is used to convert multiplicative noise into additive noise, followed by contrast enhancement. Finally, Z-score standardization is performed. ; in, To prevent small constants with a logarithm of zero, and These are the mean and standard deviation of the image patch, respectively; S1.3.2, Visible light image standardization: [This refers to the standardization of visible light images.] After performing color balance and white balance correction, pixel value normalization is performed: ; in and These are the mean and standard deviation of the visible light image patch, respectively: Or normalize it to the [0,1] interval: ; S1.3.3, Infrared Image Standardization: [This section appears to be incomplete and requires further context.] Blackbody-based non-uniformity correction and atmospheric correction are performed to eliminate the influence of environmental radiation. Then, minimum-maximum normalization is applied to map temperature or grayscale values ​​to a specific range. ; in, and The lower and upper limits of the preset normalization range, or the image. The minimum and maximum values ​​of pixel statistics.

[0017] Furthermore, such as Figure 3 As shown, in step S2, the registered multimodal image is input into three parallel feature extraction networks for feature extraction, and the extracted multimodal features are fused. The specific steps are as follows: S2.1 Feature extraction guided by detection mechanism: The normalized multimodal image output by S1 is compared with... Three parallel, structure-specific feature extraction branches are input respectively, as follows: S2.1.1 SAR Feature Extraction Branch: The backbone network of this branch adopts a deep convolutional neural network (such as a ResNet variant) with a large receptive field and residual connections to adapt to the electromagnetic scattering characteristics of targets and complex speckle noise in SAR images. Let the network parameters of this branch be... The process of extracting features is as follows: ; in, The extracted SAR mode high-level feature map focuses on characterizing the target's physical information such as the distribution of strong scattering centers and material properties; S2.1.2 Visible Light Feature Extraction Branch: The backbone network of this branch uses a deep convolutional neural network (such as VGG or CNN with attention mechanism) that focuses on texture and edge perception to capture the geometric structure and surface texture characteristics of the target in the visible light image. Let the network parameters of this branch be... The process of extracting features is as follows: ; in, High-level feature map of the extracted visible light modes; S2.1.3 Infrared Feature Extraction Branch: The backbone network of this branch uses a deep convolutional neural network that enhances thermal contour contrast to focus on the thermal radiation distribution and contour characteristics of the target in the infrared image. Let the network parameters of this branch be... The process of extracting features is as follows: ; in, High-level feature maps of the extracted infrared modes; S2.2 Feature Fusion Based on Attention Mechanism: The feature maps extracted from the above three branches are... , , By performing channel-dimensional splicing, preliminary fused features are obtained. ; ; Where [;] represents a splicing operation along the channel dimension; Then Input a cross-modal channel attention module to adaptively learn and weight the importance of different modal feature channels. The specific steps of this module are as follows: S2.2.1, Obtaining Global Context Information: Global average pooling is performed on each channel to generate a channel statistics descriptor. The formula for calculating the c-th element is: ; S2.2.2 Attention Weight Calculation: The descriptor z is input into a gating mechanism consisting of two fully connected layers to capture the non-linear dependencies between channels and output the attention weights of each channel. : ; in, , This is the weight matrix of the fully connected layer. For scaling ratio, It is the ReLU activation function. This is the Sigmoid activation function, used to map weights to the range of 0 to 1; S2.2.3 Feature Reweighting: The learned attention weights s are combined with the original concatenated features. Channel-by-channel multiplication yields the enhanced fusion features. : ; Finally, output This feature will be used as input to the subsequent meta-learning classifier.

[0018] Furthermore, such as Figure 4 As shown, in step S3, a meta-training set containing a large number of multi-classification tasks is constructed. A model-independent meta-learning framework is used to train the feature extraction network and the classifier. The specific steps are as follows: S3.1 Construction of the Meta-Training Task Set: From the basic dataset containing a large number of categories, construct the meta-training set according to the model-independent meta-learning paradigm. This set consists of a large number of meta-tasks The structure is as follows: Each task simulates an N-way K-shot (N classes, K samples) classification problem, specifically: S3.1.1 Randomly sample N categories from the basic category set; S3.1.2 From these N categories, randomly sample K labeled sample images from each category to form the support set for this task. In the formula, i is the index of the meta-task, and j is the sequence number of the sample in the set. For the j-th multimodal image sample, for Corresponding category tags; S3.1.3. From the same N categories, sample several more samples from each category to form the query set for this task. In the formula, Q is the number of samples, i is the index of the meta-task, and j is the sequence number of the sample in the set. For the j-th multimodal image sample, for Corresponding category tags; S3.2 Model Initialization: Initialize the parameters of the entire model, including the parameters of the three feature extraction branch networks defined in S2. , , Feature fusion module parameters And the parameters of a base classifier g(·) All trainable parameters are collectively referred to as ; S3.3 Meta-learning Training Loop: Employing a model-independent meta-learning framework, the model parameters are updated through inner and outer loops. The optimizations are as follows: S3.3.1 Inner Loop Update: For each metatask in the current batch , will support the set of tasks The input model first obtains fused features through a feature extraction and fusion network. Then, the predicted probability is calculated using the base classifier. And loss, the loss function is usually cross-entropy loss: ; In the mission Above, perform one or more steps of gradient descent on the model parameters Θ to obtain the task-specific adapted parameters. : ; Where α is the inner loop learning rate, which is a hyperparameter; S3.3.2, Outer Loop Update: Use the adapted parameters obtained in the previous step. In the query set of the corresponding task Calculate loss This loss is used to evaluate the generalization performance of the adapted model on new samples; After all tasks in the current batch are completed, aggregate the losses of all tasks on the query set and calculate the meta-loss: ; The most crucial step is to calculate the meta-loss through backpropagation. The gradient relative to the original parameters Θ, and using this gradient to update the initial parameters Θ of the model: ; Where β is the outer loop learning rate (meta-learning rate), this update step optimizes the "generalization potential" of the model's initial parameter Θ, so that the model can perform well on new and unseen tasks after experiencing a small number of gradient steps; Through repeated iterations of S3.3, the model parameters The initial parameters are trained to form a set of excellent initialization points. When encountering a small sample task containing new categories, the initial parameters can be updated in a few inner loops based on this set of initial parameters to quickly adapt to the new task and achieve high-precision target recognition.

[0019] Furthermore, such as Figure 5 As shown, in step S4, when faced with a new target category, the model is quickly fine-tuned using a small number of labeled samples to enable it to adapt quickly to the new task. The specific steps are as follows: S4.1 New Task Support Set Construction: During the deployment phase, when it is necessary to identify M entirely new target categories that have not appeared in the meta-training, and each category has only one... When there are a few labeled samples, these few labeled samples are used to construct the support set for the new task: ; in, Labels for the new category; S4.2, Fast Model Adaptation: Loading initial parameters of the model trained in S3, which have good generalization potential. Starting from this point, only the support set for the new task will be used. Perform G-step gradient descent to quickly fine-tune the model: S4.2.1 Forward Propagation and Loss Calculation: Input the model, and through feature extraction, fusion, and classifiers, calculate the model's loss on the current support set. : ; Where g is the current gradient step number, and the model parameters are Θ when g=0 in the initial step; S4.2.2 Parameter Update: Calculate the gradient based on the loss and update the model parameters. This update process can be applied to all parameters, or, to improve efficiency and prevent overfitting, only the parameters of the base classifier can be updated. Parameters of the feature fusion module : ; in, To adapt the learning rate to the stage, it can be compared with the inner loop learning rate during meta-training. Same or different. Parameters of the feature extraction branch. It is usually frozen during this process; After G-step updates, we obtain the optimized model parameters for the new M categories. The model has quickly transferred its recognition capabilities to new target categories.

[0020] Furthermore, such as Figure 5 As shown, in step S5, the target to be identified is input into the adapted model, and the recognition result and confidence score are output to complete the target recognition in a small sample scenario. The specific steps are as follows: S5.1 Forward reasoning of the target to be identified: The image of the target to be identified from the new category (i.e., the query sample) is used to identify the target image to be identified. The input is fed into the model after S4 fast adaptation; S5.2 Feature Extraction and Classification: The model first performs feature extraction and classification... Perform the same preprocessing as during training, and then obtain its enhanced fused feature representation through a multi-stream feature extraction network and a cross-modal fusion module. Then, this feature is input into a classifier that has been adapted to the new task. Calculate the logits (scores) of each new category: ; in, It is an M-dimensional vector; S5.3 Recognition Results and Confidence Output: The target recognition module performs Softmax normalization on the logits vector z to obtain the probability distribution of the target belonging to each new category: ; The final system output is: Recognition result: The category with the highest probability is selected as the final recognition label.

[0021] Confidence score: The highest probability value is used as the confidence score for this identification. .

Claims

1. A method for detecting mechanism-guided multi-modal meta-learning remote sensing reconnaissance target identification, characterized in that, The method comprises the following steps: S1, acquiring SAR images, visible light images and infrared images of the same reconnaissance area, and performing registration and standardization processing to form a multi-modal image pair; S2, inputting the registered multi-modal image pair into three parallel feature extraction networks for feature extraction, and fusing the extracted multi-modal features; S3, constructing a meta-training set containing a large number of multi-classification tasks, and training the feature extraction network and the classifier by using a model-independent meta-learning framework; S4, when facing a new target class, quickly fine-tuning the model by using a small amount of labeled samples, so that the model quickly adapts to the new task; S5, inputting the target to be identified into the adapted model, outputting the recognition result and the confidence, and completing the target recognition in the small sample scene.

2. The method of claim 1, wherein the method is characterized by, The S1 comprises the following steps: S1.1, Multimodal remote sensing data acquisition: By coordinating the deployment or using platforms with multi-sensor payloads, synthetic aperture radar images, visible light images and infrared images of the same geographic reconnaissance area are synchronously or quasi-synchronously acquired to ensure that the data of each mode has high consistency in time and space, thereby constructing the original multimodal remote sensing data triplets , The constituent elements of the application are as follows: ； wherein represents a synthetic aperture radar image, represents a visible light image, represents an infrared image; S1.2, spatial registration: performing spatial registration on the acquired multi-modal images to accurately align the pixels in the geometric position, so as to eliminate the spatial inconsistency caused by imaging angle, platform attitude and resolution difference; specifically comprising: S1.2.1, Control point extraction and matching: in the visible light image The significant feature points are extracted as the reference control point set Then, the corresponding feature point sets and are extracted in the SAR image and the infrared image respectively; S1.2.2, solving the transformation model: based on matched feature points and using the random sample consensus algorithm to estimate the spatial transformation model and The model is a projection transformation or an affine transformation, which has a mathematical form: ； wherein, is the source image coordinate, is the coordinate transformed to the reference image coordinate system, is the transformation matrix; S1.2.3, Image resampling: using the solved transformation matrix ,right and Resample to match the visible light image. Spatial alignment yields the registered image. and ; S1.3, image standardization: performing standardization processing on each modal image after registration, so as to eliminate the influence of sensor difference and imaging environment, normalize the pixel value to a unified numerical range, and facilitate subsequent neural network processing, specifically comprising: S1.3.1, SAR image standardization: according to the characteristics of SAR image, such as coherent speckle noise and large dynamic range, first performing radiation calibration, then converting the multiplicative noise into additive noise by using logarithmic transformation, and performing contrast enhancement, and finally performing Z-Score standardization: ； wherein, To prevent small constants with logarithm zero, and are the mean and standard deviation of the image block, respectively. S1.3.2, visible light image standardization: on After color equalization and white balance correction, pixel value normalization is performed: ； wherein and are the mean and standard deviation of the visible light image patch, respectively: Or normalize it to the interval [0, 1]: ； S1.3.3, Infrared Image Standardization: [This section appears to be incomplete and requires further context.] Blackbody-based non-uniformity correction and atmospheric correction are performed to eliminate the influence of environmental radiation. Then, minimum-maximum normalization is applied to map temperature or grayscale values ​​to a specific range. ； wherein, and are preset lower and upper limits of the normalization range, or are the minimum and maximum values of the pixel statistics of the image .

3. The method of claim 1, wherein the method is characterized by, The S2 comprises the following steps: S2.1 Feature extraction guided by detection mechanism: The normalized multimodal image output by S1 is compared with... Three parallel, structure-specific feature extraction branches are input respectively, as follows: S2.1.1, SAR feature extraction branch: the main network of this branch adopts a deep convolutional neural network with large receptive field and residual connection to adapt to the electromagnetic scattering characteristics of the target in the SAR image and the complex speckle noise. The network parameters of this branch are set as The process of extracting features is as follows: ； wherein, is the extracted SAR modality high-level feature map, which focuses on representing the physical information of the strong scattering center distribution and material properties of the target. S2.1.2, visible light feature extraction branch: the main network of this branch adopts a deep convolutional neural network focusing on texture and edge perception to capture the geometric structure and surface texture characteristics of the target in the visible light image, and the network parameters of this branch are set as The process of extracting features is as follows: ； wherein, is the extracted visible light modality high-level feature map; S2.1.3, infrared feature extraction branch: the backbone network of this branch adopts a deep convolutional neural network capable of enhancing the contrast of thermal contours, so as to focus on the thermal radiation distribution and contour characteristics of the target in the infrared image. The network parameters of this branch are set as The process of extracting features is as follows: ； wherein, is the extracted infrared modality high-level feature map; S2.2, feature fusion based on attention mechanism: the feature maps extracted by the above three branches are spliced in the channel dimension to obtain preliminary fused features , , ;​ ； Where [;] represents the concatenation operation along the channel dimension; Subsequently, the input image is input into a cross-modal channel attention module to adaptively learn and weight the importance of different modal feature channels, and the specific steps of the module are as follows: An input cross-modal channel attention module is used to adaptively learn and weight the importance of different modal feature channels, and the specific steps of the module are as follows: S2.2.1, Global context information acquisition: perform global average pooling on each channel of to generate a one-channel statistical descriptor The calculation formula of the cth element is: ； S2.2.2, Attention weight calculation: input descriptor z into a gating mechanism consisting of two fully connected layers to capture the nonlinear dependency between channels and output the attention weight for each channel : ； wherein, , is a weight matrix for a fully connected layer, is a scaling ratio, is a ReLU activation function, is a Sigmoid activation function for mapping weights between 0 and 1. S2.2.3, Feature re-weighting: multiply the learned attention weights s with the original concatenated features Channel-wise multiplication to get the enhanced fused features : ； Finally, the enhanced unified feature representation is output which will serve as input to the subsequent meta-learning classifier.

4. The method of claim 1, wherein the method is characterized by, The S3 comprises the following steps: S3.1, meta-training task set construction: from a large category containing a basic data set, according to the model-independent meta-learning paradigm to construct a meta-training set The set is composed of a large number of meta-tasks Each task simulates an N-class K-sample classification problem, specifically: S3.1.1, randomly sampling N categories from the basic category set; S3.1.

2. From the N categories, randomly sample K labeled sample images from each category to form the support set of the task where i is the index of the meta-task, j is the serial number of the sample in the set, is the jth multi-modal image sample, is the corresponding category label; S3.1.3, from the same N categories, resample several samples in each category to constitute the query set of the task , where Q is the number of samples, i is the index of the meta task, j is the sequence number of the sample in the set, is the jth multi-modal image sample, is the corresponding category label; S3.2, Model initialization: initialize the parameters of the whole model, including the parameters of the three feature extraction branch networks defined in S2 , , , the parameters of the feature fusion module and the parameters of a base classifier g(·) , all the trainable parameters are collectively referred to as ; S3.3, meta-learning training loop: adopt the model-independent meta-learning framework, update the model parameters through inner loop update and outer loop update Optimization is performed as follows: S3.3.1, Inner loop update: for each meta-task in the current batch , the task support set , the input model first gets the fused features through the feature extraction and fusion network , then calculates the prediction probability through the base classifier and loss, the loss function usually adopts cross-entropy loss: ； At task above, one or more steps of gradient descent are performed on the model parameters Θ, resulting in task-specific adapted parameters : ； Where α is the inner loop learning rate, which is a hyperparameter; S3.3.2, outer loop update: use the adapted parameters from the previous step , the query set corresponding to the task , compute the loss , the loss is used to evaluate the generalization performance of the adapted model on new samples; After all the tasks in the current batch are completed, aggregate the loss of all tasks on the query set to calculate the meta-loss: ； Compute element loss by backpropagation Gradient with respect to original parameters Θ, and update initial parameters Θ of the model using this gradient: ； Where β is the outer loop learning rate, i.e. the meta-learning rate, and this update step optimizes the generalization potential of the initial model parameters Θ, so that the model can perform well on new and unseen tasks after experiencing a small number of gradient steps; By iteratively repeating S3.3, the model parameters are trained to a set of excellent initialization points.

5. The method of claim 1, wherein the method is characterized by, The S4 comprises the following steps: S4.1, New task support set construction: in the deployment stage, when it is required to recognize M completely new target categories that have not appeared in meta-training, and there are only annotated samples for each category, these few annotated samples are constructed as the support set of the new task: ； wherein is a new class of labels; S4.2, Model rapid self-adaptation: load the initial parameters of the model with good generalization potential obtained through S3 training , using only the support set of the new task as a starting point , perform G-step gradient descent to rapidly fine-tune the model, specifically including: S4.2.1, Forward propagation and loss computation: Compute the loss of the model on the current support set by the feature extraction, fusion and classifier input model, by feature extraction, fusion and classifier : ； Where g is the current gradient step number, and the model parameters are Θ when the initial step g=0; S4.2.2, Parameter update: Compute gradient according to loss, and update model parameters, this update process can be for all parameters, or for efficiency and to prevent overfitting, only update the parameters of the base classifier and parameters of the feature fusion module : ； wherein, The learning rate for the adaptation stage can be the same or different from the inner loop learning rate during meta-training The parameters of the feature extraction branch The parameters of the feature extraction branch are typically frozen during this process, After G-step update, the model parameters optimized for the new M classes are obtained The model has quickly transferred its recognition ability to the new target classes.

6. The method of claim 1, wherein the method is a mechanism-guided multi-modal meta-learning remote sensing reconnaissance target recognition method. The S5 comprises the following steps: S5.1, forward reasoning of the to-be-identified target: the to-be-identified target image, i.e. the query sample, from the new category is input into the model after the fast adaptation of S4; S5.2 Feature Extraction and Classification: The model first performs feature extraction and classification... Perform the same preprocessing as during training, and then obtain its enhanced fused feature representation through a multi-stream feature extraction network and a cross-modal fusion module. Then, this feature is input into a classifier that has been adapted to the new task. Calculate the logits of each new category: ； wherein is an M-dimensional vector; S5.3, recognition result and confidence output: the target recognition module performs Softmax normalization on the logits vector z to obtain the probability distribution of the target belonging to each new category: ； The final system output is: Recognition result: taking the category with the maximum probability as the final recognition label: ； Confidence: taking the maximum probability value as the confidence score of this recognition: 。

Images