An enhanced multi-transform feature extraction matching network model

Abstract

This invention discloses an enhanced multi-transform feature extraction and matching network model, comprising: an EMTDL-MatchNet query set feature reconstruction module; an EMTDL-MatchNet feature learning module; and an EMTDL-MatchNet adaptive matcher. This invention incorporates a residual network module into the feature learning module to mitigate issues such as information loss and attrition during network transmission. It achieves a certain degree of identity transformation (y = x). It also adds shortcut connection branches to strike a balance between linear and nonlinear transformations, minimizing information loss during transmission.

Description

Technical Field

[0001] The present invention relates to imaging radar technology, and in particular to an enhanced multi-transformation feature extraction matching network model. Background Art

[0002] The most widely used imaging radar system is Synthetic Aperture Radar (SAR).

[0003] SAR has become almost synonymous with radar. It's an active Earth observation system that can be installed on platforms such as aircraft, satellites, and spacecraft. It provides all-day, all-weather Earth observation, possessing a certain degree of surface penetration and capable of providing high-resolution imagery using radar signals transmitted in obstructed weather. Consequently, synthetic aperture radar (SAR) has become widely used in battlefield reconnaissance, topographic mapping, geological exploration, and oceanographic observation. Notably, SAR automatic target recognition (ATR), a crucial tool for understanding SAR imagery, has been extensively studied due to its military significance.

[0004] Deep learning for SAR image target recognition is in its infancy, and available SAR image datasets are limited. SAR images use grayscale to represent the scattering intensity of a target. This is affected by imaging mechanisms such as speckle noise, making it difficult to separate the transition areas between the target, shadows, and background. Manual labeling is highly subjective and random, making acquiring large amounts of labeled SAR image data prohibitively labor-intensive. This scarcity of labeled samples poses a challenge to existing SAR target recognition methods.

[0005] Inspired by the rapid learning ability of humans, a new research direction has emerged that enables machine vision systems to rapidly learn new categories from limited labeled data. This is known as few-shot learning. Unlike traditional classification methods, which require hundreds or more labeled examples for each target category to be recognized, few-shot learning aims to perform classification under the constraint that only a few labeled examples from each possible category can be observed before making a prediction for the test instance. The input for few-shot learning consists of a support set and a query set. Specifically, in each episode, C categories are randomly selected from the dataset. Then, K examples from each category are randomly selected as the support set input, and m (m is typically 1) examples are randomly selected as the query set input. The query set consists of the remaining examples from the C categories. There is no overlap between the query set and the support set. The model develops the ability to distinguish between categories by learning features from these C × (K + 1) examples. This is known as the C-way K-shot problem. Therefore, studying deep learning for SAR target recognition with a small number of examples can not only reduce data requirements but also improve recognition accuracy.

[0006] Typically, existing deep learning-based object recognition algorithms require a large number of training samples to achieve satisfactory generalization performance and mitigate overfitting. To address the issue of insufficient image annotation samples, a large number of studies on small-sample learning object recognition algorithms have emerged.

[0007] Some have proposed prototype network models, which have been successfully applied to skin disease diagnosis and hyperspectral image classification. This network projects samples into a metric space where similar samples are closer and heterogeneous samples are farther apart. This can reduce the impact of overfitting caused by insufficient data and improve the network's generalization and recognition capabilities.

[0008] Some approaches address the scarcity of labeled data in the SAR domain through unsupervised learning. The idea is to use a large amount of unlabeled SAR data to train an autoencoder. As a result, the embedding space learned by the autoencoder is discriminative and can be used as a means of better feature extraction, benefiting from knowledge transfer. The trained encoder subnetwork can be connected to a classifier network, and both can be fine-tuned using the labeled portion of the data to map data points to the labeled space.

[0009] Some researchers have proposed a matching network model for one-shot learning by enhancing the effectiveness of memory neural networks through extended external information. This model calculates label weights using the prototype mapping concept and the cosine formula. The predicted query set labels are then multiplied by the support set labels.

[0010] Some have proposed a Conv-BiLSTM prototypical network, designing a convolutional bidirectional long short-term memory (conv-BiLSTM) network to replace the conventional CNN structure as a feature extractor. This makes SAR images insensitive to orientation changes.

[0011] Some researchers have proposed DLA-MatchNet, which uses two modules, feature learning and matcher, to extract channel-space attention features and adaptively match them. DLA-MatchNet uses two attention fusion schemes to help CNNs learn shallow simple features and deep semantic features, respectively.

[0012] The distance calculation formulas used in these models are inevitably inflexible and fail to perform well on all datasets. Even when they demonstrate good results on the MSTAR dataset, it's unclear whether this metric is optimal or most adaptable. Furthermore, they fail to refine the recognition performance to specific tasks, overlooking the benefits of such an approach. In general, they ignore the dependence of target features on the recognition task. Furthermore, they fail to consider learning reliable and robust metrics specifically suited for SAR imagery. Furthermore, most models consider feature maps only from a holistic perspective, neglecting to exploit discriminative information across their various dimensions. Summary of the Invention

[0013] The main objective of this invention is to provide an enhanced multi-transform feature extraction matching network model. By studying inter-channel and inter-spatial relationships, a new feature learning module with an attention mechanism is proposed to learn robust and distinguishable feature representations for small-sample SAR images. The feature learning module incorporates a residual network module and a multi-transform fusion mechanism. This not only reduces information loss and attrition during network transmission, but also integrates deep feature information from objects with different alignments, facilitating more accurate feature recognition. Furthermore, an adaptive matcher is used to measure similarity scores between samples, rather than a handcrafted fixed metric (e.g., Euclidean distance). It has been demonstrated that feature vectors extracted by deep neural networks with different categories can be effectively separated using Euclidean distance. Through the learnable process of the matcher, it can adaptively handle both intra-class variability and inter-class similarity. Before using the matcher to measure the recognition score, the query data features are reconstructed to enhance the dependency of the query features on the specific task.

[0014] The technical solution adopted by the present invention is: an enhanced multi-transformation feature extraction matching network model, comprising:

[0015] EMTDL-MatchNet reconstructs the query set feature module;

[0016] EMTDL-MatchNet feature learning module;

[0017] EMTDL-MatchNet adaptive matcher.

[0018] Furthermore, the EMTDL-MatchNet feature learning module includes:

[0019] The channel attention and spatial attention processing processes are expressed through formulas;

[0020] Channel attention mechanism formula:

[0021]

[0022] Spatial attention mechanism formula:

[0023]

[0024] Two different fusion schemes will be used to fuse the spatial attention mechanism and the channel attention mechanism;

[0025] At the shallow layer of the feature extraction module, attention fusion scheme 1 is used to help learn shallow features; at the deep layer of the feature extraction module, attention fusion scheme 2 is used to help learn deep semantic features;

[0026] Attention Fusion Scheme 1: Given an original feature map As input, after processing through the attention mechanism, we can obtain:

[0027]

[0028] By performing information enhancement, we can obtain:

[0029]

[0030] Attention fusion scheme 2: mainly divided into two steps;

[0031] Step 1: Convert the original feature map Perform two STN conversion processes, namely, through STN1 and STN2 respectively, to obtain and ;

[0032] Step 2: The two results obtained from the first part and ; Then pass through two attention combination modules respectively and get and ; The specific module is described by formula:

[0033] ;

[0034] Finally, the two vectors and Perform channel reduction processing on the last layer of the input to reduce the number of model parameters;

[0035] Add attention fusion scheme 1 between the third and fourth convolutional layers, and add attention fusion scheme 2 between the fifth and sixth convolutional layers;

[0036] The feature learning module is divided into three ResNet modules; the first segment ResNet1 includes two convolutional layers, the second segment ResNet2 includes two convolutional layers and the first attention fusion scheme, and the third segment ResNet3 includes two convolutional layers and the second attention fusion scheme;

[0037] For the feature learning module, the output channels of the first three convolutional modules are set to 16, 32, and 48 respectively, and the output channels of the last three convolutional blocks are the same, i.e., 64.

[0038] Furthermore, the EMTDL-MatchNet reconstructed query set feature module includes:

[0039] Assume that the support set output features processed by the feature learning module are averaged, and then the average support set attention features are fused with the query set attention features in the channel dimension;

[0040] The fused vector is input into the reconstructed query set feature module to halve the number of channels of the fused feature;

[0041] The module has three layers, each of which has a convolution block with a 3×3 convolution kernel, a batch normalization layer and a RELU nonlinear layer;

[0042] The output of the reconstructed query set feature module is used as the new query set attention feature. The new query set attention feature is concatenated with the support set attention feature in the channel dimension to form a query set-support set attention feature vector. This feature vector is input into the matching module to calculate the category score.

[0043] For the reconstructed query set feature module, the input channels of the three convolutional blocks are set to 128, 96, 96. The output channels are set to 96, 96 and 64 respectively.

[0044] Furthermore, the EMTDL-MatchNet matching module includes:

[0045] The query feature map and the support feature map are concatenated in the channel dimension, and the combined feature map is then fed into a matcher, which allows the matcher to self-adjust and select samples of the same category to obtain higher similarity scores. A nonlinear network structure is used to adaptively measure the similarity of SAR image samples, discarding hand-crafted fixed metrics that are not suitable for complex SAR image classification.

[0046] The matching module consists of two convolutional blocks and two fully connected layers. Both convolutional blocks contain a 3×3 convolutional layer, a batch normalization layer, a ReLU nonlinearity layer, and a 2×2 max pooling layer. The output of the first fully connected layer is calculated by ReLU, while the output of the last FC layer is calculated by Sigmoid. The last layer uses a large-margin softmax nonlinearity to increase the separability between samples of different categories, and batch normalization is used to accelerate network convergence.

[0047] For the matcher, the output channels of the first two convolutional blocks are set to 64 and 32; the input sizes of the two FC layers are 32*2*2 and 32, while the output sizes are 32 and 1.

[0048] Advantages of the present invention:

[0049] (1) This invention incorporates a residual network module into the feature learning module, which can reduce information loss and other issues during network information transmission. It achieves a certain degree of "identity transformation (y = x)". It also adds a quick connection branch to strike a balance between linear and nonlinear transformations, reducing information loss during transmission.

[0050] (2) In the feature learning module, a multi-transformation fusion mechanism is added to fuse the deep feature information of the object under different alignment methods, which is conducive to more accurate feature recognition. To a certain extent, it can eliminate background interference and fuse the category information of the object at different angles to fully explore the deep information of the data. In this case, it can also be regarded as two alignment methods for the input. The feature information can be expanded multiple times, which is more conducive to feature matching.

[0051] (3) A query set feature reconstruction module is added to the model. This allows the support set input and the query set input to jointly determine the characteristics of the query set data, thereby enhancing the reliance of query set feature extraction on task specificity. This helps enhance the model's generalization recognition ability and maintains good performance across different recognition tasks.

[0052] (4) The way to calculate the matching degree is not a fixed distance formula, but an adaptive neural network that can actively find the optimal measurement method through parameter updates.

[0053] (5) By studying the relationship between channels and spaces, a new feature learning module with attention mechanism is proposed to learn robust and distinguishable feature representations for small sample SAR images.

[0054] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0056] Figure 1 This is a comparison chart of the gamma processing of the present invention (original image on the left, image after gamma processing on the right);

[0057] Figure 2 The present invention Structural diagram;

[0058] Figure 3 The present invention ( )Structure diagram;

[0059] Figure 4 This is the specific structure diagram of ResNet of the present invention;

[0060] Figure 5 It is a STN structure diagram of the present invention;

[0061] Figure 6 This is the EMTDL-MatchNet structure diagram of the present invention;

[0062] Figure 7 This is the confusion matrix diagram of the classification results under the 5-way 5-shot condition of the present invention;

[0063] Figure 8 This is the 5-way 5-shot training loss change graph of the present invention;

[0064] Figure 9 This is the accuracy change graph of the 5-way 5-shot validation set of the present invention;

[0065] Figure 10 This is the 5-way 1-shot training loss change graph of the present invention;

[0066] Figure 11 This is a graph showing the accuracy change of the 5-way 1-shot validation set of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0068] The model EMTDL-MatchNet of the present invention is composed of three parts: a feature learning module, a query set feature reconstruction module and a matching module.

[0069] (1) Feature learning module

[0070] In the feature learning module, the network consists of six convolutional layers. Each convolutional block has a 3×3 convolutional layer, a batch normalization layer, and a RELU nonlinear layer.

[0071] A module combining spatial and channel-wise attention will be introduced into the feature learning module. Not all regions in an image contribute equally to the task; only those relevant to the task are of interest. Since convolution extracts informative features by blending cross-channel and spatial information, this module emphasizes meaningful features along the two primary dimensions: channel and spatial. The channel-wise and spatial-wise attention processes will be formalized.

[0072] Channel attention mechanism formula:

[0073]

[0074] Spatial attention mechanism formula:

[0075]

[0076] Two different fusion schemes will be used to fuse the spatial attention mechanism and the channel attention mechanism.

[0077] At the shallow level of the feature extraction module, attention fusion scheme 1 is used to help learn shallow features; at the deep level of the feature extraction module, attention fusion scheme 2 is used to help learn deep semantic features.

[0078] Attention fusion solution 1 (only includes ): Given an original feature map As input, after processing through the attention mechanism, we can obtain:

[0079]

[0080] By performing information enhancement, we can obtain:

[0081]

[0082] Attention Fusion Solution 2 (including and ): It is mainly divided into two steps. The first step: the original feature map Perform two STN conversion processes, namely, through STN1 and STN2 respectively, to obtain and Step 2: The two results obtained in the first part and Then, two attention combination modules are used. and get and ; The specific module is described by formula:

[0083] ;

[0084] Finally, the two vectors and The last layer of the input is connected in dimension and channel reduction is performed to reduce the number of parameters of the model.

[0085] Using an STN network for processing enhances the model's spatial invariance and improves classification performance. Multiple re-fusions are used to eliminate background interference to a certain extent, integrating object category information from different angles to fully exploit the data's deeper meaning. In this case, the input is also aligned in two ways, doubling the feature information, making image recognition easier.

[0086] The attention fusion scheme 1 is added between the third and fourth convolutional layers, and the attention fusion scheme 2 is added between the fifth and sixth convolutional layers.

[0087] Although, with the help of the attention mechanism, we can focus on the key points of learning. However, traditional convolutional networks or fully connected networks will more or less have problems such as information loss and loss when transmitting information. Therefore, the feature learning module is divided into three ResNet modules. The first section ResNet1 includes two convolutional layers, the second section ResNet2 includes two convolutional layers and the first attention fusion scheme, and the third section ResNet3 includes two convolutional layers and the second attention fusion scheme. The specific ResNet structure is as follows Figure 4 (as shown in the attached picture).

[0088] For the feature learning module, the output channels of the first three convolutional modules are set to 16, 32, and 48, respectively. The output channels of the last three convolutional blocks are the same, i.e., 64.

[0089] (2) Reconstructing the query set feature module

[0090] Assume that the support set output features processed by the feature learning module are averaged. The averaged support set attention features are then fused with the query set attention features along the channel dimension. The fused vector is input into the Reconstruct Query Set Feature module to halve the number of channels in the fused features. This module has three layers, each with a 3×3 convolution block, a batch normalization layer, and a RELU nonlinearity layer. The output of the Reconstruct Query Set Feature module serves as the new query set attention feature. The new query set attention feature is concatenated with the support set attention feature along the channel dimension to form a "query set-support set attention feature vector." This feature vector is input into the matching module to calculate the category score.

[0091] This allows the support set input and the query set input to jointly determine the characteristics of the query set data, thereby strengthening the reliance of query set feature extraction on task specificity. This helps enhance the model's generalized recognition capabilities and maintain good performance across different recognition tasks.

[0092] For the reconstructed query set feature module, the input channels of the three convolutional blocks are set to 128, 96, 96. The output channels are set to 96, 96 and 64 respectively.

[0093] (3) Matching module

[0094] Unlike the traditional approach of directly calculating the distance between support and query features, the query and support feature maps are concatenated along the channel dimension. The combined feature maps are then fed into a matcher, which self-adjusts to select samples of the same category to achieve higher similarity scores. In short, this approach uses a nonlinear network structure (i.e., an MLP) to adaptively measure the similarity of SAR image samples, eliminating handcrafted fixed metrics (such as Euclidean distance), which are unsuitable for complex SAR image classification.

[0095] The matching module consists of two convolutional blocks and two fully connected layers. Both convolutional blocks contain a 3×3 convolutional layer, a batch normalization layer, a Reluctant Unit (ReLU) nonlinearity layer, and a 2×2 max pooling layer. The output of the first fully connected (FC) layer is calculated using a Reluctant Unit (ReLU), while the output of the last FC layer is calculated using a Sigmoid. The final layer uses a large-margin softmax nonlinearity to increase the separability between samples of different categories, while batch normalization (BN) is used to accelerate network convergence.

[0096] For the matcher, the output channels of the first two convolutional blocks are set to 64 and 32. The input sizes of the two FC layers are 32*2*2 and 32, while the output sizes are 32 and 1.

[0097] Other details: The model was optimized using ADAM with an initial learning rate of 0.0001. In STN localization network 1, the bias of the last fully connected layer was set to the vector [−0.1, 0.1, −0.1, 0, −0.05, 0.05]. In STN localization network 2, the bias of the last fully connected layer was set to the vector [−0.75, 0, −0.5, 0, −0.75, 0.75]. In addition, the original image was gamma-transformed (gamma was set to 0.6). The dataset used was the MSTAR dataset. The mean squared error was used as the loss function.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An enhanced multi-transform feature extraction matching network model, characterized in that: include: EMTDL-MatchNet reconstructs the query set feature module; EMTDL-MatchNet feature learning module; EMTDL-MatchNet adaptive matcher; The EMTDL-MatchNet feature learning module includes: The channel attention and spatial attention processing processes are expressed through formulas; Channel attention mechanism formula: ； Spatial attention mechanism formula: ； Two different fusion schemes will be used to fuse the spatial attention mechanism and the channel attention mechanism; At the shallow layer of the feature extraction module, attention fusion scheme 1 is used to help learn shallow features; at the deep layer of the feature extraction module, attention fusion scheme 2 is used to help learn deep semantic features; Attention Fusion Scheme 1: Given an original feature map As input, after processing through the attention mechanism, we can obtain: ； By performing information enhancement, we can obtain: ； Attention fusion scheme 2: mainly divided into two steps; Step 1: Convert the original feature map Perform two STN conversion processes, namely, through STN1 and STN2 respectively, to obtain and ; Step 2: The two results obtained from the first part and ; Then pass through two attention combination modules respectively and get and ; The specific module is described by formula: ； Finally, the two vectors and Perform channel reduction processing on the last layer of the input to reduce the number of model parameters; Add attention fusion scheme 1 between the third and fourth convolutional layers, and add attention fusion scheme 2 between the fifth and sixth convolutional layers; The feature learning module is divided into three ResNet modules; the first segment ResNet1 includes two convolutional layers, the second segment ResNet2 includes two convolutional layers and the first attention fusion scheme, and the third segment ResNet3 includes two convolutional layers and the second attention fusion scheme; For the feature learning module, the output channels of the first three convolutional modules are set to 16, 32, and 48 respectively, and the output channels of the last three convolutional blocks are the same, i.e., 64; The EMTDL-MatchNet reconstructed query set feature module includes: Assume that the support set output features processed by the feature learning module are averaged, and then the average support set attention features are fused with the query set attention features in the channel dimension; The fused vector is input into the reconstructed query set feature module to halve the number of channels of the fused feature; The module has three layers, each of which has a convolution block with a 3×3 convolution kernel, a batch normalization layer and a RELU nonlinear layer; The output of the reconstructed query set feature module is used as the new query set attention feature. The new query set attention feature is concatenated with the support set attention feature in the channel dimension to form a query set-support set attention feature vector. This feature vector is input into the matching module to calculate the category score. For the reconstructed query set feature module, the input channels of the three convolutional blocks are set to 128, 96, and 96, and the output channels are set to 96, 96, and 64 respectively; The EMTDL-MatchNet matching module includes: The query feature map and the support feature map are concatenated in the channel dimension, and the combined feature map is then fed into a matcher, which allows the matcher to self-adjust and select samples of the same category to obtain higher similarity scores. A nonlinear network structure is used to adaptively measure the similarity of SAR image samples, discarding hand-crafted fixed metrics that are not suitable for complex SAR image classification. The matching module consists of two convolutional blocks and two fully connected layers. Both convolutional blocks contain a 3×3 convolutional layer, a batch normalization layer, a ReLU nonlinearity layer, and a 2×2 max pooling layer. The output of the first fully connected layer is calculated by ReLU, while the output of the last FC layer is calculated by Sigmoid. The last layer uses a large-margin softmax nonlinearity to increase the separability between samples of different categories, and batch normalization is used to accelerate network convergence. For the matcher, the output channels of the first two convolutional blocks are set to 64 and 32; the input sizes of the two FC layers are 32*2*2 and 32, while the output sizes are 32 and 1.

Images