A zero-shot SAR image target recognition method based on joint distribution adaptation

Abstract

The present invention relates to the field of image target recognition technology, and more specifically to a zero-shot SAR image target recognition method based on joint distribution adaptation. The training phase of the method does not require the participation of measured data for the task of interest. Instead, the method utilizes simulated SAR images and utilizes multi-kernel maximum mean difference to minimize the distance between the dual-domain mappings of the simulated and measured domains. This encourages similarity between the two sets of representations, learns domain-invariant features, and effectively mitigates domain differences. By capturing category information, the method automatically achieves fine-grained sub-domain adaptation, automatically and efficiently improving the robustness and recognition rate of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of image target recognition, and in particular to a zero-sample SAR image target recognition method based on joint distribution adaptation. Background Art

[0002] While the success of traditional optical imagery target recognition relies on large-scale labeled image datasets, the field of Automatic Target Recognition (ATR) from SAR images often faces sample constraints. Because acquiring field-measured SAR data is often complex and time-consuming, SAR image acquisition is strictly regulated, and the constant emergence of new tasks in SAR applications requires the frequent expansion of existing datasets, the field-measured SAR image training data required for new classes of interesting tasks is often unavailable in practice.

[0003] One feasible approach to this problem is to use simulated synthetic SAR data of the target to be identified, instead of measured data, to train the recognition model and identify new targets in real time. For example, in their 2019 paper, "EM simulation-aided zero-shot learning for SAR automatic target recognition," Qian Song et al. proposed preprocessing the simulated data to suppress non-essential factors before feeding it into a CNN for feature extraction. These methods often require manual preprocessing or similarity metrics.

[0004] Since there are distribution differences between the training data in the simulation domain and the test data in the measured domain, the test performance of the trained model will be greatly reduced. Therefore, it is difficult for a recognition model trained solely on simulation data to directly recognize measured data. One approach to alleviate domain differences is the unsupervised domain adaptation (UDA) method, which learns deep neural network transformations by mapping the two domains to a common feature space and optimizes the representation to minimize the measure of domain differences. For example, in the paper "Learning Transferable Features with Deep Adaptation Networks" published by Long et al. in 2015, a deep adaptive network was proposed to learn domain-invariant features through the Maximum Mean Discrepancy (MMD) indicator.

[0005] However, most domain adaptation methods use a certain number of target domain samples during the training phase. In real-world applications, not all classes have samples in the target domain. Research on zero-shot domain adaptation fills this gap. In SAR image target recognition based on zero-shot domain adaptation (ZSDA), when labeled data for the task-relevant classes of interest in the measured domain is unavailable, both measured and simulated SAR image data of known categories of task-irrelevant classes are available. This approach uses similarity metrics to learn domain-invariant feature representations, and combines them with task-relevant simulated data to train the task of interest, ultimately identifying new measured samples belonging to the task-relevant classes.

[0006] However, when adapting to the distribution of source and target domains, existing technologies often ignore category labels and only learn global feature changes, resulting in poor discrimination ability of the final trained model. How to achieve better recognition rates remains a huge challenge. Summary of the Invention

[0007] In view of the problems existing in the prior art, the object of the present invention is to provide a zero-shot SAR image target recognition method based on joint distribution adaptation.

[0008] In order to achieve the above objectives, the present invention adopts the following technical solutions to achieve them.

[0009] A zero-shot SAR image target recognition method based on joint distribution adaptation includes the following steps:

[0010] Step 1: Establish a training set, which includes a simulated SAR image sample set with labels and independent of the task, a measured SAR image sample set with labels and independent of the task, and a simulated SAR image sample set with labels and relevant to the task; establish a test set, which includes an unlabeled measured SAR image sample set relevant to the task; and preprocess all samples in the training set and the test set;

[0011] Step 2: Build a zero-shot joint adaptive network;

[0012] Step 3: Construct a network loss function based on joint distribution adaptation;

[0013] Step 4: training the zero-shot joint adaptive network to obtain a trained zero-shot joint adaptive network;

[0014] Step 5: Combine the third feature extractor and the first classifier in the trained zero-shot joint adaptive network to form a SAR image target recognition model. Input the task-related measured SAR images in the test set into the SAR image target recognition model to obtain the target category of the SAR image.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] (1) The training phase does not require the participation of measured data for the task of interest: When the measured SAR image training data required for the newly emerging task of interest is not available, the deep learning SAR ATR model is trained based on the corresponding simulated SAR images, and high-performance classification is achieved for the new measured samples belonging to the task of interest.

[0017] (2) Alleviating the domain differences between simulated and measured SAR images: By resorting to task-independent simulated and measured SAR images, the multi-kernel maximum mean difference is used to minimize the distance between the mappings of the two domains, encouraging the similarity between the two sets of representations of simulated and measured SAR images, learning domain-invariant features, and effectively alleviating the domain differences.

[0018] (3) Maintaining the category separability of the target mapping: Different from the existing methods that enhance the generalization performance of simulation data through manual design, the present invention utilizes the existing labeled task-independent data and automatically realizes fine-grained sub-domain adaptation by capturing category information, thereby automatically and efficiently improving the robustness and recognition rate of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Figure 1 Schematic diagram of the process of the present invention;

[0021] Figure 2 Schematic diagram of the structure of the zero-sample joint adaptive network of the method of the present invention;

[0022] Figure 3 This is a schematic diagram of the structure of the ResNet50 feature extraction module;

[0023] Figure 4 This is a schematic diagram of the structure of the convolution block in the ResNet50 feature extraction module;

[0024] Figure 5 Schematic diagram of the structure of the feature block in the ResNet50 feature extraction module. DETAILED DESCRIPTION

[0025] The embodiments of the present invention will be described in detail below with reference to examples. However, those skilled in the art will understand that the following examples are only used to illustrate the present invention and should not be construed as limiting the scope of the present invention.

[0026] refer to Figure 1 , a zero-shot SAR image target recognition method based on joint distribution adaptation, comprising the following steps:

[0027] Step 1: Establish a training set, which includes a simulated SAR image sample set with labels and independent of the task, a measured SAR image sample set with labels and independent of the task, and a simulated SAR image sample set with labels and relevant to the task; establish a test set, which includes an unlabeled measured SAR image sample set relevant to the task; and preprocess all samples in the training set and the test set;

[0028] Obtain a labeled dataset of simulated SAR images of all categories from the simulation domain where x i s and y i s ∈y={1,2,…,C} is the i-th labeled simulated SAR image sample and its label, and C is the number of categories;

[0029] Obtain a labeled dataset of observed SAR images of task-irrelevant (TI) categories from the observed domain where x i t，TI and y i t，TI ∈y TI are the i-th labeled measured SAR image sample and its label, y TI ∈y;

[0030] Obtain a new unlabeled dataset of task-relevant (TR) measured SAR images of unseen categories from the measured domain It's X t，TR The corresponding sample label to be predicted, where The complete label space is y=y TI ∪y TR ,and

[0031] During preprocessing, the original image is first cropped to a size of 64*64 with the target as the center. Then, the cropped image is modulo normalized to prevent the difference in echo amplitude from affecting the model.

[0032] Step 2: Build a zero-shot joint adaptive network;

[0033] Specifically, refer to Figure 2 The zero-shot joint adaptive network includes a first feature extractor, a second feature extractor, a third feature extractor, a first classifier, and a second classifier; the first feature extractor is sequentially connected to the first classifier, and the second feature extractor and the third feature extractor are respectively connected to the second classifier;

[0034] The input of the first feature extractor is a simulated SAR image related to the task, the input of the second feature extractor is a simulated SAR image unrelated to the task, and the input of the third feature extractor is a measured SAR image unrelated to the task; the outputs of the first classifier and the second classifier are both the categories of SAR image targets;

[0035] The first feature extractor, the second feature extractor, and the third feature extractor all include a ResNet50 feature extraction module and a bottleneck layer connected in sequence; the first feature extractor, the second feature extractor, and the third feature extractor share weights; the first classifier and the second classifier are both fully connected layers FC;

[0036] Among them, reference Figure 3 The ResNet50 feature extraction module includes the input layer INPUT, the first stage module stage1, the second stage module stage2, the third stage module stage3, the fourth stage module stage4, the fifth stage module stage5, the average pooling layer Avgpool, and the flattening layer Flatten.

[0037] The first stage module stage1 contains the sequentially connected convolution layer Conv, batch normalization layer BN, Relu activation layer, and maximum pooling layer Maxpool; the second stage module stage2 contains the sequentially connected convolution block Conv Block and two feature blocks ID Block; the third stage module stage3 contains the sequentially connected convolution block Conv Block and three feature blocks IDBlock; the fourth stage module stage4 contains the sequentially connected convolution block Conv Block and five feature blocks ID Block; the fifth stage module stage5 contains the sequentially connected convolution block Conv Block and two feature blocks ID Block;

[0038] refer to Figure 4 In the convolution block Conv Block, the input features of the convolution block Conv Block pass through the convolution layer Conv, batch normalization layer BN, Relu activation layer, convolution layer Conv, batch normalization layer BN, Relu activation layer, convolution layer Conv, batch normalization layer BN, and the input features of the convolution block Conv Block pass through the convolution layer Conv, batch normalization layer BN, and then pass through the Relu activation layer before output;

[0039] refer to Figure 5In the feature block ID Block, the input features of the feature block ID Block are sequentially passed through the convolution layer Conv, batch normalization layer BN, Relu activation layer, convolution layer Conv, batch normalization layer BN, Relu activation layer, convolution layer Conv, batch normalization layer BN, and then added to the input features of the feature block ID Block, and then output after passing through the Relu activation layer;

[0040] refer to Figure 2 , the bottleneck layer contains a 256-dimensional fully connected layer FC, a batch normalization layer BN, a Relu activation layer and a random inactivation layer Dropout, which are connected in sequence.

[0041] Step 3: Construct a network loss function based on joint distribution adaptation;

[0042] The network loss function is constructed as follows:

[0043]

[0044] in, L d =∑ q∈Q MKMMD(S TIq (X s,TI ),T q (X t,TI ));

[0045] Where λ is the network adaptation weight parameter;

[0046] is the first classifier C TR For the simulated SAR image sample set D with labels and relevant to the task s,TR Samples in Classification loss;

[0047] is the second classifier C TI For the simulated SAR image sample set D with labels and irrelevant to the task s,TI Samples in and a set of labeled and task-independent measured SAR image samples D t,TI Samples in Classification loss;

[0048] J(·) is the classification loss function, specifically the cross entropy loss function; S TR (·) represents the first feature extractor; S TI (·) represents the second feature extractor; T(·) represents the third feature extractor;

[0049] L dIt is a domain adaptation loss based on the MK-MMD distance between task-independent simulated SAR images and task-independent measured SAR images; q (X) is the qth hidden layer feature of sample X in the network, and Q is the set of input feature layers of domain adaptation loss.

[0050] refer to Figure 2 , the second feature extractor S TI The output features of and the output features of the third feature extractor T are also used as inputs for the calculation of the domain adaptation loss based on MK-MMD.

[0051] MMD distance is the reproducing kernel Hilbert space The square of the kernel embedding distance of the data distribution in the Reproducing Kernel Hilbert Space (RKHS) is calculated as follows:

[0052]

[0053] In the formula, it is assumed that the simulated SAR image sample set D is irrelevant to the task s,TI The SAR images in the dataset and the measured SAR image sample set D that is irrelevant to the task t,TI The SAR images in the distribution p(X s ) and q(X t ); k is the kernel function defined on RKHS, specifically the Gaussian function k(x, x′)=exp(-||xx′|| 2 / (2σ 2 )), due to the kernel technique It can be viewed as an infinite-dimensional feature map φ(x); is the set of mapping functions in this RKHS, further constrained to be the unit ball of the reproducible kernel Hilbert space.

[0054] MK-MMD is a multi-core variant of MMD, and its RKHS corresponds to the core For m semi-positive definite kernels {k u} weighted combination, kernel As shown in the following formula:

[0055]

[0056] Where m is the number of semi-positive definite kernels, β u The multi-core MK-MMD metric is used to improve the feature representation ability and transferability of the model.

[0057] Step 4: training the zero-shot joint adaptive network to obtain a trained zero-shot joint adaptive network;

[0058] Specifically, using the network loss function L ZSJAN End-to-end joint training of the first feature extractor S TR , the second feature extractor S TI , the third feature extractor T, the first classifier C TR and the second classifier C TI , update the network parameters according to the back-propagation iteration, minimize the target loss, save the final network parameters, and obtain the trained zero-shot joint adaptive network.

[0059] During the training process, in order to obtain a stronger feature network, a pre-trained model obtained on the optical image set ImageNet is loaded as the model initialization. All layers are fine-tuned through backpropagation, and the classification layer and bottleneck layer are trained from scratch. The weight of the MK-MMD loss function is set to gradually increase from 0 to 1. When the training progress is p%, And γ=10.

[0060] Step 5: Combine the third feature extractor and the first classifier in the trained zero-shot joint adaptive network to form a SAR image target recognition model. Input the task-related measured SAR images in the test set into the SAR image target recognition model to obtain the target category of the SAR image.

[0061] After the SAR image target recognition model is tested, any measured SAR image target related to the task can be identified.

[0062] Simulation experiment

[0063] The measured and simulated datasets used are the MSTAR three-category dataset and a simulated SAR image dataset. The MSTAR three-category dataset includes three types of vehicles: the BMP2, BTR70, and T72. The vehicles are centered in the image, with a stationary background and no interfering information. The dataset has an elevation angle of 17° and is densely sampled from 0° to 359° in azimuth. Depending on the object, an image is captured every 1° to 2°. The polarization scheme is HH, the resolution is 0.3m × 0.3m, and the image size is 128 × 128. The simulated SAR image dataset was generated using electromagnetic simulation techniques using 3D models of the targets created by researchers. The dataset includes three types: the BMP2, BTR70, and T72R. The dataset is generated at an elevation angle of 17° and an azimuth angle of 0° to 359°. The polarization scheme is HH, the resolution is 0.2m × 0.2m, and the image size is 128 × 128. Table 1 provides a description of the data types and sample sizes for the real and simulated datasets. There are 233, 233, and 232 real data samples of the three categories respectively, and there are 360 simulated data samples of the three categories respectively.

[0064] Table 1

[0065] category BMP2 BTR70 T72 Number of measured images 233 233 232 Number of simulated images 360 360 360

[0066] Because three categories of samples are insufficient to distinguish between the seen category TI (irrelevant to the task of interest) and the unseen category TR (relevant to the task of interest) at the category level, and considering the azimuth sensitivity of SAR images, which leads to significant differences between samples from the same target at different azimuths, this experiment divides the three categories of samples at azimuths of 0° to 359° into 0° to 179° and 180° to 359°, respectively, as task-irrelevant and task-relevant targets, thus expanding the three categories to six. Table 2 shows the division method of the training and test data.

[0067] Table 2

[0068]

[0069] Simulation 1: Using a traditional deep learning method with a ResNet50 network structure to identify the target category in the test data SAR image;

[0070] Simulation 2: using the SAR image target recognition method of the present invention to identify the category of the SAR image target of the test data;

[0071] The implementation details of the simulation experiment are as follows: all layers of the network are trained using stochastic gradient descent with a momentum of 0.9 and a learning rate of 0.001, and the number of batch samples is 18.

[0072] Simulation experiment results

[0073] The recognition rate of the test data in simulation 1 is 36.96%, which shows that the domain differences between different data distributions cannot be eliminated by extracting deep features alone. The traditional method has poor performance in zero-shot SAR image target recognition.

[0074] The recognition rate of the test data in simulation 2 is 81.66%, which is 44.7% higher than that in simulation 1, proving the effectiveness of the proposed method in zero-shot SAR image target recognition.

[0075] In summary, the zero-shot SAR image target recognition method based on joint distribution adaptation of the present invention first uses SAR images generated by electromagnetic simulation technology of visual representation space to replace the semantic representation space used in traditional zero-shot visual recognition as side information, effectively alleviating the problem of missing measured data for the task of interest. Secondly, due to the difference in distribution between the simulation domain and the measured domain, a zero-shot joint adaptive network based on a convolutional neural network is used to innovatively extract the features of the simulation and measured data of task-independent targets, and adapt the joint distribution of specific feature layers in different fields based on the multi-kernel maximum mean difference. Not only can the model learn a robust and transferable representation that can be generalized to the task of interest, it avoids overfitting the unique features of the simulation data and reduces the test error that increases with the increase of the domain error. In addition, by locally matching the classification surfaces of the data labels in the two domains, it can also retain the migration relationship between the categories of each data domain, automatically capture the fine-grained discriminant structure between categories, realize the conditional probability distribution adaptation of local matching, maintain the category separability of the target mapping, and effectively improve the recognition accuracy and robustness of the model.

[0076] Although this specification has provided a detailed description of the present invention using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made based on the present invention. Therefore, such modifications and improvements, which do not depart from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A zero-shot SAR image target recognition method based on joint distribution adaptation, characterized in that: The following steps are involved: Step 1: Establish a training set, which includes a simulated SAR image sample set with labels and independent of the task, a measured SAR image sample set with labels and independent of the task, and a simulated SAR image sample set with labels and relevant to the task; establish a test set, which includes an unlabeled measured SAR image sample set relevant to the task; and preprocess all samples in the training set and the test set; Step 2: construct a zero-shot joint adaptive network. Specifically, the zero-shot joint adaptive network includes a first feature extractor, a second feature extractor, a third feature extractor, a first classifier, and a second classifier. The first feature extractor is sequentially connected to the first classifier, and the second feature extractor and the third feature extractor are respectively connected to the second classifier. The input of the first feature extractor is a simulated SAR image related to the task, the input of the second feature extractor is a simulated SAR image unrelated to the task, and the input of the third feature extractor is a measured SAR image unrelated to the task; the outputs of the first classifier and the second classifier are both the categories of SAR image targets; The first feature extractor, the second feature extractor, and the third feature extractor all include a ResNet50 feature extraction module and a bottleneck layer connected in sequence; the first feature extractor, the second feature extractor, and the third feature extractor share weights; the first classifier and the second classifier are both fully connected layers FC; Among them, the ResNet50 feature extraction module includes the input layer INPUT, the first stage module stage1, the second stage module stage2, the third stage module stage3, the fourth stage module stage4, the fifth stage module stage5, the average pooling layer Avgpool, and the flattening layer Flatten. The first stage module stage1 contains the sequentially connected convolution layer Conv, batch normalization layer BN, Relu activation layer, and maximum pooling layer Maxpool; the second stage module stage2 contains the sequentially connected convolution block Conv Block and two feature blocks IDBlock; the third stage module stage3 contains the sequentially connected convolution block Conv Block and three feature blocks ID Block; the fourth stage module stage4 contains the sequentially connected convolution block Conv Block and five feature blocks ID Block; the fifth stage module stage5 contains the sequentially connected convolution block Conv Block and two feature blocks ID Block; The bottleneck layer contains a 256-dimensional fully connected layer FC, a batch normalization layer BN, a Relu activation layer, and a random inactivation layer Dropout. Step 3: Construct a network loss function based on joint distribution adaptation; Step 4: training the zero-shot joint adaptive network to obtain a trained zero-shot joint adaptive network; Step 5: Combine the third feature extractor and the first classifier in the trained zero-shot joint adaptive network to form a SAR image target recognition model. Input the task-related measured SAR images in the test set into the SAR image target recognition model to obtain the target category of the SAR image.

2. The zero-shot SAR image target recognition method based on joint distribution adaptation according to claim 1, characterized in that: The network loss function constructed in step 3 is as follows: in, Where λ is the network adaptation weight parameter; is the first classifier C TR For the simulated SAR image sample set D with labels and relevant to the task s,TR Samples in Classification loss; is the second classifier C TI For the simulated SAR image sample set D with labels and irrelevant to the task s,TI Samples in and a set of labeled and task-independent measured SAR image samples D t,TI Samples in Classification loss; J(·) is the classification loss function, specifically the cross entropy loss function; S TR (·) represents the first feature extractor; S TI (·) represents the second feature extractor; T(·) represents the third feature extractor; L d It is a domain adaptation loss based on the MK-MMD distance between task-independent simulated SAR images and task-independent measured SAR images; q (X) is the qth hidden layer feature of sample X in the network, and Q is the set of input feature layers of domain adaptation loss.

3. The zero-shot SAR image target recognition method based on joint distribution adaptation according to claim 1, characterized in that: The first feature extractor, the second feature extractor, the third feature extractor, the first classifier and the second classifier are jointly trained end-to-end using the network loss function. The network parameters are iteratively updated according to back-propagation to minimize the target loss. The final network parameters are saved to obtain a trained zero-shot joint adaptive network.

Images