Robust classification method for small sample sequence ISAR images based on semi-supervised transfer learning

Abstract

The present invention discloses a robust classification method for small-sample sequence ISAR images based on semi-supervised transfer learning, comprising the following steps: obtaining ISAR images to be classified; inputting the image sequence of the ISAR images to be classified into a trained sequence homography network to obtain corresponding embedded feature vectors; preprocessing the embedded feature vectors using a power transformation, and then iteratively estimating the classification labels and classification centers using an optimal transfer mapping to obtain a classification result. The present invention proposes a sequence homography network. A backbone network is used to extract sequence features, and a homography parameter generator is used to generate homography transformation parameters. Homography transformation adjustment is then performed on the sequence ISAR deformable images, thereby achieving temporal feature extraction and deformation-robust ISAR image classification.

Description

Technical Field

[0001] The present invention belongs to the field of radar remote sensing technology, and in particular relates to a robust classification method for small sample sequence ISAR images based on semi-supervised transfer learning. Background Art

[0002] Inverse Synthetic Aperture Radar (ISAR) is an active imaging system with all-day, all-weather, long-range, and high-resolution capabilities. It is now widely used in fields such as space and aerial target surveillance. However, in practice, ISAR targets are non-cooperative, and their motion and posture are unknown, resulting in unknown deformation and distortion in the image. Furthermore, limited observation conditions often result in only a very small number (e.g., a few) of certain ISAR images, resulting in small sample sizes. In such cases, traditional large-scale deep networks suffer from severe overfitting due to this lack of samples, resulting in reduced recognition accuracy and poor generalization performance. This poses significant challenges for robust feature extraction and accurate classification of ISAR targets.

[0003] Few-shot learning is a machine learning paradigm that learns effective features from a very limited number of labeled examples. Currently, a series of few-shot learning methods, such as data augmentation, transfer learning, and meta-learning, have been proposed in the field of computer vision. These methods have effectively addressed issues such as sample imbalance and labeling difficulties in few-shot classification. However, existing ISAR few-shot target classification methods suffer from the following problems: 1) They only extract spatial features from a single image and lack targeted processing of unknown deformations, resulting in insufficient robustness in target classification; 2) They train the embedding network using only support set samples. The resulting embedded features are easily confused when classifying the query set, resulting in low accuracy in few-shot classification and insufficient model generalization. Therefore, there is an urgent need to develop robust deformation classification methods for sequential ISAR images in few-shot scenarios.

[0004] The paper "A Gaussian Prototype-Based Target Recognition Method for Small-Sample ISAR Images" discloses a Gaussian prototype-based target recognition method for small-sample ISAR images. This method consists of two main steps: first, a Gaussian prototype is computed for each target class in the support set. Then, the class label of each ISAR image sample is obtained by calculating the distance between the embedding vector corresponding to the Gaussian prototype and the query set ISAR image sample. However, this method has the following drawbacks: It only extracts spatial features from a single image and lacks targeted processing for unknown deformations, resulting in insufficient robustness in target classification. Furthermore, the embedding network is trained using only support set samples, so the resulting embedded features are easily confused when classifying the query set, resulting in low small-sample classification accuracy and insufficient model generalization. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention designs a sequence homography network (SHN) and a small-sample ISAR image classification architecture based on semi-supervised learning and optimal transfer mapping to achieve feature extraction of deformation-robust ISAR sequences, enhance the generalization of the network and improve the classification accuracy.

[0006] A robust classification method for small sample sequence ISAR images based on semi-supervised transfer learning, the classification method comprising:

[0007] Obtain the ISAR image to be classified;

[0008] Inputting the image sequence of the ISAR images to be classified into a trained sequence homography network to obtain corresponding embedded feature vectors, wherein the sequence homography network includes a cascaded backbone network, a homography parameter generator, and an image sampler. The backbone network is used to extract features from the time dimension of the image sequence and the length and height dimensions of a single frame image to obtain an output feature map, the homography parameter generator is used to query the homography parameter sequence from the output feature map to obtain homography transformation parameters, the elements of the homography transformation parameters respectively corresponding to the deformation adjustment parameters of each frame image in the sequence sample, and the image sampler is used to perform homography transformation on each frame image in the image sequence using the homography transformation parameters corresponding to each frame image to obtain a deformation-adjusted image sequence;

[0009] The embedded feature vector is preprocessed by using power transformation, and then the classification label and classification center are iteratively estimated through optimal transmission mapping to obtain the classification result.

[0010] In one embodiment of the present invention, the backbone network includes a cascaded 5-layer 3D-CNN structure, each of the 3D-CNN structures is connected to a 3D batch normalization and a ReLU activation function, and the 1st to 4th layer 3D-CNN structures also use 2×2 maximum pooling for feature compression, wherein the 1st layer 3D-CNN structure is an expanded 3D convolution with 8 channels, and the 2nd to 5th layer 3D-CNN structures are all expanded 3D convolutions with 16 channels.

[0011] The calculation method of the dilated 3D convolution is:

[0012]

[0013] Where x(i, j, k) is the (i, j, k)th element in the input image, K is a 3D convolution kernel of size U×V×W, d1, d2, and d3 are the dilation coefficients in the three dimensions respectively. Before convolution, a zero of length (d1-1)(U-1) is added to the temporal head to make the temporal dimension of the output feature map unchanged and causal.

[0014] In one embodiment of the present invention, the homography parameter generator includes an encoder and a decoder;

[0015] The input of the encoder is the true value V h , key value K h and query value Q h , the true value V h , key value K h and query value Q h They are:

[0016]

[0017]

[0018]

[0019] Among them, z1 is the output feature map of the backbone network, is a learnable positional encoding, d is the dimension of the embedding vector, F is the length of the sequence image, is the dimension transformation matrix, d h is the dimension of each head, h is the index of the attention head, and LN(·) is layer normalization;

[0020] The output of the encoder is:

[0021] z2=FFN(z′1)=max(0,z′1W1+b1)W2+b2

[0022] z′1=Concat(a 1 ,...,a H )W m +z1

[0023]

[0024] Where z2 is the output of the encoder, FFN is a feedforward network, W1 is the weight of the first fully connected layer of FFN, b1 is the bias of the first fully connected layer of FFN, W2 is the weight of the second fully connected layer of FFN, b2 is the bias of the second fully connected layer of FFN, H is the number of attention heads, W m is the aggregation weight, T represents the transpose operation;

[0025] The truth value of the decoder Key-value Query Value They are:

[0026]

[0027]

[0028]

[0029] in, is the dimension transformation matrix.

[0030] In one embodiment of the present invention, the output image of the image sampler is The pixel value at (x′, y′) is:

[0031]

[0032] Among them, x f (m, n) is the input image x f The pixel value at position (m, n).

[0033] In one embodiment of the present invention, the training method of the sequence homography network includes:

[0034] S1. Acquire ISAR images with multiple target categories and category labels under different imaging conditions, and divide the ISAR images into a training sample set and a test sample set according to the categories, wherein the sample categories of the training sample set and the test sample set do not overlap;

[0035] S2. With the target as the center, cropping the ISAR images in the training sample set and the test sample set, and processing the cropped ISAR images by a sliding window method to generate sequence images, where all sequence images corresponding to the training sample set constitute a training set, and all sequence images corresponding to the test sample set constitute a test set;

[0036] S3, rotating the ISAR image in step S1 by 0°, 90°, 180°, and 270° respectively, and generating rotation labels for the ISAR image samples according to the rotation angles to obtain an unsupervised sample set;

[0037] S4. Based on a semi-supervised learning method, the sequence homography network is trained using the training set and the unsupervised sample set to obtain a trained sequence homography network.

[0038] In one embodiment of the present invention, step S4 includes:

[0039] S4.1. Input the sequence sample x in the training set into the sequence homography network. After obtaining the output features, use the cosine classifier to obtain the predicted category label, and use the cross entropy loss to calculate the supervision loss.

[0040] S4.2, the sample x in the unsupervised sample set r Input to the sequence homography network to obtain the output feature f(xr ), use a 4-class linear classifier to classify the output feature f(x r ) is used for classification to obtain the predicted rotation angle, and the self-supervisory loss is calculated based on the cross entropy loss;

[0041] S4.3. Based on the loss function of semi-supervised learning, updating the network parameters of the sequence homography network until a preliminarily trained sequence homography network is obtained;

[0042] S4.4. Based on the manifold mixing loss function, the preliminarily trained sequence homography network is trained using manifold mixing to obtain a trained sequence homography network.

[0043] In one embodiment of the present invention, the loss function of the semi-supervised learning is:

[0044] L s =L rot +L c

[0045] Among them, L s is the loss function for semi-supervised learning, L c is the cross entropy loss function for supervised learning, L rot is the loss function for self-supervised learning;

[0046] The loss function of the self-supervised learning is:

[0047]

[0048] Where L(·) is the cross entropy loss function y k is the rotated label of the sample, is the predicted rotated label, K=4 is the total number of categories, D b represents the training sample set, R = {0°, 90°, 180°, 270°} is the rotation angle set, W r It is a 4-class linear classifier;

[0049] The manifold mixing loss function is:

[0050] L m =L[f L-l (Mix λ (f l (x1), f l (x2))), Mix λ (y1, y2)]

[0051] Mix λ (a, b) = λ·a + (1-λ)·b

[0052] Among them, fl (x1) and f l (x2) are the feature maps of the sample image x1 and the sample image x2 in the lth layer of the sequence homography network, respectively. The mixing coefficient λ is sampled from the Beta distribution β(α, α), where α is a hyperparameter, L represents the number of layers of the network, and f L-l (·) is the network layer after the lth layer, L(·) is the cross entropy loss function, y1 and y2 are the category labels of sample image x1 and sample image x2.

[0053] In one embodiment of the present invention, after step S4, the method further includes:

[0054] S5. Randomly select w categories from the test set, and each type of labeled sample image constitutes the support set D s , the unlabeled samples constitute the query set D q , where s and q represent the number of labeled and unlabeled samples in each class, respectively;

[0055] S6. Merge the support set and query set D s ∪D q , to obtain w(s+q) sequence samples;

[0056] S7, inputting w(s+q) sequence samples into the trained sequence homography network to obtain corresponding sample embedding feature vectors;

[0057] S8. Use power transformation to preprocess the sample embedding feature vector, and iteratively estimate the classification label and classification center through optimal transmission mapping to obtain the classification result of the test set.

[0058] In one embodiment of the present invention, step S8 includes:

[0059] S8.1. Preprocess the sample embedding feature vector using a power transform to obtain a preprocessed sample embedding feature vector;

[0060] S8.2. Fix the category center and embed the preprocessed samples in the query set into the feature vector As data, by maximizing the sample embedding feature vector The posterior distribution of the classification label is used to optimize the distribution, and the optimal class mapping matrix M is obtained by OTM calculation. * ;

[0061] S8.3, based on the optimal class mapping matrix M * , re-estimate the classification center using the mapping label of the sample embedding feature vector and the true label of the support set, and increase the learning rate in the classification center to obtain the re-estimated classification center;

[0062] S8.4. The re-estimated classification center is used as the initial value for each iteration. After a preset number of iterations of steps S8.2 and S8.3, the sample labels of the query set are obtained from the optimal class mapping matrix M* of the last iteration, thereby obtaining the classification result.

[0063] In one embodiment of the present invention, the sample embedding feature vector after power transformation is:

[0064]

[0065] Where z is the sample embedding feature vector, ε = 10 -6 , β is a hyperparameter, ||·|| is the two-norm normalization term;

[0066] The optimal class mapping matrix M * for:

[0067]

[0068] Among them, L is the Euclidean distance between the unlabeled sample features and the category center in the query set, is the preprocessed sample embedding feature vector to w categories of class mapping matrix, M ij is the element in the i-th row and j-th column of M, μ is the regularization coefficient, is a set of matrices whose elements are all positive, and each matrix satisfies the sum of rows is a and the sum of columns is b. cj is the classification center, H(M)=-∑ ij M ij logM ij .

[0069] Beneficial effects of the present invention:

[0070] First, the present invention proposes a sequential homography network. It uses a backbone network to extract sequence features and generates homography transformation parameters through a homography parameter generator. This is then used to perform homography transformation adjustments on sequential ISAR deformable images, thereby achieving temporal feature extraction and deformation-robust ISAR image classification.

[0071] Second, the present invention designs a small-sample ISAR image classification architecture based on semi-supervised learning and optimal transfer mapping. By constructing an unsupervised sample set and combining it with the training set for semi-supervised training, the network's generalization performance is enhanced. Feature preprocessing normalizes the embedded features, allowing the transfer learning process to focus more on inter-class differential features. The optimal transfer mapping is used to iteratively estimate the labels of the test set samples, effectively improving the average classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 11 is a flow chart of a robust classification method for small sample sequence ISAR images based on semi-supervised transfer learning provided by an embodiment of the present invention;

[0073] Figure 2 4 is a structural diagram of a sequence homography network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0075] Example 1

[0076] See Figure 1 and Figure 2 , Figure 1 1 is a flow chart of a robust classification method for small sample sequence ISAR images based on semi-supervised transfer learning provided by an embodiment of the present invention. Figure 2 is a structural diagram of a sequence homography network provided by an embodiment of the present invention. The present invention provides a robust classification method for small sample sequence ISAR images based on semi-supervised transfer learning, which classification method includes:

[0077] Step 1: Obtain the ISAR image to be classified;

[0078] Step 2: Input the image sequence constructed from the ISAR image to be classified into a trained sequence homography network to obtain the corresponding embedded feature vector, wherein the sequence homography network includes a cascaded backbone network, a homography parameter generator, and an image sampler. The backbone network is used to extract features from the time dimension of the image sequence and the length and height dimensions of a single frame image to obtain an output feature map. The homography parameter generator is used to query the homography parameter sequence from the output feature map to obtain homography transformation parameters, the elements of which correspond to the deformation adjustment parameters of each frame image in the sequence sample. The image sampler is used to perform homography transformation on each frame image in the image sequence using the homography transformation parameters corresponding to each frame image to obtain a deformation-adjusted image sequence.

[0079] Step 3: Use power transformation to preprocess the embedded feature vector, and then iteratively estimate the classification label and classification center through optimal transmission mapping to obtain the classification result.

[0080] In a specific embodiment, the constructed sequential homography network is formed by cascading a backbone network, a homography parameter generator, and an image sampler, and the construction steps are as follows:

[0081] (1) Establish backbone network module:

[0082] The input of the backbone network is the sequence image x, and the output is the output feature map Where d is the dimension of the embedding vector, and F is the length of the image sequence. To fully utilize the sequential imaging characteristics of ISAR, the backbone network adopts a five-layer 3D-CNN structure, which can simultaneously extract features from the temporal dimension of the image sequence and the long and high dimensions of a single frame. Each 3D convolution is followed by 3D batch normalization and ReLU activation functions. Layers 1-4 also use 2×2 max pooling for feature compression. The detailed structure of the backbone network is shown in Table 1. For example, "Dilated Conv.3D 8@2×3×3" indicates dilated 3D convolution with 8 channels and a kernel size of 2×3×3. D = [1, 4, 4] indicates that the dilation coefficients in the three dimensions are set to 1, 4, and 4, respectively. Other structures are similar.

[0083] Table 1 Detailed structure of the backbone network

[0084]

[0085] 3D convolution is an extension of 2D convolution in the time dimension, meaning both the convolution kernel and the feature map have three computational dimensions: time, length, and height. Because transfer learning models tend to overfit on base class data, resulting in decreased classification performance for new class data, the backbone network uses dilated convolution to increase the temporal and spatial receptive field while reducing the number of parameters. The calculation for dilated 3D convolution is as follows:

[0086]

[0087] Where x(i, j, k) is the (i, j, k)th element in the input image, K is a 3D convolution kernel of size U×V×W, d1, d2, and d3 are the dilation coefficients in the three dimensions respectively. Before convolution, a zero of length (d1-1)(U-1) is added to the temporal head to make the temporal dimension of the output feature map unchanged and causal.

[0088] The 3D pooling operation is:

[0089] P 3D (i,j,k)=max{x(i·s1+r,j·s2+p,k·s3+q)|

[0090] r=0, 1, ..., R-1; p=0, 1, ..., P-1; q=0, 1, ..., Q-1}

[0091] Among them, s1, s2, and s3 are the pooling steps in the time dimension, image length, and height respectively, and R, P, and Q are the pooling window lengths in the corresponding dimensions.

[0092] (2) Establish the homography parameter generator module:

[0093] SHN (i.e. Sequential Homography Network) uses homography transformation for deformation adjustment. Among them, the homography parameter generator uses Transformer to query the homography parameter sequence from the output features of the backbone network. The sequence elements correspond to the deformation adjustment parameters of each frame image in the sequence sample. The input of the Transformer encoder is the true value V h , key value K h and query value Q h , are obtained by linear mapping the output z1 of the backbone network, and the true value V h , key value K h and query value Q h They are:

[0094]

[0095]

[0096]

[0097] Among them, z1 is the output feature map of the backbone network, is a learnable positional encoding, is the dimension transformation matrix, d h is the dimension of each head, h is the index of the attention head, and LN(·) is the layer normalization, which is to subtract the mean u of the samples in all layers. L and divided by the variance Where α and β are learnable parameters, and ε is a very small number to prevent the calculation from dividing by 0. The calculation formula for layer normalization is:

[0098]

[0099] Encoder feature extraction is achieved by scaled dot product attention, i.e.:

[0100]

[0101] Where T represents the transpose operation.

[0102] Connecting all attention heads to form a multi-head attention can increase the feature diversity during key-value query, namely:

[0103] z′1=Concat(a 1 ,...,a H )W m +z1

[0104] in, is the aggregation weight, H is the number of attention heads, and z1 is added as a residual connection to stabilize the training error. After that, feature mapping is performed through the feedforward network FFN, which consists of a fully connected layer and a ReLU activation function:

[0105] z2=FFN(z′1)=max(0,z′1W1+b1)W2+b2

[0106] Among them, W1 is the weight of the first fully connected layer of FFN, b1 is the bias of the first fully connected layer of FFN, W2 is the weight of the second fully connected layer of FFN, and b2 is the bias of the second fully connected layer of FFN.

[0107] The output z2 of the encoder continues to serve as the input of the decoder, corresponding to the true value of the decoder Key-value Query Value They are:

[0108]

[0109]

[0110]

[0111] in, is the dimension transformation matrix, and the query value uses a learnable homography query The decoder has similar scaled dot product attention and FFN calculations as the encoder, and maps the output features to z3 = {θ f，i} f＝1，...，F，i＝1，...，8 , that is, each image corresponds to 8 parameters of homography transformation.

[0112] (3) Establish the image sampler module:

[0113] The two-dimensional ISAR image of a target is the projection of its three-dimensional scattering center onto the two-dimensional imaging plane. Due to changes in imaging parameters and observation angles, ISAR images correspond to different imaging projection planes (IPPs), resulting in large morphological differences in the ISAR images of the same target. Unlike the image affine transformation with 6 parameters, the homography transformation adds 2 parameters for controlling the perspective relationship, so it has a stronger spatial projection representation capability. Specifically, the homography transformation can effectively adjust the image geometric deformation and projection deformation caused by different IPPs, thereby facilitating robust feature extraction in small sample classification problems. Specifically, the image sampler uses the transformation parameters corresponding to each frame of the image to transform each frame of the input image sequence x. f Perform homography transformation to achieve an effective description of the projective transformation relationship between two planes in space.

[0114] According to the homogeneity of the homography transformation, the parameters z3 output by the homography transformation generator are arranged into a homography transformation parameter sequence Θ = [Θ1, Θ2, ...Θ F ],in:

[0115]

[0116] Among them, θ f，1 to θ f，8 There are 8 parameters corresponding to the homography transformation for each image.

[0117] The homography transform defines the input image x f point (x, y) on the output image The mapping of a point (x′, y′) on [x′, y′, 1] T =Θ f [x, y, 1] T . In the input image x f When sampling according to the coordinates (x', y'), the sampling points often offset the pixel position, so bilinear interpolation is used for pixel sampling. The output of the sequence sampler The pixel value at (x′, y′) is calculated as follows:

[0118]

[0119] Among them, x f (m, n) is the input image x f The pixel value at position (m, n). Similar processing is performed on each image to obtain the output image sequence of the sequence sampler.

[0120] After constructing the sequential homography network, the present invention also provides a training method for the network. This method utilizes the constructed training set and the constructed unsupervised sample set to train the network using a semi-supervised learning approach. This approach introduces self-supervised learning, using both self-supervised and supervised classification losses to guide network training. Furthermore, a manifold hybrid algorithm is employed to further improve model generalization performance.

[0121] In a specific embodiment, the training method of the sequence homography network includes:

[0122] S1. Under different imaging conditions, ISAR images with multiple target categories and category labels are acquired, and the ISAR images are divided into a training sample set and a test sample set according to their categories. The sample categories of the training sample set and the test sample set do not overlap.

[0123] Specifically, ISAR images with multiple classes of targets and class labels are acquired under imaging conditions such as different elevation angles, azimuth angles, bandwidths and accumulation angles, and the above ISAR images are divided into training sample sets (base class) and test sample sets (new class) according to their categories.

[0124] S2. With the target as the center, the ISAR images in the training sample set and the test sample set are cropped, and the cropped ISAR images are processed by the sliding window method to generate sequence images. All sequence images corresponding to the training sample set constitute the training set, and all sequence images corresponding to the test sample set constitute the test set.

[0125] Specifically, the samples in the training sample set and the test sample set are cropped, and after obtaining the cropped sample set, a sequence of images is generated by a sliding window method to obtain the training set and the test set.

[0126] The specific operations of the sliding window method to obtain the sequence image are as follows:

[0127] In the first step, all images of the cropped sample set are arranged in a row to obtain the total sequence of the sample set.

[0128] In the second step, a rectangular sliding window with a length of F images and a width of 1 image is used to slide on the total sequence of samples of each category in the sample set with a step size of 1 image, and all image sequences within the sliding window are taken out to form the sequence samples after the sliding window.

[0129] Here, a satellite target ISAR image dataset is constructed. This dataset contains seven types of satellites, namely CALIPSO, Jason-3, Icesat, QuikSCAT, OCO-2, Cloudsat, and SWAS. Electromagnetic calculations are used to generate HV polarization radar echoes of various types of targets with an azimuth angle of γ = 0° to 359°, with an azimuth interval of 0.05° and an elevation angle of The radar carrier frequency is 17 GHz, and the bandwidths are B = 1, 1.5, and 2 GHz, resulting in range-dimensional image deformation. After correcting the echoes for range walk, continuous ISAR imaging is performed at 1° azimuth intervals and with imaging integration angles Δθ = 4, 5, and 6°, respectively, resulting in azimuth-dimensional image deformation. Based on this, the ISAR image is cropped to 120 × 120 pixels centered on the target, and sequential ISAR image samples are generated using a sliding window with a window length of F = 3 and a step size of 1. In this example, the training set consists of four categories of samples: CALIPSO, Jason-3, Icesat, and QuikSCAT, with a depression angle of 50°. The test set consists of three categories of samples: OCO-2, Cloudsat, and SWAS, with a depression angle of 55°. Considering that ISAR images can exhibit various deformations due to differences in bandwidth and integration angle, this paper designs two experiments: one with no deformation and one with deformation to test the robustness of the small sample model to deformation.

[0130] Table 2 Training set and test set configuration of the invisible transformation experiment

[0131]

[0132] Table 3. Training set and test set configuration for deformation experiments

[0133]

[0134]

[0135] Among them, the non-deformation experiment configuration is shown in Table 2. The bandwidth-accumulation angle of the training set and test set are all combinations of Δθ = 6° B = 2 GHz, Δθ = 5° B = 1.5 GHz, and Δθ = 4° B = 1 GHz, and the azimuth angle is γ = 0° to 359°. This experiment mainly tests the model's ability to migrate to new types of data. The dataset configuration and sample number of the deformation experiment are shown in Table 3. The training set and test set have different bandwidth-accumulation angle combinations and azimuth angles. That is, the experiment includes two deformation modes: scaling and rotation. It is used to test the robustness of the model to multiple deformations in small-sample learning.

[0136] S3. Rotate the ISAR image in step S1 by 0°, 90°, 180°, and 270° respectively, and generate rotation labels for the ISAR image samples according to the rotation angles to obtain an unsupervised sample set.

[0137] S4. Based on the semi-supervised learning method, the sequence homography network is trained using the training set and the unsupervised sample set to obtain a trained sequence homography network.

[0138] In a specific embodiment, step S4 includes:

[0139] S4.1. Input the sequence sample x in the training set into the sequence homography network. After obtaining the output features, use the cosine classifier to obtain the predicted category label, and use the cross entropy loss to calculate the supervision loss.

[0140] S4.2. Set the sample x in the unsupervised sample set r Input to the sequence homography network and get the output feature f(x r ), use 4-class linear classifier to classify the output feature f(x r ) is classified to obtain the predicted rotation angle, and the self-supervised loss is calculated based on the cross entropy loss.

[0141] S4.3. Based on the loss function of semi-supervised learning, update the network parameters of the sequence homography network until a preliminarily trained sequence homography network is obtained.

[0142] Specifically, semi-supervised learning is a network training method that combines self-supervision and traditional supervised learning. Self-supervised learning trains network parameters by learning the internal structural information of the data, thereby increasing the semantic information of the sample and reducing the category bias. This embodiment adopts the rotation self-supervised learning task. First, the input image x is rotated by 0°, 90°, 180° and 270° to obtain the sample x r , where r∈R={0°, 90°, 180°, 270°}. Then use the output feature f(x r ) predict the rotation angle. Specifically, construct a 4-class linear classifier f(x r ) is classified, and its output is the predicted rotation angle. For the rotation prediction task, the self-supervised loss on the base class data is:

[0143]

[0144] Where L(·) is the cross entropy loss function K=4 is the total number of categories.

[0145] For supervised training tasks, the cosine classifier and cross entropy loss are used to obtain the supervised loss L c The classifier regularizes the features by calculating their cosine similarity. Compared with the traditional fully connected layer, the metric-based cosine classifier can reduce the intra-class distance. For the embedded feature vector z, a K-class cosine classifier is defined, and its normalized output belonging to the kth class is:

[0146]

[0147] Where γ is the scaling factor, cos(z i , w k ) is the cosine distance:

[0148]

[0149] is a learnable weight vector. As can be seen, the cosine classifier essentially performs L2 regularization on the feature vector, thereby reducing the impact of the z value on classification performance and allowing the network to focus more on inter-class differences in samples. Specifically, the cosine classifier is only used to optimize network parameters during SHN training. When the model is inferring new classes, the OTM is directly run on the feature vector z to determine the class label.

[0150] The loss of semi-supervised learning is the sum of unsupervised loss and supervised loss: L s =L rot +L c , where L c is the cross entropy loss for supervised learning.

[0151] S4.4. Based on the manifold mixing loss function, the preliminarily trained sequence homography network is trained using manifold mixing to obtain a trained sequence homography network.

[0152] Specifically, the feature manifold is a collection of feature maps at each layer during network inference. Few-shot learning hopes that the feature manifold learned from base class data is also applicable to new class data, so the feature manifold must have strong generalization performance. Manifold mixing smoothes the model classification decision boundary by interpolating multiple hidden layer features and provides a wide low confidence interval. Given different class samples x1 and x2 of the same batch of data, and let their feature maps at the lth layer of SHN be f l (x1) and f l (x2), then the manifold mixing loss L m for:

[0153] L m =L[f L-l (Mix λ (f l (x1), f l (x2))), Mix λ (y1, y2)]

[0154] in:

[0155] Mix λ (a, b) = λ·a + (1-λ)·b

[0156] The mixing coefficient λ is sampled from the Beta distribution β(α, α), α is a hyperparameter, L represents the number of layers of the network, and f L-l (·) is the network layer after the lth layer, L(·) is the cross entropy loss function, y1 and y2 are the category labels of sample image x1 and sample image x2. Finally, the overall loss in the manifold hybrid algorithm is L all =L c +L rot +L m During training, the manifold mixing layer l of each batch is randomly selected from {1, ..., L} to improve the generalization ability of all layers.

[0157] For the example of the present invention, the training is divided into two stages: pre-training and fine-tuning. In pre-training, only base class data and semi-supervised loss function (i.e., L s ); when fine-tuning, the base class and new class data are used, and the loss function is a semi-supervised loss and a manifold mixed loss (i.e., L all). At the same time, to increase sample diversity and reduce overfitting, the input image sequence is scaled by ±0.2 times, rotated by ±90°, mirrored, and transformed by ±0.2 times in brightness with a probability of 0.5. The sample sequence length is set to F = 3, the embedding feature dimension of SHN is set to d = 128, the homography parameter generator is a 2-layer Transformer structure, and the number of its attention heads is h = 2. The scaling factor γ of the cosine classifier is 2. The Adam optimizer and batch gradient descent method are used for training, with a batch size of 64 and an initial learning rate of 0.001 decayed according to the cosine law. The number of iterations for pre-training and fine-tuning is 50.

[0158] S5. Randomly select w categories from the test set, and each type of labeled sample image constitutes the support set D s , the unlabeled samples constitute the query set D q , where s and q represent the number of labeled and unlabeled samples in each class, respectively.

[0159] Specifically, the test set data in step S1 generates w-way, s-shot tasks, randomly selects w categories from the test set category, and each type of labeled samples constitutes the support set D s ={(x i ,y i ), i=1,...ws}, and the samples without labels constitute the query set D q ={(x i ), i=1,...wq}, where s and q represent the number of labeled and unlabeled samples in each class, respectively.

[0160] S6. Merge the support set and query set D s ∪D q , to obtain w(s+q) sequence samples;

[0161] S7, inputting w(s+q) sequence samples into the trained sequence homography network to obtain corresponding sample embedding feature vectors;

[0162] S8. Use power transformation to preprocess the sample embedding feature vector, and iteratively estimate the classification label and classification center through optimal transmission mapping to obtain the classification result of the test set.

[0163] After preprocessing the embedded feature vector, this embodiment needs to project the query set feature vector into the label space. To achieve this goal, the EM algorithm is used to perform two steps of iteration. Step 1: Fix the classification center c j , the maximum a posteriori estimate of the feature vector is used as the optimization objective function of the category assignment, and the OTM calculation is used to solve it; the second step: fix the category estimation result of the feature vector and re-estimate the classification center c j . Loop this process nsteps times, and finally get the optimal category label

[0164] In this embodiment, step S8 may include:

[0165] S8.1. Use power transformation to preprocess the sample embedding feature vector to obtain the preprocessed sample embedding feature vector.

[0166] Specifically, due to the uncertainty of the sample distribution and the multi-layer linear mapping and nonlinear activation function of the embedding network, the output embedding feature vector generally does not satisfy the Gaussian distribution. In addition, since the base class and the new class samples do not intersect, it is necessary to align the distribution of the two. The power transformation can map non-Gaussian data to a distribution close to Gaussian, thereby promoting accurate category estimation by stabilizing variance and minimizing deviation. Let the new class sample x∈D n The eigenvector of Then the embedded feature vector after normalized power transformation is for:

[0167]

[0168] where ε = 10 -6 , which ensures that the divisor is non-zero, β is a hyperparameter, and ||·|| is the two-norm normalization term, which is used to avoid excessive variance of the embedded features.

[0169] S8.2. Fix the category center and embed the preprocessed samples in the query set into the feature vector As data, by maximizing the sample embedding feature vector The posterior distribution of the classification label is used to optimize the distribution, and the optimal class mapping matrix M is obtained by OTM calculation. * .

[0170] Specifically, according to the maximum a posteriori estimation criterion, the embedding feature vector of the query set is As data, by maximizing The posterior distribution of To optimize the classification label distribution and obtain the optimal label According to Bayes' formula, we can get:

[0171]

[0172] The likelihood function Obey the category centers c1,...,c w is the mean, with λ / 2 as the variance of Gaussian distribution, c j Initialized by the mean of the embedding features of the labeled samples in the support set, λ is a hyperparameter. The above formula can be expanded as:

[0173]

[0174] The prior P(y i ) is the category label generated by initialization or the previous iteration, which is a constant. Therefore, the maximum posterior can be converted into the minimum Wasserstein distance, that is, the distribution arrive The OTM problem can be solved by using the discrete Sinkhom algorithm.

[0175] Specifically, the definition Class mapping matrix to categories Among them, M ij =P(y i =j) is a soft weight, that is, the same sample is allowed to have a mapping weight of [0, 1] for different categories, then the optimal class mapping matrix M * It is calculated by entropy regularization method:

[0176]

[0177] where μ is the regularization coefficient, It is a set of matrices whose elements are all positive, and each matrix satisfies the sum of rows is a and the sum of columns is b, that is, it contains all possible class assignment schemes:

[0178]

[0179] Loss Matrix is the Euclidean distance between the query set unlabeled sample features and the category center, where At the same time, the entropy of the class assignment matrix H(M) = -∑ ij M ij logM ij Acts as a regularization term to limit the complexity of class assignment.

[0180] S8.3, based on the optimal class mapping matrix M * , the classification center is re-estimated using the mapping label of the sample embedding feature vector and the true label of the support set, and the learning rate is increased in the classification center to obtain the re-estimated classification center.

[0181] Specifically, after obtaining the optimal class mapping matrix M * Then, using the eigenvector z i The mapping label and the true label of the support set are used to re-estimate the classification center c′ j :

[0182]

[0183] in, represents the embedding features of the support set samples, s1w represents the number of samples of each class in the support set.

[0184] To avoid class center jumps, the update of the classification center increases the learning rate η to smooth the iteration step, that is:

[0185] c j =c j +η(c j ′-c j )

[0186] The classification center obtained by this formula is used as the initial value for subsequent iterations.

[0187] S8.4. The re-estimated classification center is used as the initial value for each iteration. After a preset number of iterations of steps S8.2 and S8.3, the sample labels of the query set are obtained from the optimal class mapping matrix M* of the last iteration, thereby obtaining the classification result.

[0188] Specifically, the above steps S8.2 and S8.3 are performed after n steps After rounds of iteration, the sample labels of the query set are obtained from the last iteration class assignment matrix M*:

[0189]

[0190] For the example of the present invention, the test set sample category w=3, and the experiment is set to 1-shot and 5-shot respectively according to the number of support sets s. At the same time, the query set size q=200, each group of experimental tests is repeated 1000 times, and the accuracy mean and variance are taken. The other hyperparameters are set to α=2, β=-0.5, μ=20, η=0.8, n steps =20.

[0191] In addition, it should be noted that the specific process of preprocessing the embedded feature vector using power transformation and then iteratively estimating the classification label and classification center through optimal transmission mapping to obtain the classification result is the same as step S8 in the target classification process of the test set, and will not be repeated here. The effect of the present invention can be further illustrated by the following simulation experiment:

[0192] 1. Simulation experiment conditions:

[0193] The simulation experiments in this paper use a dataset of satellite target ISAR images. The target categories are CALIPSO, Jason-3, Icesat, QuikSCAT, OCO-2, Cloudsat, and SWAS, totaling seven categories. The training set consists of the first four categories, and the test set consists of the remaining three categories. The experiments are divided into two deformation modes: non-deformation and deformation. The training and test set configurations for the two deformation modes are shown in Tables 1 and 2. Each deformation mode is further divided into 1-shot and 5-shot experiments based on the value of s (s represents the number of samples of each category in the support set).

[0194] 2. Simulation content and result analysis:

[0195] Under the above simulation conditions, the present invention and the existing "Gaussian prototype-based small-sample ISAR image target recognition method" were trained on the training set, and the classification results were tested on the test set. The target classification results are summarized in Table 4.

[0196] Table 4 Comparison of experimental results

[0197]

[0198]

[0199] From the experimental results in Table 4, it can be seen that the classification performance of the present invention is significantly better than that of the existing technical solutions, and it can maintain good recognition performance in both non-deformation and deformation modes, which proves the robustness of the present invention to multiple deformations in small sample learning.

[0200] In response to the shortcomings of existing technologies, this paper proposes a robust classification method for small-sample sequence ISAR images based on semi-supervised transfer learning. By proposing a sequence homography network to extract deformation-robust temporal features of sequence ISAR images, this method addresses the problem that existing methods only extract spatial features from a single image, lack targeted processing of unknown deformations, and thus lack robustness in target classification. A small-sample ISAR image classification architecture based on semi-supervised learning and optimal transfer mapping is proposed to address the problem that existing methods only use support set samples to train the embedding network, resulting in easy confusion of the resulting embedded features when classifying the query set, resulting in low small-sample classification accuracy and insufficient model generalization.

[0201] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0202] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other changes to the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0203] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A robust classification method for small sample sequence ISAR images based on semi-supervised transfer learning, characterized by: The classification method includes: Obtain the ISAR image to be classified; Inputting the image sequence of the ISAR image to be classified into a trained sequence homography network to obtain a corresponding embedded feature vector, wherein the sequence homography network includes a cascaded backbone network, a homography parameter generator, and an image sampler; the backbone network is used to extract features from the time dimension of the image sequence and the length and height dimensions of a single frame image to obtain an output feature map, the homography parameter generator is used to query the homography parameter sequence from the output feature map to obtain homography transformation parameters, the elements of the homography transformation parameters respectively correspond to the deformation adjustment parameters of each frame image in the sequence sample, and the image sampler is used to perform homography transformation on each frame image in the image sequence using the homography transformation parameters corresponding to each frame image to obtain a deformation-adjusted image sequence; The homography parameter generator includes an encoder and a decoder; The input of the encoder is a true value , key value and query value , the true value , key value and query value They are: in, is the output feature map of the backbone network, is a learnable positional encoding, is the dimension of the embedding vector, is the length of the sequence image, 、 、 is the dimension transformation matrix, is the dimension of each head, is the index of the attention head, is layer normalization; The output of the encoder is: in, is the output of the encoder, FFN is the feedforward network, is the weight of the first fully connected layer of FFN, is the bias of the first fully connected layer of FFN, is the weight of the second fully connected layer of FFN, is the bias of the second fully connected layer of FFN, is the number of attention heads, is the aggregation weight, Represents a transpose operation; The truth value of the decoder , key value , query value They are: in, 、 、 is the dimension transformation matrix; The output image of the image sampler exist The pixel value at is: in, is the input image exist The pixel value at the position; The embedded feature vector is preprocessed by using power transformation, and then the classification label and classification center are iteratively estimated through optimal transmission mapping to obtain the classification result.

2. The robust classification method for small sample sequence ISAR images according to claim 1, characterized in that: The backbone network includes a cascade of 5 layers of 3D-CNN structures, each of which is connected to a 3D batch normalization and ReLU activation function, and the 1st to 4th layer 3D-CNN structures also use The maximum value pooling is used for feature compression. The first layer of 3D-CNN structure is dilated 3D convolution with 8 channels. The second to fifth layers of 3D-CNN structure are all dilated 3D convolution with 16 channels. The calculation method of the dilated 3D convolution is: in, is the first elements, Is the size of 3D convolution kernel, d 1. d 2. d 3 are the expansion coefficients in three dimensions respectively. Before convolution, the length of the time dimension is increased by to zero so that the temporal dimension of the output feature map remains unchanged and causal.

3. The robust classification method for small sample sequence ISAR images according to claim 1, characterized in that: The training method of the sequence homography network includes: S1. Acquire ISAR images with multiple target categories and category labels under different imaging conditions, and divide the ISAR images into a training sample set and a test sample set according to the categories, wherein the sample categories of the training sample set and the test sample set do not overlap; S2. With the target as the center, cropping the ISAR images in the training sample set and the test sample set, and processing the cropped ISAR images by a sliding window method to generate sequence images, where all sequence images corresponding to the training sample set constitute a training set, and all sequence images corresponding to the test sample set constitute a test set; S3, the ISAR images in step S1 are respectively 、 、 and Rotate and generate rotation labels for ISAR image samples according to the rotation angle to obtain an unsupervised sample set; S4. Based on a semi-supervised learning method, the sequence homography network is trained using the training set and the unsupervised sample set to obtain a trained sequence homography network.

4. The robust classification method for small sample sequence ISAR images according to claim 1, characterized in that: Step S4 includes: S4.1, the sequence samples in the training set Input to the sequence homography network, obtain the output features, use the cosine classifier to get the predicted category label, and use the cross entropy loss to calculate the supervision loss; S4.

2. The samples in the unsupervised sample set Input into the sequence homography network to obtain the output feature , using a 4-class linear classifier to classify the output features Perform classification to obtain the predicted rotation angle, and calculate the self-supervised loss based on the cross entropy loss; S4.

3. Based on the loss function of semi-supervised learning, updating the network parameters of the sequence homography network until a preliminarily trained sequence homography network is obtained; S4.

4. Based on the manifold mixing loss function, the preliminarily trained sequence homography network is further trained using manifold mixing to obtain a trained sequence homography network.

5. The robust classification method for small sample sequence ISAR images according to claim 4, characterized in that: The loss function of the semi-supervised learning is: in, is the loss function for semi-supervised learning, is the cross entropy loss function for supervised learning, is the loss function for self-supervised learning; The loss function of the self-supervised learning is: in, is the cross entropy loss function , is the rotated label of the sample, is the predicted rotation label, is the total number of categories, represents the training sample set, is the set of rotation angles, It is a 4-class linear classifier; The manifold mixing loss function is: in, and The sample images are and sample images In the first Feature map of the layer, mixing coefficient From Beta distribution Medium sampling, is a hyperparameter, represents the number of layers in the network, For the The network layer after the layer, is the cross entropy loss function, and is a sample image and sample images The category label of .

6. The robust classification method for small sample sequence ISAR images according to claim 1, characterized in that: After step S4, the method further includes: S5. Randomly take out Class, various types of labeled sample images constitute the support set , unlabeled samples constitute the query set ,in, and Represents the number of labeled and unlabeled samples in each category respectively; S6. Merge the support set and query set , to get sequence samples; S7, will Input the trained sequence homography network into the sequence samples to obtain the corresponding sample embedding feature vector; S8. Use power transformation to preprocess the sample embedding feature vector, and iteratively estimate the classification label and classification center through optimal transmission mapping to obtain the classification result of the test set.

7. The robust classification method for small sample sequence ISAR images according to claim 6, characterized in that: Step S8 includes: S8.

1. Preprocess the sample embedding feature vector using a power transform to obtain a preprocessed sample embedding feature vector; S8.

2. Fix the category center and embed the preprocessed samples in the query set into the feature vector As data, by maximizing the sample embedding feature vector The posterior distribution of is used to optimize the classification label distribution, and the optimal class mapping matrix is ​​obtained by OTM calculation. ; S8.3, based on the optimal class mapping matrix , re-estimate the classification center using the mapping label of the sample embedding feature vector and the true label of the support set, and increase the learning rate in the classification center to obtain the re-estimated classification center; S8.4, the re-estimated classification center is used as the initial value of each iteration. After the preset number of iterations of steps S8.2 and S8.3, the sample labels of the query set are mapped from the optimal class mapping matrix of the last iteration to In order to obtain the classification results.

8. The robust classification method for small sample sequence ISAR images according to claim 7, characterized in that: The sample embedding feature vector after power transformation is: in, is the sample embedding feature vector, , is a hyperparameter, is the two-norm normalization term; The optimal class mapping matrix for: Among them, L is the Euclidean distance between the unlabeled sample features and the category center in the query set, is the preprocessed sample embedding feature vector arrive The class mapping matrix of categories, yes No. Rank Elements of the column, is the regularization coefficient, is a set of matrices whose elements are all positive and whose rows sum to , the sum of the columns is , , For classification center, .

Images