A method for generating and classifying ship targets in SAR images
The SAR image with category tags is generated through the LST-ACGAN algorithm and the smooth label and triple loss is used to solve the problems of overfitting small-scale data sets in ship classification and limited information, which improves the generalization ability of the classifier and classification performance.
Patent Information
- Application Number
- CN202310771564.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-27
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-06-27
AI Technical Summary
The existing SAR image ship classification methods are prone to overfitting on small-scale data sets. Due to the limited SAR image information, high intra-class diversity and inter-class similarity, it is difficult to learn classification boundaries, which reduces the classification performance of the algorithm.
A GAN-based algorithm LST-ACGAN is proposed to generate images with category tags through generator G, and use discriminator D to verify the authenticity of the input image, while preventing overfitting by smoothing labels and triple losses, improving The generalization ability and category differentiation of the model.
By generating samples, expanding data distribution, reducing manual annotation costs, improving the robustness and generalization capabilities of classifiers, enhancing the category discrimination of model learning, and improving classification performance.
Smart Images

Figure CN116895016B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of synthetic aperture radar image (SAR image) target generation and classification, and in particular to a SAR image ship target generation and classification method. Background Art
[0002] Ship classification plays an important role in various maritime activities, such as national defense intelligence, fishery monitoring and maritime search. SAR has gradually become a key device for ship monitoring due to its all-day and all-weather operation. With the launch of a new generation of synthetic aperture radar satellites, a large number of medium and high-resolution synthetic aperture radar images have gradually become easy to obtain, making it possible to identify ship types.
[0003] Traditional SAR ship classification methods usually extract low-level features such as geometric features, scattering features, and statistical features through manual methods, and then use traditional machine learning algorithms to classify ships. However, these manually extracted features are difficult to fully represent ship targets and their performance is limited in complex scenes.
[0004] In recent years, SAR ship classification methods based on deep learning have received more and more attention and become a hot topic in related research. For example, Li et al. (Li J, Qu C, Peng S. Ship classification for unbalanced SAR datasetbased on convolutional neural network[J]. Journal of Applied Remote Sensing, 2018) designed a dense residual network with resampling and integrated cost-sensitive learning to address the class imbalance problem in ship classification. Dechesne et al. (Dechesne, C.; Lefèvre, S.; Vadaine, R.; Hajduch, G.; Fablet, R. Ship Identification and Characterization in Sentinel-1 SAR Images with Multi-Task Deep Learning. Remote Sens. 2019, 11, 1-18, doi: 10.3390 / rs11242997.) designed a multi-task structure to simultaneously achieve detection, classification, and length estimation tasks. Firoozy et al. (Firoozy, N.; Sandirasegaram, N. Tackling SAR Imagery Ship Classification Imbalance via Deep Convolutional Generative Adversarial Network. Can. J. Remote Sens. 2021, 47, 295-308, doi: 10.1080 / 07038992.2021.1910499.) used adversarial training to generate samples for the minority class to balance the training dataset. He et al. (He, J.; Wang, Y.; Liu, H. Ship Classification in Medium-ResolutionSAR Images via Densely Connected Triplet CNNs Integrating Fisher Discrimination Regularized Metric Learning. 2021, 59, 3022-3039.) proposed a densely connected triplet neural network with Fisher regularization term for medium-resolution SAR image ship classification tasks.Zeng et al. (Zeng, L.; Zhu, Q.; Lu, D.; Zhang, T.; Wang, H.; Yin, J.; Member, S. Dual-Polarized SARShip Grained Classification Based on CNN With Hybrid Channel Feature Loss. 2022, 19, 16-20.) proposed to jointly use the information contained in the polarization channels (VV and VH) by designing a hybrid channel feature loss for dual-polarization SAR ship grain classification. Zhang et al. (Zhang, Y.; Lei, Z.; Yu, H.; Zhuang, L. Imbalanced High-Resolution SAR Ship Recognition Method Based on a Lightweight CNN. IEEE Geosci. Remote Sens. Lett. 2022, 19, 0-4, doi: 10.1109 / LGRS.2021.3083262.) designed a lightweight CNNs classification model that combines DML with stepwise balanced sampling to reduce computational complexity and solve the class imbalance problem in high-resolution SAR images. In summary, deep learning-based methods can directly extract high-level features from large-scale data and perform end-to-end training, showing great potential in practical applications. However, the amount of SAR image ship classification data currently available is relatively small, especially for some maritime military targets such as destroyers and aircraft carriers, and manual labeling is expensive. Due to the large number of deep learning model parameters, deep learning-based models trained on small-scale datasets usually have overfitting problems, which reduces the generalization ability of the model. In addition, unlike optical images, due to the defects of SAR imaging mechanism, SAR images have limited information, resulting in relatively small differences in the appearance of different types of ships, while relatively large differences in the appearance of the same type of ships. High intra-class diversity and inter-class similarity make it difficult for the algorithm to learn classification boundaries, which will further reduce the classification performance of the algorithm.
[0005] To solve the above problems, this paper proposes LST-ACGAN for SAR ship classification. New samples are generated by introducing ACGAN to expand the sample distribution. Smooth labels are assigned to the generated samples to prevent the algorithm from overfitting and improve the generalization ability of the model. The triplet loss is integrated into ACGAN to improve the distinguishability of samples of different classes. Summary of the invention
[0006] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and provide a method for generating and classifying ship targets in SAR images.
[0007] The technical solution provided by the present invention is as follows: Although the SAR image ship classification based on deep learning has achieved remarkable performance in large-scale training samples, when the training data is insufficient, these methods are difficult to learn the distribution of data and are prone to overfitting. In view of this problem, the present invention proposes a GAN-based algorithm, called LST-ACGAN, for completing classification while generating images. The overall model of LST-ACGAN includes a generator G and a discriminator D. The generator G can generate images with category labels, while the discriminator D can verify whether the input image is real and realize classification at the same time. By assigning smooth labels to the images generated by the generator G, and proposing a smooth label-based cross entropy loss (SL-CE) as the classification loss of the generated image, the model is regularized during training and the model collapse is avoided. The triplet loss is integrated into the classification network of the discriminator D to learn the distinguishable features related to the category. During the training process, a triplet is constructed by sampling anchor samples, positive samples and negative samples from real images, and it is input into the discriminator D together with the generated image, and then the discriminator and the generator are jointly optimized through multiple loss functions. After training is completed, the discriminator is used to classify the input test samples.
[0008] A method for generating and classifying ship targets in SAR images, characterized in that it comprises the following steps:
[0009] Step 1: Construction of training and testing sets:
[0010] Collect SAR ship target images and divide them into training and test sample sets; then perform sample amplification on the training samples so that the number of training samples of each type is the same as the number of training samples of the class with the largest number of training samples;
[0011] Step 2: Generator G construction:
[0012] Generate a generated image x with a category label consistent with the distribution of training samples fake , construct generator G based on DCNN;
[0013] Step 3: Design the network structure of the discriminator D;
[0014] The input of the discriminator D is the real image x real And the generated image x output by the generator G in step 2 fake The obfuscated data block is output as a triple feature, a category prediction result, and a true or false prediction result after passing through a feature extraction network and three multi-task learning sub-networks, and the network parameters of the discriminator D and the generator G are updated using the corresponding losses. The three multi-task learning sub-networks are a triple feature learning sub-network, a category prediction sub-network, and a true or false prediction sub-network;
[0015] Step 4: Classification subnetwork smoothing label construction and loss function construction:
[0016] The input image of the discriminator D constructed in step 3 is the real image x real and generate image x fake After the confusion, the above two images are input into the category prediction subnetwork. For the real image x real , based on the hard label, the cross entropy loss is used to calculate the real image x real The category prediction loss for generating an image x fake , the category prediction loss of the generated images is calculated based on the smoothed labels.
[0017] Step 5: Triplet loss construction:
[0018] Input the feature vector learned by the feature extraction network in step 3 of the discriminator into the fully connected layer (FC) to obtain the output f of the triple feature learning subnetwork triplets , triplet features are optimized using triplet loss to learn class-related discriminative features.
[0019] Step 6: Generator and discriminator loss function construction:
[0020] Based on the generator and discriminator networks constructed in steps 2 and 3 and the loss functions constructed in steps 4 and 5, construct the overall loss function of the generator G and the discriminator D, and use the constructed overall loss function to alternately optimize the generator and the discriminator.
[0021] Step 7: Model training and testing:
[0022] The model constructed from step 2 to step 6 is trained. After the training is completed, the discriminator is used to classify the input test samples.
[0023] Furthermore, in the step 2 generator G structure:
[0024] Generate a generated image x that is consistent with the distribution of the training image based on the input vector z′ fake ; The input vector z′ is obtained by formula (1), where embedding(y fake ) indicates that the category label y fake The transformation is done to obtain a vector of length m. The specific transformation process is as follows: Assuming that the classification task contains C categories {1, ..., C}, first randomly generate a random matrix M with a dimension of C×m, and then randomly generate a category label y fake ∈{1,...,C}, and transform it into a one-hot label y with dimension 1×C oh , and then calculate y oh ×M, and finally obtain a vector embedding(yfake ); embedding the above vector (y fake ) is multiplied element by element with the randomly generated noise vector z of length m, and finally the input vector z′ of the generator is obtained;
[0025] z′=embedding(y fake )oz (1)
[0026] Where o represents element-by-element multiplication;
[0027] Then, the generator G takes the input vector z′ as input and outputs the generated image x fake , the generator G is constructed based on DCNN. The generator G includes a conversion module, two UC modules and a generation module. For the input vector z′, the conversion module converts z′ from a one-dimensional vector to a three-dimensional feature map through full connection (FC), reshape operation and batch normalization (BN), and then inputs it into the UC module. The UC module first upsamples the three-dimensional feature map output by the previous module by 2 times through the Upsampling layer, and then realizes feature learning through the convolution layer (Conv), batch normalization layer (BN) and activation layer (Leaky ReLU). In the convolution layer Conv, c is the number of convolution kernels, k is the convolution kernel size, s is the step size, and p is the boundary expansion parameter; β is the slope coefficient. Finally, the generation module reduces the three-dimensional feature map output by the UC2 module through the convolution layer Conv, and constrains the values in the above three-dimensional feature map to (-1, 1) through the Tanh function.
[0028] Furthermore, in the step 3 discriminator D network structure design:
[0029] The pre-trained ResNet18 with the FC layer removed is used as the feature extraction network; b input images are input into the feature extraction network to obtain b feature maps, and the b feature maps are converted into b feature vectors through the global average pooling layer (GAP) and the Reshape layer, and then the b feature vectors are normalized using L2 normalization to finally obtain b feature vectors; finally, the feature vectors are respectively input into the three multi-task learning sub-networks mentioned above;
[0030] The category prediction subnetwork obtains the category prediction value of the image input by the discriminator D through the softmax function:
[0031]
[0032] Where C represents the number of categories, p i ,p j (i, j = 1, 2, ..., C) represents the classification vector of softmax output, q iRepresents the probability that the input image belongs to the i-th category, that is, the category prediction value;
[0033] The authenticity prediction subnetwork uses the sigmoid function to map the output to (0, 1) to obtain the authenticity prediction value of the input image:
[0034]
[0035] Where v represents a binary classification vector, and p(v) represents the probability that the input image belongs to the true category, that is, the true or false prediction value.
[0036] Furthermore, in the step 4, the classification subnetwork smoothing label construction and loss function construction are:
[0037] Input the above two images into the category prediction subnetwork and obtain the category prediction value q i , and then calculate the losses separately:
[0038] For a real image x real , hard labels are directly used for calculation, and hard labels can be expressed as:
[0039]
[0040] where y i (i=1,2,…,C) represents a one-hot encoding vector. real When y belongs to the i-th category i =1; the category prediction subnetwork outputs the real image x through the softmax function real The probability q of belonging to each category i (See formula (2)), and then use cross entropy loss to calculate the true image x real The category prediction loss l CE :
[0041]
[0042] For the generated image x fake , calculate the smooth label Assigned to the generated image x output by the generator G fake , used to calculate the category prediction loss of the generated image, the smoothed label is a vector of length C in (i={1,...,C}) represents the generated image x fake The smoothed label of category i means the probability that the image belongs to category i, which can be calculated by formula (6):
[0043]
[0044] Where ε is a hyperparameter, C represents the number of categories; for the generated image x fake According to formula (2), the category prediction value q is obtained i , the category prediction loss l of the generated image SL-CE It can be expressed as:
[0045]
[0046] Furthermore, in the step 5 triplet loss construction:
[0047] Assume x a is the real image obtained by random sampling, x p For x a Real images belonging to the same category, x n For x a Real images belonging to different categories, f(x) is the feature vector obtained after the real image passes through the feature extraction network and the triple feature learning subnetwork, then select the input image in f(x) that does not meet the following distance constraints to construct the triple sample <x a ,x p ,x n >:
[0048] d(f(x a ),f(x p ))+α<d(f(x a ),f(x n )) (8)
[0049] Where d(·) represents the Euclidean distance and α is the classification interval;
[0050] For samples that satisfy the constraints of formula (8), the loss value is 0. For triplet samples that do not satisfy formula (8), the loss value is x a With x p and x n The distance difference of the feature vector learned by f(x), that is, the triple loss l triplets :
[0051] l triplets =d(f(x a ),f(x p ))-d(f(x a ),f(x n )) (9)
[0052] Furthermore, in step 6, the generator and discriminator loss functions are constructed:
[0053] First, for the true-false prediction subnetwork in the discriminator D, the binary cross entropy loss function is used to calculate the true-false prediction loss lBCE To distinguish the real image x real and generate image x fake , which can be expressed as:
[0054] l BCE =-(y v *log(p(v))+(1-y v )*log(1-p(v))) (10)
[0055] where y v Represents the true or false binary classification label. When the input image is a real image, y v = 0, when the input image is the generated image y v =1; p(v) is the true or false prediction value of the input image calculated by formula (3);
[0056] Then, construct the total loss function L of the generator G G ; This function is composed of the true and false prediction loss l BCE And the category prediction loss l of the generated image SL-CE It consists of two parts, and its formula is as follows:
[0057]
[0058] in represents the smooth label of the generated image calculated according to formula (6); λ1, λ2 represent the trade-off parameters;
[0059] Finally, the total loss function of the discriminator D is constructed; the true and false prediction loss l is expressed as BCE , the category prediction loss l of the generated image described in formula (7) SL-CE , the category prediction loss l of the real image described in formula (5) CE And the triplet loss l described in formula (9) triplets Optimize; the total loss function L of the discriminator D D The formula is as follows:
[0060]
[0061] where x real 、x fake Represent the real image and the generated image respectively, y real represents the hard label of the real image, represents the smoothed label of the generated image, y v is the true or false binary classification label described in formula (10), x a ,x p ,x n is the triplet sample constructed by formula (8), λ i(i=3,4,···,7) represents the trade-off parameter.
[0062] Furthermore, in the step 7 model training and testing:
[0063] Using Adam as the optimizer, train the model constructed from steps 2 to 6 according to the following steps:
[0064] ① Randomly generate b vectors according to formula (1) and input them into generator G to obtain b generated images;
[0065] ② The b generated images are mixed with the b real images and then input into the discriminator D. The losses are calculated according to formula (12) and the weights of the discriminator D are updated.
[0066] ③ Calculate the generator loss and update the generator G weight according to formula (11);
[0067] ④Repeat processes ① to ③ until the training times are reached.
[0068] After the training is completed, the discriminator D in the model designed in steps 2-6 is retained, the test sample is input into the discriminator D, the prediction result of the category prediction subnetwork is output, and the sample category is obtained.
[0069] The beneficial effects of the present invention are as follows: through steps 2 and 3, the present invention can construct an ACGAN model to generate various types of ship target images, so as to reduce the cost of manual annotation, and increase the diversity of data distribution from the perspective of the model, thereby improving the robustness and generalization ability of the classifier.
[0070] Through step 4, soft labels of samples are generated for the ACGAN model, and the model is trained with soft labels to prevent overfitting of the algorithm and improve the generalization ability of the algorithm.
[0071] Through step 5, the category distinguishability features can be better learned, thereby increasing the intra-class similarity and inter-class difference of the semantic features learned by the model to improve the distinction between different categories in the classification space. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] Figure 1 It is the overall framework diagram of the present invention;
[0073] Figure 2 is a network architecture diagram of the discriminator of the present invention;
[0074] Figure 3 It is a schematic diagram of the effect of triplet loss of the present invention;
[0075] Figure 4 is the confusion matrix output by the designed model of the present invention on the three-classification task,
[0076] (a)ACGAN; (b)ResNet18; (c)DenseNet121; (d)Xception; (e)LST-ACGAN.;
[0077] Figure 5 is the confusion matrix output by the designed model of the present invention on the five-category task,
[0078] (a)ACGAN; (b)ResNet18; (c)DenseNet121; (d)Xception; (e)LST-ACGAN.. DETAILED DESCRIPTION
[0079] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings:
[0080] like Figure 1-Figure 5 As shown, a method for generating and classifying ship targets in SAR images specifically comprises the following steps:
[0081] Step 1: Construction of training and testing sets.
[0082] SAR ship target images are collected and uniformly cropped into 1×64×64 size using the bicubic interpolation method, and divided into training and test sample sets; then the training samples are augmented by translation and rotation, and the number of training samples of classes with fewer numbers is made the same by performing more sample augmentation operations on the training samples of each type, that is, the number of training samples of each type is made the same as the number of training samples of the class with the largest number of training samples.
[0083] Step 2: Generator G is constructed.
[0084] The constructed generator G is used to generate a generated image x that is consistent with the distribution of the training image according to the input vector z′ fake . The input vector z′ is obtained by formula (1). In this formula, embedding(y fake ) indicates that the category label y fake The transformation is done to obtain a vector of length m. The specific transformation process is as follows: Assuming that the classification task contains C categories {1, ..., C}, first randomly generate a random matrix M with a dimension of C×m, and then randomly generate a category label y fake ∈{1,...,C}, and transform it into a one-hot label y oh (dimension is 1×C), and then calculate y oh ×M, and finally obtain a vector embedding(y fake ). The above vector embedding(y fake) is multiplied element-by-element with the randomly generated noise vector z of length m, and finally the input vector z′ of the generator is obtained.
[0085] z′=embedding(y fake )oz (1)
[0086] Where o represents element-by-element multiplication. In this way, y fake The category label represented is deeply fused with the noise vector z.
[0087] Then, the generator G takes the input vector z′ as input and outputs a generated image x with a dimension of 1×64×64 fake . The generator G is constructed based on DCNN (Unsupervised Representation Learning with Deep ConvolutionalGenerative Adversarial Networks). Its specific structure is shown in Table 1. The generator G includes a conversion module, two UC modules and a generation module (see Table 1). For the input vector z′, the conversion module converts z′ from a one-dimensional vector of length 128 to a three-dimensional feature map of size 128×16×16 through full connection (FC), reshape operation and batch normalization (BN), and then inputs it into the UC module. The UC module first upsamples the three-dimensional feature map output by the previous module by 2 times through the Upsampling layer, and then implements feature learning through the convolution layer (Conv), batch normalization layer (BN) and activation layer (Leaky ReLU). In the convolution layer Conv, c is the number of convolution kernels, k is the convolution kernel size, s is the step size, and p is the boundary expansion parameter; the slope coefficient β in the Leaky ReLU is 0.2. Finally, the generation module reduces the three-dimensional feature map of size 64×64×64 output by the UC2 module to 1×64×64 through a 3×3 convolutional layer Conv with a channel size of 1, and constrains the values in the above three-dimensional feature map to (-1, 1) through the Tanh function.
[0088] Table 1 Generator network structure
[0089]
[0090] Step 3: Design the discriminator D network structure.
[0091] The input of the discriminator D is the real image x real And the generated image x output by the generator G in step 2 fakeThe obfuscated data block passes through the feature extraction network and three multi-task learning subnetworks (triplet feature learning subnetwork, category prediction subnetwork, and authenticity prediction subnetwork), and the output is triplet features, category prediction results, and authenticity prediction results, and the corresponding losses are used to update the network parameters of the discriminator D and the generator G.
[0092] The discriminator D uses a multi-task learning structure to learn triple features, category prediction results, and true and false prediction results. Its network architecture is as follows: Figure 2 As shown in the figure, b is the batch size.
[0093] The discriminator D mainly includes the feature extraction network and the above three multi-task learning sub-networks. The present invention uses the pre-trained ResNet18 with the FC layer removed as the feature extraction network. real and generate image x fake ) is input into the feature extraction network to obtain b 512×4×4 size feature maps, and the above b feature maps are converted into b feature vectors with a length of 512 through the global average pooling layer (GAP) and the Reshape layer, and then the above b feature vectors are normalized using L2 normalization to stabilize the learning process, and finally b feature vectors with a length of 512 are obtained. Finally, the above feature vectors are respectively input into the above three multi-task learning sub-networks.
[0094] Specifically, the category prediction subnetwork obtains the category prediction value of the image input by the discriminator D through the softmax function:
[0095]
[0096] Where C represents the number of categories, p i ,p j (i, j = 1, 2, ..., C) represents the classification vector of softmax output, q i It represents the probability that the input image belongs to the i-th category, that is, the category prediction value.
[0097] The authenticity prediction subnetwork uses the sigmoid function to map the output to (0, 1) to obtain the authenticity prediction value of the input image:
[0098]
[0099] Where v represents a binary classification vector, and p(v) represents the probability that the input image belongs to the true category, that is, the true or false prediction value.
[0100] Step 4: Smooth label construction and loss function construction of classification sub-network.
[0101] The input image of the discriminator D constructed in step 3 is the real image x real and generate image x fake After the confusion, the above two images are input into the category prediction subnetwork and the category prediction value q is obtained. i , and then calculate the loss in the following two ways:.
[0102] For a real image x real , hard labels can be directly used for calculation, and hard labels can be expressed as:
[0103]
[0104] where y i (i=1,2,…,C) represents a one-hot encoding vector. real When y belongs to the i-th category i = 1. The category prediction subnetwork outputs the real image x through the softmax function real The probability q of belonging to each category i (See formula (2)), and then use cross entropy loss (CE) to calculate the real image x real The category prediction loss l CE :
[0105]
[0106] For the generated image x fake , a smooth label method is designed to calculate the classification loss. Specifically, the quality of images generated by GAN is not perfect, especially ACGAN with class labels. When the training data is insufficient, it is more difficult for ACGAN to generate images with accurate class labels. However, hard labels with CE loss encourage the model to output integer prediction labels equal to 1 or 0 for target or non-target categories. Training based on the above prediction labels will cause the target class p in the logit classification vector to be i The value of tends to infinity, making the model overconfident in its predictions. Therefore, when there are insufficient training samples, the ACGAN that generates images with hard labels is prone to model collapse during training. To address this problem, we calculate the smoothed label Instead of hard labels assigned to the generated image x output by the generator G fake , used to calculate the category prediction loss of the generated image, the smoothed label is a vector of length C in (i={1,...,C}) represents the generated image x fake The smoothed label of category i means the probability that the image belongs to category i, which can be calculated by formula (6):
[0107]
[0108] Where ε is a hyperparameter and C represents the number of categories. fake According to formula (2), the category prediction value q is obtained i , the category prediction loss l of the generated image SL-CE It can be expressed as:
[0109]
[0110] In this way, by adding noise to the generated images, the model is prevented from being overconfident in the classification results, and the prediction difference between the target category and the non-target category is appropriately reduced, thereby regularizing the model to reduce the collapse of the model during training.
[0111] Step 5: Triplet loss construction
[0112] The feature vector learned by the feature extraction network in step 3 is input into the fully connected layer (FC) with a dimension of 128 to obtain the output f of the triple feature learning subnetwork triplets In order to better learn the class-related distinguishing features to solve the problem of high intra-class diversity and high inter-class similarity, the present invention uses triplet loss to optimize triplet features. Triplet loss aims to learn distinguishable features by bringing intra-class samples closer and pushing inter-class samples further away. Its function diagram is shown in Figure 3 As shown in , this conflicts with label smoothing for generated images. Therefore, the triplet loss is only used for real images x real Specifically, suppose x a is the real image obtained by random sampling, x p For x a Real images belonging to the same category, x n For x a Real images belonging to different categories, f(x) is the feature vector obtained after the real image passes through the feature extraction network and the triple feature learning subnetwork, then select the input image in f(x) that does not meet the following distance constraints to construct the triple sample <x a ,x p ,x n >:
[0113] d(f(x a ),f(x p ))+α<d(f(x a ),f(x n )) (8)
[0114] Wherein d(·) represents the Euclidean distance, and α is the classification interval, which is uniformly set to 0.2 in this embodiment.
[0115] For samples that satisfy the constraints of formula (8), the loss value is 0. For triplet samples that do not satisfy formula (8), the loss value is x a With x p and x n The distance difference of the feature vector learned by f(x), that is, the triple loss l triplets :
[0116] l triplets =d(f(x a ),f(x p ))-d(f(x a ),f(x n )) (9)
[0117] Step 6: Construct the generator and discriminator loss functions.
[0118] Based on the generator and discriminator networks constructed in steps 2 and 3 and the loss functions constructed in steps 4 and 5, the overall loss function of the generator G and the discriminator D is constructed in the following way, and the constructed overall loss function is used to alternately optimize the generator and the discriminator.
[0119] First, for the true-false prediction subnetwork in the discriminator D, the binary cross entropy loss function is used to calculate the true-false prediction loss l BCE To distinguish the real image x real and generate image x fake , which can be expressed as:
[0120] l BCE =-(y v *log(p(v))+(1-y v )*log(1-p(v))) (10)
[0121] where y v Represents the true or false binary classification label. When the input image is a real image, y v = 0, when the input image is the generated image y v =1; p(v) is the true or false prediction value of the input image calculated by formula (3).
[0122] Then, construct the total loss function L of the generator G G The function is expressed by formula (10) as the true or false prediction loss l BCE And the category prediction loss l of the generated image described in formula (7) SL-CE It consists of two parts, and its formula is as follows:
[0123]
[0124] in represents the smooth label of the generated image calculated according to formula (6); v is the true or false binary classification label in formula (10); λ1 and λ2 represent the trade-off parameters, and λ1 and λ2 are set to 1.
[0125] Finally, the total loss function of the discriminator D is constructed. The discriminator D aims to distinguish between real images and generated images and learn the class distinguishability features of real images. Its network parameters are expressed by the true and false prediction loss l described in formula (10). BCE , the category prediction loss l of the generated image described in formula (7) SL-CE , the category prediction loss l of the real image described in formula (5) CE And the triplet loss l described in formula (9) triplets Optimize. The total loss function L of the discriminator D D The formula is as follows:
[0126]
[0127] where x real 、x fake Represent the real image and the generated image respectively, y real represents the hard label of the real image, represents the smoothed label of the generated image, y v is the true or false binary classification label described in formula (10), x a ,x p ,x n is the triplet sample constructed by formula (8), λ i (i=3,4,···,7) represents the trade-off parameter, λ i Both are set to 1.
[0128] Step 7: Model training and testing
[0129] Adam is used as the optimizer, the learning rate is set to 0.0002, the batch size b is set to 128, and the number of training times is set to 200. The model constructed from steps 2 to 6 is trained according to the following steps.
[0130] ① Randomly generate b vectors according to formula (1) and input them into generator G to obtain b generated images;
[0131] ② The b generated images are mixed with the b real images and then input into the discriminator D. The losses are calculated according to formula (12) and the weights of the discriminator D are updated.
[0132] ③ Calculate the generator loss and update the generator G weight according to formula (11);
[0133] ④Repeat processes ① to ③ until the training times are reached.
[0134] After the training is completed, the discriminator D in the model designed in steps 2-6 is retained, the test sample is input into the discriminator D, the prediction result of the category prediction subnetwork is output, and the sample category is obtained.
[0135] In order to prove the effectiveness of the method, the present invention verifies the effect of the algorithm through experiments. The dataset used is OpenSARShip, which contains five types of ships, and their numbers are shown in Table 2.
[0136] Table 2. Number of vessels of different types
[0137]
[0138] Based on the above ship types, two sub-datasets were reconstructed, as shown in Table 3. First, 100 samples of each ship type were randomly selected as the standard test set T. Then, according to the class with the least number of samples, a class-balanced three-class training set D1 and a class-balanced five-class training set D2 were reconstructed, thereby constructing two classification tasks D1 and D2.
[0139] Table 3. Number of training and testing samples for the two constructed classification tasks (D1 and D2)
[0140]
[0141] To ensure a fair comparison, the input images are resized to 1×64×64 pixels for all experiments. Adam is used as the optimizer and the learning rate is set to 0.0002. The training batch size is set to 128 and the number of training epochs is 200. The trade-off parameter λ is i (i=1,2,···,7) is set to 1.
[0142] In order to verify the effectiveness of label smoothing, ε=0.1, 0.2, 0.3, 0.4, 0.5, and 0.6 were set respectively, and classification experiments were carried out on D1. The results are shown in Table 4. It can be found that the performance of the algorithm is relatively stable under different label smoothing parameters. When ε=0.4 is set, the best performance is 88.33%. Therefore, in the following, ε is fixed to 0.4 for all experiments.
[0143] Table 4. Accuracy when using different label smoothing parameters on D1.
[0144]
[0145] In order to verify the influence of the feature dimension m of z′ in formula (1) on the effect, the dimension m of z′ is set to 64, 128, 256, and 512 respectively. From Table 5, it can be found that the accuracy will decrease when the feature dimension is too small or too large. This may be because the feature representation ability is weaker when the dimension is small, and the model will overfit to the limited number of samples when the feature dimension is too large. Therefore, for the following experiments, all feature dimensions are fixed to 128.
[0146] Table 5. Accuracy when using different feature dimensions on the D1 task.
[0147]
[0148] In order to further prove the effectiveness of the label smoothing method described in formula (6) and the triplet loss described in formula (9), the algorithm effects under different loss function combinations were tested by ablation experiments. The results are shown in Table 6, where "CE loss" means using hard labels for the generated images, that is, setting ε = 0. At this time, the smoothed labels described in formula (6) are converted into hard labels. "SL-CE loss" means using the designed label smoothing method for the generated images. According to the results in Table 4, ε is set to 0.4, and "Triplet loss" means using triplet loss. It can be found that compared with the model results optimized by CE loss, embedding SL-CE loss and triplet loss obtains better results, among which the combination of "SL-CE loss + Triplet loss" obtains the best performance of 88.33%, which is 13% higher than the baseline method (75.33%). Therefore, the integration of label smoothing method and triplet loss is crucial to improving the ship classification performance.
[0149] Table 6. Accuracy of using different loss function combinations for three-class classification on D1 data.
[0150]
[0151] In order to prove the effectiveness of the algorithm, the proposed algorithm is compared with the existing related classification algorithms, including ACGAN, ResNet18, DenseNet121, and Xception.
[0152] Table 7. Accuracy of different algorithms for three-category tasks on D1
[0153]
[0154] The three-classification results of different algorithms are shown in Table 7. The proposed LST-CGAN achieves the best accuracy of 88.33%, which is 24.66% higher than ACGAN (63.67%), 9.66% higher than ResNet18 (78.67%), 5.33% higher than DenseNet121 (83.00%), and 7% higher than Xception (81.33%). In addition, by combining the proposed method with DenseNet21 and Xception, the accuracy is improved by 4.33% and 2.67% respectively compared with the original algorithm.
[0155] The confusion matrix of different algorithms is as follows Figure 4 As shown in the figure, it can be found that the classification accuracy of LST-ACGAN on these three ship types is better than most previous methods, which shows the effectiveness of the designed method.
[0156] The five-class classification results of different algorithms are shown in Table 8. The proposed LST-ACGAN achieved the best accuracy of 56.40%, which is 15% higher than ACGAN (41.40%), 2.4% higher than ResNet18 (54.00%), 5.1% higher than DenseNet121 (51.30%), and 6.32% higher than Xception (50.08%). In addition, by replacing the feature extraction network and category prediction subnetwork in classifier D with DenseNet 121 or Xception network, the proposed method is combined with DenseNet 121 and Xception, which improves the accuracy by 0.8% and 0.6% compared with the original method.
[0157] Table 8. Accuracy of the algorithm when performing five-category classification on the D2 task
[0158]
[0159]
[0160] The accuracy of different algorithms is as follows Figure 5 As shown in Figure 2, it can be found that the classification accuracy of LST-ACGAN for these five ship types is better than most previous methods.
[0161] It should be understood that the parts not elaborated in detail in this specification belong to the prior art. The above embodiments are only descriptions of the preferred implementation methods of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made by ordinary engineers and technicians in this field to the technical solution of the present invention should fall within the protection scope determined by the claims of the present invention.
Claims
1. A method for generating and classifying ship targets in SAR images. Features It includes the following steps: Step 1: Construction of training and testing sets: Collect SAR ship target images and divide them into training and test sample sets; then perform sample amplification on the training samples so that the number of training samples of each type is the same as the number of training samples of the class with the largest number of training samples; Step 2: Generator G construction: Generate a generated image x that is consistent with the distribution of the training image based on the input vector z′ fake ; The input vector z′ is obtained by formula (1), where embedding(y fake ) indicates that the category label y fake The transformation is done to obtain a vector of length m. The specific transformation process is as follows: Assuming that the classification task contains C categories {1, ..., C}, first randomly generate a random matrix M with a dimension of C×m, and then randomly generate a category label y fake ∈{1,...,C}, and transform it into a one-hot label y with dimension 1×C oh , and then calculate y oh ×M, and finally obtain a vector embedding(y fake ); embedding the above vector (y fake ) is multiplied element by element with the randomly generated noise vector z of length m, and finally the input vector z′ of the generator is obtained; in Represents element-wise multiplication; Then, the generator G takes the input vector z′ as input and outputs the generated image x fake , the generator G is constructed based on DCNN. The generator G includes a conversion module, two UC modules and a generation module; the two UC modules are UC1 module and UC2 module. For the input vector z′, the conversion module converts z′ from a one-dimensional vector to a three-dimensional feature map through full connection FC, reshape operation and batch normalization BN, and then inputs it into the UC module; the UC module first upsamples the three-dimensional feature map output by the previous module by 2 times through the Upsampling layer, and then realizes feature learning through the convolution layer Conv, batch normalization layer BN and activation layer Leaky ReLU; finally, the generation module reduces the three-dimensional feature map output by the UC2 module through the convolution layer Conv, and constrains the value in the above three-dimensional feature map to (-1, 1) through the Tanh function; Step 3: Design the network structure of the discriminator D; The input of the discriminator D is the real image x real And the generated image x output by the generator G in step 2 fake The obfuscated data block is output as triple features, category prediction results and authenticity prediction results after passing through the feature extraction network and three multi-task learning sub-networks, and the network parameters of the discriminator D and the generator G are updated using the corresponding losses. The three multi-task learning sub-networks are the triple feature learning sub-network, the category prediction sub-network and the authenticity prediction sub-network; specifically, they include: The pre-trained ResNet18 with the FC layer removed is used as the feature extraction network; b input images are input into the feature extraction network to obtain b feature maps, and the b feature maps are converted into b feature vectors through the global average pooling layer GAP and the Reshape layer, and then the b feature vectors are normalized using L2 normalization to finally obtain b feature vectors; finally, the feature vectors are respectively input into the three multi-task learning sub-networks mentioned above; The category prediction subnetwork obtains the category prediction value of the image input by the discriminator D through the softmax function: Where C represents the number of categories, p i ,p j (i, j = 1, 2, ..., C) represents the classification vector of softmax output, q i Represents the probability that the input image belongs to the i-th category, that is, the category prediction value; The authenticity prediction subnetwork uses the sigmoid function to map the output to (0, 1) to obtain the authenticity prediction value of the input image: Where v represents a binary classification vector, and p(v) represents the probability that the input image belongs to the real category, that is, the true or false prediction value; Step 4: Classification subnetwork smoothing label construction and loss function construction: The input image of the discriminator D constructed in step 3 is the real image x real and generate image x fake The data block after confusion; the above two images are input into the category prediction subnetwork, for the real image x real , based on the hard label, the cross entropy loss is used to calculate the real image x real Category prediction loss; for the generated image x fake , based on the smoothed label calculation, the category prediction loss of the generated image is calculated; specifically: Input the above two images into the category prediction subnetwork and obtain the category prediction value q i , and then calculate the losses separately: For a real image x real , hard labels are directly used for calculation, and hard labels can be expressed as: where y i (i=1,2,…,C) represents a one-hot encoding vector. real When y belongs to the i-th category i =1; the category prediction subnetwork outputs the real image x through the softmax function real The probability q of belonging to each category i , see formula (2), and then use cross entropy loss to calculate the real image x real The category prediction loss l CE : For the generated image x fake , calculate the smooth label Assigned to the generated image x output by the generator G fake , used to calculate the category prediction loss of the generated image, the smoothed label is a vector of length C in Represents the generated image x fake The smoothed label of category i means the probability that the image belongs to category i, which can be calculated by formula (6): Where ε is a hyperparameter, C represents the number of categories; for the generated image x fake According to formula (2), the category prediction value q is obtained i , the category prediction loss l of the generated image SL-CE It can be expressed as: Step 5: Triplet loss construction: Input the feature vector learned by the feature extraction network in step 3 of the discriminator into the fully connected layer FC to obtain the output f of the triple feature learning subnetwork triplets , using triplet loss to optimize triplet features and learn class-related distinguishable features; specifically including: Assume x a is the real image obtained by random sampling, x p For x a Real images belonging to the same category, x n For x a Real images belonging to different categories, f(x) is the feature vector obtained after the real image passes through the feature extraction network and the triple feature learning subnetwork, then select the input image in f(x) that does not meet the following distance constraints to construct the triple sample <x a ,x p ,x n >: d(f(x a ),f(x p ))+α<d(f(x a ),f(x n ))(8) Where d(·) represents the Euclidean distance and α is the classification interval; For samples that satisfy the constraints of formula (8), the loss value is 0. For triplet samples that do not satisfy formula (8), the loss value is x a With x p and x n The distance difference of the feature vector learned by f(x), that is, the triple loss l triplets : l triplets =d(f(x a ),f(x p ))-d(f(x a ),f(x n ))(9) Step 6: Generator and discriminator loss function construction: Based on the generator and discriminator networks constructed in steps 2 and 3 and the loss functions constructed in steps 4 and 5, the overall loss functions of the generator G and the discriminator D are constructed respectively, and the constructed overall loss functions are used to alternately optimize the generator and the discriminator respectively; specifically, the following steps are performed: First, for the true-false prediction subnetwork in the discriminator D, the true-false prediction loss l is calculated using the binary cross entropy loss function BCE To distinguish the real image x real and generate image x fake , which can be expressed as: l BCE =-(y v *log(p(v))+(1-y v )*log(1-p(v)))(10) where y v Represents the true or false binary classification label. When the input image is a real image, y v = 0, when the input image is the generated image y v =1; p(v) is the true or false prediction value of the input image calculated by formula (3); Then, construct the total loss function L of the generator G G ; This function is composed of the true and false prediction loss l BCE And the category prediction loss l of the generated image SL-CE It consists of two parts, and its formula is as follows: in represents the smooth label of the generated image calculated according to formula (6); 1 , 2 represents the trade-off parameter; Finally, the total loss function of the discriminator D is constructed; the true and false prediction loss l is expressed as BCE , the category prediction loss l of the generated image described in formula (7) SL-CE , the category prediction loss l of the real image described in formula (5) CE And the triplet loss l described in formula (9) triplets Optimize; the total loss function L of the discriminator D D The formula is as follows: where x real 、x fake Represent the real image and the generated image respectively, y real represents the hard label of the real image, represents the smoothed label of the generated image, y v is the true or false binary classification label described in formula (10), x a ,x p ,x n is the triplet sample constructed by formula (8), λ i (i=3,4,···,7) represents the trade-off parameter; Step 7: Model training and testing: The model constructed from step 2 to step 6 is trained. After the training is completed, the discriminator is used to classify the input test samples.
2. According to the method for generating and classifying ship targets in SAR images as described in claim 1, Features In step 7 of model training and testing: Using Adam as the optimizer, train the model constructed from steps 2 to 6 according to the following steps: ① Randomly generate b vectors according to formula (1) and input them into generator G to obtain b generated images; ② The b generated images are mixed with the b real images and then input into the discriminator D. The losses are calculated according to formula (12) and the weights of the discriminator D are updated. ③ Calculate the generator loss and update the generator G weight according to formula (11); ④ Repeat the process from ① to ③ until the training times are reached; After the training is completed, the discriminator D in the model designed in steps 2-6 is retained, the test sample is input into the discriminator D, the prediction result of the category prediction subnetwork is output, and the sample category is obtained.
