High-resolution SAR image scene classification method based on multi-prototype learning

Abstract

This invention discloses a high-resolution SAR image scene classification method based on multi-prototype learning. Firstly, addressing the problem of limited available SAR images and a scarcity of high-quality labeled SAR images, this invention introduces a few-shot learning method, achieving good performance even with a small dataset. Furthermore, this invention proposes a multi-prototype learning method. Since a single prototype for each category cannot adequately represent the category, multiple prototypes for each category are generated through a support set partitioned by few-shot learning to guide the classification of the query set. The dataset used in this invention is a small-sample dataset for SAR image scene classification. Experiments show that the proposed high-resolution SAR image scene classification method based on multi-prototype learning improves the accuracy of high-resolution SAR image scene classification and effectively overcomes interference from speckle noise.

Description

Technical Field

[0001] This invention belongs to the field of mechanical manufacturing equipment, specifically relating to a high-resolution SAR image scene classification method based on multi-prototype learning. Background Technology

[0002] Synthetic Aperture Radar (SAR) is an all-weather, all-time, long-range, active Earth observation system widely used in various military and civilian fields, including automatic target identification, crop growth prediction, vegetation monitoring, natural disaster risk assessment, and marine environmental monitoring. SAR imaging technology has evolved from early low spatiotemporal resolution, single-band, and single-polarization methods to high spatiotemporal resolution, multi-polarization, multi-band, multi-mode, interference, and polarimetric interferometry and tomographic imaging. This has led to a leap forward in the ability to observe ground targets, greatly enriching observation methods and data. Given the massive amounts of high-resolution SAR images acquired, there is an urgent need to develop SAR image interpretation techniques to extract knowledge from the images and provide theoretical support for various important applications. Scene classification is one of the important directions in SAR image interpretation and is widely used in urban planning, ship identification, marine pollution monitoring, and Earth resource surveying. Because SAR is an active echo imaging system, it has the characteristics of long imaging distance and complex imaging structure, which leads to severe speckle noise in SAR images, making it extremely difficult to effectively distinguish between different scenes.

[0003] The two key steps in SAR image scene classification technology are: 1. Extracting key features representing the image from the SAR image. 2. Designing a suitable classifier based on the learned key features. The step of extracting key image features is particularly important. Early traditional methods mainly relied on feature extraction methods based on low-level handcrafted features, including statistical features, texture features, and structural features, primarily using methods such as Scale Invariant Feature Transform (SIFT). Subsequently, methods for extracting mid-level features emerged, encoding the obtained low-level features to obtain more discriminative mid-level features, primarily using methods such as BagOf Word (BOW). With the development of deep learning, deep neural networks can adaptively learn more discriminative high-level image features and adopt an end-to-end solution, thus overcoming the limitations of previously requiring manually designed features. Commonly used deep neural networks include Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Belief Networks (DBN), and Generative Adversarial Networks (GAN).

[0004] Deep neural networks require a large number of high-quality labeled images to achieve ideal training results. Since SAR image labeling requires more expert knowledge, and the results of different people labeling images vary, high-quality labeled SAR images are very scarce. Therefore, existing deep learning methods cannot be directly applied to SAR image scene classification. Thus, current technology needs a SAR image scene classification method to address the problem of limited high-quality labeled samples and to overcome speckle noise to extract more discriminative features. Summary of the Invention

[0005] The purpose of this invention is to provide a high-resolution SAR image scene classification method based on multi-prototype learning. It utilizes few-shot learning to solve the problem of insufficient labeled samples in SAR images, and combines multi-prototype learning to achieve SAR image scene classification. This method has fewer model parameters, which can greatly save resources and has high classification accuracy.

[0006] The technical solution adopted in this invention is a high-resolution SAR image scene classification method based on multi-prototype learning, and the specific operation steps are as follows:

[0007] Step 1: Obtain a small sample dataset for SAR image scene classification, and divide the dataset into training set, validation set, and test set according to the categories.

[0008] Step 2: Randomly select multiple small sample data blocks from the training set according to the category as the support set, and randomly select multiple sample data from the remaining samples of the support set class as the query set. Preprocess the images of the support set and the query set.

[0009] Step 3: Extract features from multiple small sample data blocks of the preprocessed support set using a feature extractor. Obtain the prototype of each class using all features of each class in each data block, and then integrate the multiple prototypes from multiple data blocks.

[0010] Step 4: Extract features from the query set image, then project the extracted query set features onto the multi-prototype of each class, and classify the data based on the distance from the query set features to the multi-prototype projection.

[0011] The invention is further characterized in that,

[0012] In step 1, a small sample dataset of SAR image scene classification is obtained. The images in the dataset are single-channel grayscale images, and the dataset is divided into 12 classes, including aquaculture farms, barren land, densely built-up areas, and farmland. This dataset is then divided into three classes: a training set (6 classes), a validation set (3 classes), and a test set (3 classes). The classes in the training, validation, and test sets are mutually exclusive. Each image has its own corresponding class label. Each image is 256×256 pixels in size.

[0013] In step 2, following the N-way k-shot learning setup, N classes are randomly selected from the training set. Then, k samples are randomly selected from each of these N classes. These N×k samples form a data block of the support set. In the same way, samples are randomly selected from these N classes to form other data blocks. These multiple data blocks form the support set. Finally, samples are randomly selected from the remaining samples in these N classes to form the query set. A set of support and query sets constitutes a training task. Combinations of different categories allow the model to learn the commonalities and differences between different categories, making the model easier to generalize. After obtaining the support and query sets, the samples in the support and query sets are preprocessed. The image size is scaled to 84×84. Experiments show that this speeds up the process, saves resources, and does not affect the accuracy of the experiments. Then, the images are standardized. Since the images are single-channel grayscale images, the mean is set to 0.485 and the variance to 0.229.

[0014] In step 3, the feature extractor of this invention consists of four convolutional blocks. An 84×84 support set image is input into the feature extractor. The first convolutional block has 1 input channel and 64 output channels. It first passes through a convolutional layer with a kernel size of 3×3 and padding of 1, then through a block normalization layer to normalize the data. This prevents the large data size from affecting network performance before ReLU. Next, it passes through a ReLU activation function layer, and finally, it undergoes max pooling with a 2×2 kernel. The remaining three convolutional blocks are identical to the first convolutional block except for the 64 input channels. After four convolutional layers, the output is 5×5 with 64 channels. Finally, the output is flattened into a single line.

[0015] When the support set has its features extracted by the feature extractor, the prototype of each class in each data block is obtained by averaging according to formula (1). The prototype is meant to represent each class. Then, the prototypes of the same class in different data blocks are integrated together to obtain the multiple prototypes of each class in these N classes. The formula for calculating the prototype is defined as:

[0016]

[0017] Where, ρ c x represents the prototype of class c. i X is the i-th sample of class c. c Let f represent the c-th class of samples. θ Here, k represents the feature extractor, and k represents the total number of samples in class c. The goal is to distinguish the query set using multiple prototypes across these N classes. Because a single prototype may not adequately represent a category, this patent employs multiple prototypes to represent a category, making it more representative.

[0018] In step 4, the query set image also undergoes preprocessing, scaling it to 84×84 pixels and then standardizing it. Since the image is a single-channel grayscale image, the mean is set to 0.485 and the variance to 0.229. Unlike the support set, the query set is not composed of multiple data blocks. Instead, it consists of multiple randomly selected samples from the training set, excluding the support set samples. Multiple samples are randomly selected from each of the N categories to form the query set, ensuring that the samples in the query set and support set do not overlap.

[0019] After preprocessing, the query set is used to extract features from the query set using a feature extractor. The support set and the query set share the same feature extractor. Then, according to formula (2), the obtained query sample features are projected onto the multi-prototypes of these N categories. The projection formula from the query sample features to the multi-prototypes is defined as:

[0020]

[0021] y i，c Is it a query sample q? i In multi-prototype p c The projection on, μ c It is the mean of all samples in class c. Next, according to formula (3), the distance between the projection of the query feature onto these N class prototypes is calculated, and the distance from the query image to the prototype p is calculated. c The distance formula is defined as:

[0022] d i，c =||f θ (q i )-μ c -p c y i，c || (3)

[0023] Finally, the probability of the query sample belonging to each of the N categories is obtained using the softmax function in formula (4). i The probability formula for assigning a value to class c is defined as follows:

[0024]

[0025] The category with the highest probability is determined as the classification result.

[0026] The beneficial effects of this invention are as follows: This invention proposes a high-resolution SAR image scene classification method based on multi-prototype learning. First, addressing the problem of limited available SAR images and a scarcity of high-quality labeled SAR images, this invention introduces a few-shot learning method, achieving good performance even with a small dataset. Furthermore, this invention proposes a multi-prototype learning method. Since a single prototype for each category cannot adequately represent the category, multiple prototypes for each category are generated through a support set partitioned by few-shot learning to guide the classification of the query set. The dataset used in this invention is a small-sample dataset for SAR image scene classification. Experiments show that the high-resolution SAR image scene classification method based on multi-prototype learning proposed in this invention improves the accuracy of high-resolution SAR image scene classification and effectively overcomes interference from speckle noise. Attached Figure Description

[0027] Figure 1 This is a diagram illustrating the training process of a high-resolution SAR image scene classification method based on multi-prototype learning disclosed in this invention.

[0028] Figure 2 This is a network structure diagram of a high-resolution SAR image scene classification method based on multi-prototype learning disclosed in this invention.

[0029] Figure 3 This is a feature extractor structure diagram of a high-resolution SAR image scene classification method based on multi-prototype learning disclosed in this invention. Detailed Implementation

[0030] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0031] The present invention provides a high-resolution SAR image scene classification method based on multi-prototype learning, the specific operation steps of which are as follows:

[0032] Step 1: Obtain a small sample dataset for SAR image scene classification. The dataset consists of single-channel grayscale images and includes 12 classes such as aquaculture farms, barren land, densely populated areas, and farmland. This dataset contains 49,483 images. It is then divided into three classes: a training set (6 classes), a validation set (3 classes), and a test set (3 classes). The training, validation, and test sets do not overlap. The training set contains 28,091 images, the validation set contains 10,332 images, and the test set contains 11,060 images. Each image has its own corresponding class label. Each image is 256×256 pixels in size.

[0033] Step 2, as follows Figure 2Following the N-wayk-shot learning approach, N classes are randomly selected from the training set. Then, k samples are randomly selected from each of these N classes. These N×k samples form a data block of the support set. This process is repeated to randomly select samples from these N classes to form other data blocks, which together form the support set. Finally, samples are randomly selected from the remaining samples in these N classes to form the query set. A set of support and query sets constitutes a training task. Combinations of different classes allow the model to learn the commonalities and differences between them, making the model easier to generalize. After obtaining the support and query sets, preprocessing is performed on the samples. The images are scaled to 84×84 pixels. Experiments show that this speeds up the process, saves resources, and does not affect the accuracy. Then, the images are standardized. Since the images are single-channel grayscale images, the mean is set to 0.485 and the variance to 0.229.

[0034] Step 3, as follows Figure 3 The feature extractor of this invention consists of four convolutional blocks. An 84×84 support set image is input into the feature extractor. The first convolutional block has 1 input channel and 64 output channels. It first passes through a convolutional layer with a kernel size of 3×3 and padding of 1, then through a block normalization layer to normalize the data. This prevents the large data size from affecting network performance before ReLU. Next, it passes through a ReLU activation function layer, and finally, it undergoes max pooling with a 2×2 kernel. The remaining three convolutional blocks are identical to the first convolutional block except for the 64 input channels. After four convolutional layers, the output is 5×5 with 64 channels. Finally, the output is flattened into a single line.

[0035] like Figure 2 When the support set is processed by the feature extractor, the average value of each class in each data block is calculated according to formula (1) to obtain multiple prototypes for each class. The prototypes represent each class. Then, multiple prototypes of the same class in different data blocks are integrated together to obtain the multi-prototype matrix of each class in these N classes. The formula for calculating the prototype is defined as:

[0036]

[0037] Where, ρ c x represents the prototype of class c. i X is the i-th sample of class c. c Let f represent the c-th class of samples. θHere, k represents the feature extractor, and k represents the total number of samples in class c. The goal is to distinguish the query set using multiple prototypes across these N classes. Because a single prototype may not adequately represent a category, this patent employs multiple prototypes to represent a category, making it more representative.

[0038] Step 4: The query set image also undergoes preprocessing. The image size is scaled to 84×84, and then standardized. Since the image is a single-channel grayscale image, the mean is set to 0.485 and the variance to 0.229. Unlike the support set, the query set is not composed of multiple data blocks. Instead, it consists of multiple randomly selected samples from the training set, excluding the support set samples. Multiple samples are randomly selected from each of the N categories to form the query set, ensuring that the query set and support set samples do not overlap.

[0039] like Figure 2 After preprocessing, the query set is used to extract features from the query set using a feature extractor. The support set and the query set share the same feature extractor. Then, according to formula (2), the obtained query set sample features are projected onto the multi-prototypes of these N categories. The projection formula from the query sample features to the multi-prototypes is defined as:

[0040]

[0041] Among them, y i，c Is the query set sample q i In multi-prototype p c The projection on, μ c It is the mean of all samples in class c. Next, according to formula (3), the distance between the projection of the query feature onto these N class prototypes is calculated, and the distance from the query image to the prototype p is calculated. c The distance formula is defined as:

[0042] d i，c =||f θ (q i )-μ c -p c y i，c || (3)

[0043] Finally, the probability of the query sample belonging to each of the N categories is obtained using the softmax function in formula (4). i The probability formula for assigning a value to class c is defined as follows:

[0044]

[0045] The category with the highest probability is determined as the classification result.

[0046] The loss function used in this invention is the cross-entropy loss function. After the classification probability of the query set is obtained, the cross-entropy loss function is used to calculate the loss on the classification result of the query sample, and then the network parameters are updated through backpropagation. The learner uses the Adam learner, with an initial learning rate defined as 0.001, which is updated to half of its original value every 50 epochs.

[0047] The validation phase is performed every 10 epochs of training. The dataset used in the validation phase is the pre-defined validation set, which includes three classes and contains 10,332 images and their corresponding labels. Before validation begins, `model.eval()` is used to adjust the model to validation mode. This patent sets up a weight file to be saved every 50 epochs during training, named "current epoch".pth. Additionally, this patent sets up a weight file `max-acc.pth`. When the accuracy of the current epoch is higher than the previous highest accuracy during validation, the weights of the current epoch are updated to `max-acc.pth`, ensuring that the weight parameters saved in `max-acc.pth` represent the optimal parameters obtained during training.

[0048] After completing the training and validation phases, this patent will proceed to the testing phase. The dataset used in the testing phase is the test set that was pre-defined before training. The test set has three classes and contains a total of 11,060 images and their corresponding class labels. The testing phase is conducted in the same way as the training and validation phases. A total of 600 sets of support and query sets are randomly selected for testing. First, the previously trained max-acc.pth weight file is loaded onto the model, and then model.eval() is used to adjust the model to test mode. Then, as described above, the support set samples are preprocessed, and features are extracted to generate multiple prototypes for each class according to formula (1). After preprocessing the query set samples, features are extracted and then projected onto the multiple prototypes of each class according to formula (2). Then, the distance from the query features to the multiple prototypes of each class is calculated using formula (3). Finally, the probability value of the query sample belonging to each class is obtained using formula (4). After the test is completed, the average accuracy of these 600 sets of support and query sets is calculated and output.

[0049] In summary, the proposed scheme can effectively improve the classification accuracy of high-resolution SAR image scene classification. Furthermore, since a single prototype for each category cannot adequately represent that category (as some features may be missed), using multiple prototypes for each category provides more comprehensive feature coverage and achieves higher classification accuracy than a single prototype. Additionally, the model of this invention has fewer parameters, saving significant resources during training and allowing for faster training, thus saving time.

Claims

1. A high-resolution SAR image scene classification method based on multi-prototype learning, characterized in that, The specific operating steps are as follows: Step 1: Obtain a small sample dataset for SAR image scene classification, and divide the dataset into training set, validation set, and test set according to the categories; Step 2: Randomly select multiple small sample data blocks from the training set according to the category as the support set, and randomly select multiple sample data from the remaining samples of the support set class as the query set. Preprocess the images of the support set and the query set. In step 2, following the N-way k-shot learning setup, N classes are randomly selected from the training set. Then, k samples are randomly selected from each of these N classes. These N×k samples form a data block of the support set. In the same way, samples are randomly selected from these N classes to form other data blocks. These multiple data blocks form the support set. Then, samples are randomly selected from the remaining samples in these N classes to form the query set. A set of support set and query set constitutes a training task. Next, the support set and query set samples are preprocessed by scaling the image size to 84×84 and then standardizing the image. Since the image is a single-channel grayscale image, the mean is set to 0.485 and the variance is set to 0.

229. Step 3: Extract features from multiple small sample data blocks of the preprocessed support set using a feature extractor. Obtain the prototype of each class using all features of each class in each data block, and then integrate the multiple prototypes of multiple data blocks. The feature extractor described in step 3 consists of four convolutional blocks. An 84×84 support set image is input into the feature extractor. The first convolutional block has 1 input channel and 64 output channels. It first passes through a convolutional layer with a kernel size of 3×3 and padding of 1, then through a block normalization layer to normalize the data. Next, it passes through a ReLU activation function layer, and finally performs max pooling with a 2×2 kernel. The remaining three convolutional blocks are identical to the first convolutional block except for the 64 input channels. After four convolutional layers, the output is a 5×5 feature map with 64 channels. Finally, the output feature map is flattened into a single row. When the support set has its features extracted by the feature extractor, the prototype of each class in each data block is obtained by averaging according to formula (1). The prototype exists as a representative of each class. Then, the prototypes of the same class in different data blocks are integrated together to obtain the multiple prototypes of each class in these N classes. The formula for calculating the prototype is defined as: (1) in, Represents the prototype of class c. It is the i-th sample of class c. Represents the c-th class of samples, The feature extractor is represented by k, which represents the total number of samples in class c. Step 4: Extract features from the query set image, then project the extracted query set features onto the multi-prototype of each class, and classify the data based on the distance from the query set features to the multi-prototype projection. In step 4, the query set image is also preprocessed: the image size is scaled to 84×84, and then the image is standardized. Since the image is a single-channel grayscale image, the mean is set to 0.485 and the variance is set to 0.

229. After preprocessing, the query set is used to extract features from the query set using a feature extractor. The support set and the query set share the same feature extractor. Then, according to formula (2), the obtained query sample features are projected onto the multi-prototypes of these N categories. The projection formula from the query sample features to the multi-prototypes is defined as: (2) This is a query sample In multiple prototypes The projection on It is the mean of all samples in class c; next, according to formula (3), the distance between the query feature and the projection onto these N class prototypes is calculated, and the distance between the query image and the prototype is calculated. The distance formula is defined as: (3) Finally, the probability of the query sample belonging to each of the N categories is obtained using the softmax function in formula (4). The probability formula for assigning a value to class c is defined as follows: (4) The category with the highest probability is determined as the classification result.

2. The high-resolution SAR image scene classification method based on multi-prototype learning according to claim 1, characterized in that, In step 1, a small sample dataset for SAR image scene classification is obtained. The images in the dataset are single-channel grayscale images, and the dataset has 12 classes. The dataset is divided into 6 training classes, 3 validation classes, and 3 test classes. The training, validation, and test classes do not overlap. Each image has its own corresponding class label, and each image is 256×256 pixels in size.

Images