A label smoothing-based method for scene classification and semantic segmentation tasks in remote sensing images

Abstract

The present invention relates to the field of remote sensing image processing, and in particular to a method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing, comprising the following steps: preprocessing remote sensing image scene classification and semantic segmentation datasets; performing label smoothing on training data labels based on prior statistical information, and introducing category distribution information to guide network training; constructing a deep neural network and pre-training it on a large dataset, and then feeding the label-smoothed remote sensing image training dataset into the network for fine-tuning. The present invention performs label smoothing on data labels based on prior statistical information, incorporates prior knowledge of category statistical distribution into network training, and reduces the weight of positive samples when calculating network loss, thereby alleviating the problem of uneven sample category distribution. The present invention can effectively improve the classification performance and generalization of remote sensing image scene classification and semantic segmentation network models.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing. Background Art

[0002] Label smoothing is a common regularization strategy used in deep learning network training. It has been proven to be an effective method for preventing overfitting in classification tasks, thereby improving the accuracy of image classification and segmentation. When training a neural network, it tends to learn in the direction that maximizes the difference between the correct and incorrect labels. When the training data is insufficient to represent all sample features, this can lead to overfitting. Label smoothing methods smooth one-hot labels by adding noise, reducing the weight of the true sample label category in the loss function calculation, ultimately preventing overfitting.

[0003] Remote sensing refers to non-contact, long-distance detection technology. It generally refers to the use of sensors / remote sensors to detect the radiation and reflection characteristics of electromagnetic waves from objects. Remote sensing is a science and technology that uses electromagnetic-sensitive instruments, such as remote sensors, to detect target objects at a distance and without contact with the target object, obtaining information about the electromagnetic waves reflected, radiated, or scattered by the objects (such as electric fields, magnetic fields, electromagnetic waves, and seismic waves), and then extracting, identifying, processing, analyzing, and applying them. With the continuous advancement of remote sensing technology, the acquisition of remote sensing images has become increasingly easier, and a growing number of public remote sensing image datasets have emerged. Remote sensing images are films or photographs that record the electromagnetic characteristics of various objects, and are primarily categorized as aerial and satellite images. Compared to natural images, remote sensing images contain more features such as color, texture, and shape, and also suffer from complex backgrounds and a wide variety of image types. Most of these public datasets suffer from large variations within the same category and high similarity across different categories. This results in low class discrimination in existing scene classification and semantic segmentation methods for remote sensing images, posing certain challenges.

[0004] Remote sensing image scene classification involves classifying a given remote sensing image into a set of predefined semantic categories based on the information it contains. Remote sensing image semantic segmentation involves classifying each pixel in the image into a set of predefined semantic categories based on the information it contains. In recent years, remote sensing image scene classification and semantic segmentation have become increasingly researched due to the demands of practical applications such as urban planning, natural disaster detection, environmental regulation, geographic mapping, and geospatial object detection.

[0005] Unlike the classification and segmentation tasks of natural images, remote sensing images have richer texture and color feature information. At the same time, there are also problems such as complex backgrounds and numerous types. Therefore, there are certain challenges for the classification and segmentation of remote sensing images. There is no good method based on label smoothing for remote sensing image scene classification and semantic segmentation tasks in the existing technology. Summary of the Invention

[0006] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing. Starting from the characteristics of remote sensing images themselves, such as large intra-class differences and small inter-class differences, a remote sensing image scene classification and semantic segmentation label smoothing method based on prior statistical information is proposed to improve the classification performance of the network model.

[0007] The method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing includes the following steps:

[0008] Divide the training set and test set: Construct a high-resolution remote sensing scene image sample dataset of various scenes, and divide the high-resolution remote sensing scene image sample dataset into a training set and a test set according to a preset ratio;

[0009] Data preprocessing: Preprocess the data in the training set and test set separately;

[0010] Label smoothing: Label smoothing is performed on the preprocessed training set data based on prior statistical information, and label smoothing is performed on the remote sensing image scene classification training data and remote sensing image semantic segmentation data in the training set respectively;

[0011] Neural network model: Build a neural network model and train it based on the label-smoothed data;

[0012] Model testing: Input the sample data in the test set into the trained neural network model to obtain the predicted category of the test set images.

[0013] The present invention fully considers the characteristics of remote sensing images themselves, that is, large intra-class differences and small inter-class differences, through a label smoothing method based on prior statistics, and smoothes the training sample labels, thereby improving the generalization ability of the model.

[0014] Preferably, the dividing of the training set and the test set comprises the following steps:

[0015] Construct a remote sensing scene image sample dataset and create corresponding sample labels;

[0016] The remote sensing scene image sample dataset is divided into a training set and a test set. For each type of remote sensing image in the remote sensing scene image sample dataset, the total number of remote sensing images is set to n, and m images are randomly selected from them to construct the training set, and the remaining nm images are used to construct the test set.

[0017] Preferably, the data preprocessing includes: performing noise reduction, random erasing and random rotation processing on the training set and test set samples; and performing scaling processing on the training set samples.

[0018] Preferably, the calculation method of the scaling process is as follows:

[0019] For an input image x of scale s×s, at the beginning of each training iteration, it is scaled to scale d×d, where d is calculated with respect to s as follows:

[0020]

[0021] Where t is the current training iteration cycle, and iters is the total number of iterations experienced during a round of training.

[0022] Preferably, the calculation method for label smoothing of remote sensing image scene classification training data is as follows: calculate the category prior statistical results of each batch of remote sensing image scene classification training data in the training set, multiply the category prior statistical results by the label smoothing factor, and then add them to the corresponding original labels to obtain the classification labels after smoothing the category prior statistical information.

[0023] Preferably, the calculation steps for label smoothing of the remote sensing scene classification training data are as follows:

[0024] A. Statistical results of the priori category of small batch remote sensing image scene classification training data labels:

[0025]

[0026] Where: c is the total number of all categories in the entire dataset; batch_size is the number of data in the batch input data; is the number of data of class i in the batch of data, where 1≤i≤c;

[0027] B. Define a smoothing factor α, where 0≤α≤1; add the category prior statistics to the smoothing factor to obtain the smoothing factor α′ added to the category prior statistics:

[0028]

[0029] The original label l for remote sensing scene classification training data n Perform label smoothing, where 1≤n≤batch size; get smooth labels

[0030]

[0031] C. Introducing the smoothing factor α′ into the smoothed label Get the classification label:

[0032]

[0033] Where: l′ n Where n is greater than or equal to 1 and less than or equal to batch_size; is the preliminary smoothed label of the nth data in the mini-batch data.

[0034] Preferably, a method for label smoothing of remote sensing image semantic segmentation data is as follows: a label-smoothing moving window is designed for each training image data in the remote sensing image semantic segmentation data, the prior statistical category distribution vector of the category in the corresponding label image window is calculated at each position where the moving window slides, and the label containing the prior distribution information is smoothed using a smoothing factor to obtain a smoothed classification label.

[0035] Preferably, the calculation steps for label smoothing of remote sensing image semantic segmentation data are as follows:

[0036] A. Set the side length of the square sliding window to sindow_size. Each time the window is slid, the statistical distribution of the labels in the sliding window is counted to obtain the prior statistical category distribution vector, which represents the statistical distribution of the labels in each category in the current window for the semantic segmentation training data with c semantic category labels:

[0037]

[0038] Where: Indicates the number of pixels belonging to class i in the current window, where i is greater than or equal to 1 and less than or equal to c; window_size 2 Indicates the total number of pixels contained in the rectangular window;

[0039] B. Define a smoothing factor α, where α is greater than or equal to 0 and less than or equal to 1; smooth the label containing the prior statistical category distribution vector and calculate the smoothing factor α′;

[0040]

[0041] The original label l of the data in the training set n Perform label smoothing, where l n where n is greater than or equal to 1 and less than or equal to window_size 2; Get the smooth label corresponding to each pixel

[0042]

[0043] Where: e pq It is the label component corresponding to the pixel with spatial coordinates [p,q] in the window.

[0044] C. Mapping the smoothing factor α′ into the smoothing label Get the classification label l′ of each pixel based on the prior statistical label smoothing n :

[0045]

[0046] Where: is the preliminary smoothed semantic label of the n-th pixel in the corresponding window, l′ n where n is greater than or equal to 1 and less than or equal to window_size 2 .

[0047] Preferably, establishing the neural network model also includes setting hyperparameters of the neural network model, wherein the hyperparameters include parameter initialization, training batch, scaling factor, learning rate, optimization method, number of iterations and window size of the sliding window.

[0048] Preferably, the constructed neural network model is pre-trained based on the ImageNet data set, and the model is trained by minimizing the loss function that measures the sample label and the network prediction value, and the parameters of the applied network model are updated until the network parameter values ​​converge. The condition for convergence is that the value of the current loss function no longer decreases compared with the value calculated by the previous training, and a pre-trained neural network model is obtained. The pre-trained neural network model is then trained with the training set to obtain a trained neural network model.

[0049] The beneficial effects of the present invention include:

[0050] 1. Compared with traditional label smoothing methods, the label smoothing method based on prior statistics fully considers the characteristics of remote sensing images that have large intra-class differences and small inter-class differences, and smoothes the training sample labels, thereby improving the generalization ability of the model.

[0051] 2. This paper improves upon the traditional classification method of manually extracting image features by employing a deep neural network to automatically extract deep features. Because remote sensing image data acquisition is relatively difficult, the backbone network requires pre-training to enhance its feature extraction capabilities. To mitigate overfitting in deep neural networks, a label smoothing method based on prior statistics is designed to improve classification and segmentation accuracy.

[0052] 3. The present invention smoothes data labels based on prior statistical information, incorporates prior knowledge of category statistical distribution into network training, and reduces the weight of positive samples when calculating network loss, thereby alleviating the problem of uneven distribution of sample categories; it can effectively improve the classification performance and generalization of remote sensing image scene classification and semantic segmentation network models. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a training flow chart of the present invention.

[0054] Figure 2 This is a test flow chart of the present invention.

[0055] Figure 3 Schematic diagram of the remote sensing image scene classification label smoothing calculation method of the present invention.

[0056] Figure 4 Schematic diagram of the remote sensing image semantic segmentation label smoothing calculation method of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0058] The following is combined with Figure 1 To the attached Figure 4 Describe the embodiments of the present invention in detail;

[0059] The method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing includes the following steps:

[0060] Divide the training set and test set: Construct a high-resolution remote sensing scene image sample dataset of various scenes, and divide the high-resolution remote sensing scene image sample dataset into a training set and a test set according to a preset ratio; the training set and the test set are divided according to two ratios of 4:1 and 1:1.

[0061] The division of the training set and the test set comprises the following steps:

[0062] Construct a remote sensing scene image sample dataset and create corresponding sample labels;

[0063] The remote sensing scene image sample dataset is divided into a training set and a test set. For each type of remote sensing image in the remote sensing scene image sample dataset, the total number of remote sensing images is set to n, and m images are randomly selected from them to construct the training set, and the remaining nm images are used to construct the test set.

[0064] Data preprocessing: Preprocess the data in the training set and test set separately;

[0065] The data preprocessing includes: performing noise reduction and rotation processing on the training set and test set samples; and performing scaling processing on the training set samples.

[0066] The calculation method of the scaling process is as follows:

[0067] Due to factors such as remote sensing imaging angle, perspective, and equipment, remote sensing images often have the characteristic of drastic scale changes. To enhance the robustness of the remote sensing scene classification network to the scale changes of ground targets and extract scale-invariant features, a dynamic scale network training strategy is adopted. The image is scaled according to certain rules before being fed into the network: for an input image x of scale s×s, at the beginning of each training iteration, it is scaled to scale d×d. The calculation formula of d with respect to s is as follows:

[0068]

[0069] Where t is the current training iteration cycle, and iters is the total number of iterations experienced during a round of training.

[0070] Label smoothing: Label smoothing is performed on the preprocessed training set data based on prior statistical information, and label smoothing is performed on the remote sensing image scene classification training data and remote sensing image semantic segmentation data in the training set respectively;

[0071] Based on the obtained pre-trained model, train and fine-tune it on the remote sensing image dataset for 22 epochs. After 10 epochs, reduce the learning rate to 0.1 times the original value. After 20 epochs, reduce it again by 0.1 times until the training is completed, and save the network.

[0072] The present invention sets a label smoothing method based on prior statistics to smooth the labels of the training set data based on prior statistics; specifically, the computational efficiency is improved by dividing the training into small batches, and the prior information is introduced into the supervisory information to guide the learning of the neural network, and the obtained supervisory information is used for transfer learning.

[0073] The calculation method for label smoothing of remote sensing image scene classification training data is as follows: calculate the category prior statistical results of each batch of remote sensing image scene classification training data in the training set, multiply the category prior statistical results by the label smoothing factor, and then add them to the corresponding original labels to obtain the classification labels smoothed by the category prior statistical information.

[0074] The calculation steps for label smoothing of the remote sensing scene classification training data are as follows:

[0075] A. Statistical results of the priori category of small batch remote sensing image scene classification training data labels:

[0076]

[0077] Where: c is the total number of all categories in the entire dataset; batch_size is the number of data in the batch input data; is the number of data of class i in the batch of data, where 1≤i≤c;

[0078] B. Define a smoothing factor α, where 0≤α≤1; add the category prior statistics to the smoothing factor to obtain the smoothing factor α′ added to the category prior statistics:

[0079]

[0080] The original label l for remote sensing scene classification training data n Perform label smoothing, where 1≤n≤batch s ize; get smooth labels

[0081]

[0082] C. Introducing the smoothing factor α′ into the smoothed label Get the classification label:

[0083]

[0084] Where: l′ n Where n is greater than or equal to 1 and less than or equal to batch_size; is the preliminary smoothed label of the nth data in the mini-batch data.

[0085] A method for label smoothing of remote sensing image semantic segmentation data: a label smoothing moving window is designed for each training image data in the remote sensing image semantic segmentation data, the prior statistical category distribution vector of the category in the corresponding label image window is calculated at each position where the moving window slides, and the label containing the prior distribution information is smoothed using a smoothing factor to obtain the smoothed classification label.

[0086] The calculation steps for label smoothing of remote sensing image semantic segmentation data are as follows:

[0087] A. Set the side length of the square sliding window to window_size. Each time the window slides, the statistical distribution of the labels in the sliding window is counted to obtain the prior statistical category distribution vector, which represents the statistical distribution of the labels in each category in the current window for the semantic segmentation training data with c semantic category labels:

[0088]

[0089] Where: Indicates the number of pixels belonging to class i in the current window, where i is greater than or equal to 1 and less than or equal to c; window_size 2 Indicates the total number of pixels contained in the rectangular window;

[0090] B. Define a smoothing factor α, where α is greater than or equal to 0 and less than or equal to 1; smooth the label containing the prior statistical category distribution vector and calculate the smoothing factor α′;

[0091]

[0092] The original label l of the data in the training set n Perform label smoothing, where l n where n is greater than or equal to 1 and less than or equal to window_size 2 ; Get the smooth label corresponding to each pixel

[0093]

[0094] Where: e pq It is the label component corresponding to the pixel with spatial coordinates [p,q] in the window.

[0095] C. Mapping the smoothing factor α′ into the smoothing label Get the classification label l′ of each pixel based on the prior statistical label smoothing n :

[0096]

[0097] Where: is the preliminary smoothed semantic label of the n-th pixel in the corresponding window, l′ n where n is greater than or equal to 1 and less than or equal to window_size 2 .

[0098] Neural network model: Build a neural network model and train it based on the label-smoothed data;

[0099] Establishing the neural network model also includes setting hyperparameters of the neural network model, wherein the hyperparameters include parameter initialization, training batch, scaling factor, learning rate, optimization method, number of iterations, and window size of the sliding window.

[0100] The constructed neural network model is pre-trained based on the ImageNet dataset. The model is trained by minimizing the loss function that measures the sample label and the network prediction value, and the parameters of the network model are updated until the network parameter values ​​converge. The condition for convergence is that the value of the current loss function no longer decreases compared with the value calculated by the previous training. A pre-trained neural network model is obtained, and then the pre-trained neural network model is trained using the training set to obtain a trained neural network model.

[0101] Model testing: Use the trained remote sensing image scene classifier and segmentation network to perform scene classification and semantic segmentation on the remote sensing images in the corresponding test sample set, output the probability of each category of scene classification and segmentation result map, and obtain the classification results and segmentation results.

[0102] The present invention fully considers the characteristics of remote sensing images themselves, that is, large intra-class differences and small inter-class differences, through a label smoothing method based on prior statistics, and smoothes the training sample labels, thereby improving the generalization ability of the model.

[0103] The above-described embodiments merely represent specific implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the technical concept of the present application, and all such variations and improvements fall within the scope of protection of the present application.

Claims

1. A method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing, characterized by: The following steps are involved: Divide the training set and test set: Construct a high-resolution remote sensing scene image sample dataset of various scenes, and divide the high-resolution remote sensing scene image sample dataset into a training set and a test set according to a preset ratio; Data preprocessing: preprocess the data in the training set and test set respectively; Label smoothing: Label smoothing is performed on the preprocessed training set data based on prior statistical information, and label smoothing is performed on the remote sensing image scene classification training data and remote sensing image semantic segmentation data in the training set respectively; Neural network model: Build a neural network model and train it based on the label-smoothed data; Model testing: Input the sample data in the test set into the trained neural network model to obtain the predicted category of the test set image; The calculation steps for label smoothing of the remote sensing scene classification training data are as follows: A. Statistical results of the priori category of small batch remote sensing image scene classification training data labels: Where: c is the total number of all categories in the entire dataset; batch_size is the number of data in the batch input data; is the number of data of class i in the batch of data, where 1≤i≤c; B. Define a smoothing factor α, where 0≤α≤1; add the category prior statistics to the smoothing factor to obtain the smoothing factor α′ added to the category prior statistics: The original label l for remote sensing scene classification training data n Perform label smoothing, where 1≤n≤batch s ize; get smooth labels C. Introducing the smoothing factor α′ into the smoothed label Get the classification label: Where: l′ n Where n is greater than or equal to 1 and less than or equal to batch_size; is the preliminary smoothed label of the nth data in the mini-batch data; The calculation steps for label smoothing of remote sensing image semantic segmentation data are as follows: A. Set the side length of the square sliding window to window_size. Each time the window slides, the statistical distribution of the labels in the sliding window is counted to obtain the prior statistical category distribution vector, which represents the statistical distribution of the labels in each category in the current window for the semantic segmentation training data with c semantic category labels: Where: Indicates the number of pixels belonging to class i in the current window, where i is greater than or equal to 1 and less than or equal to c; window_size 2 Indicates the total number of pixels contained in the rectangular window; B. Define a smoothing factor α, where α is greater than or equal to 0 and less than or equal to 1; smooth the label containing the prior statistical category distribution vector and calculate the smoothing factor α′; The original label l of the data in the training set n Perform label smoothing, where l n where n is greater than or equal to 1 and less than or equal to window_size 2 ; Get the smooth label corresponding to each pixel Where: e pq is the label component corresponding to the pixel with spatial coordinates [p,q] in the window; C. Mapping the smoothing factor α′ into the smoothing label Get the classification label l′ of each pixel based on the prior statistical label smoothing n : Where: is the preliminary smoothed semantic label of the n-th pixel in the corresponding window, l′ n where n is greater than or equal to 1 and less than or equal to window_size 2 .

2. The method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing according to claim 1, characterized in that: The division of the training set and the test set comprises the following steps: Construct a remote sensing scene image sample dataset and create corresponding sample labels; The remote sensing scene image sample dataset is divided into a training set and a test set. For each type of remote sensing image in the remote sensing scene image sample dataset, the total number of remote sensing images is set to n, and m images are randomly selected from them to construct the training set, and the remaining nm images are used to construct the test set.

3. The method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing according to claim 1, characterized in that: The data preprocessing includes: performing noise reduction, random erasing and random rotation processing on the training set and test set samples; and performing scaling processing on the training set samples.

4. The method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing according to claim 3, characterized in that: The calculation method of the scaling process is as follows: For an input image x of scale s×s, at the beginning of each training iteration, it is scaled to scale d×d, where d is calculated with respect to s as follows: Where t is the current training iteration cycle, and iters is the total number of iterations experienced during a round of training.

5. The method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing according to claim 1, characterized in that: Establishing the neural network model also includes setting hyperparameters of the neural network model, wherein the hyperparameters include parameter initialization, training batch, scaling factor, learning rate, optimization method, number of iterations, and window size of the sliding window.

6. The method for remote sensing image scene classification and semantic segmentation tasks based on label smoothing according to any one of claims 1 to 5, characterized in that: The constructed neural network model is pre-trained based on the ImageNet dataset. The model is trained by minimizing the loss function that measures the sample label and the network prediction value, and the parameters of the network model are updated until the network parameter values ​​converge. The condition for convergence is that the value of the current loss function no longer decreases compared with the value calculated by the previous training. A pre-trained neural network model is obtained, and then the pre-trained neural network model is trained using the training set to obtain a trained neural network model.

Images