Remote sensing image semantic segmentation method based on semantic driving and graph convolution knowledge distillation
Through the method based on semantic driving and graph convolution knowledge distillation, the problems of high memory and high computational cost in semantic segmentation of remote sensing images are solved, and efficient semantic segmentation effect is achieved, reducing in-class variance and processing noise, and improving remote sensing image segmentation is achieved. accuracy.
Patent Information
- Application Number
- CN202411144603.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-03
AI Technical Summary
Existing remote sensing image semantic segmentation technology has problems with high memory consumption and high computing costs when processing large-scale and complex data, and the multimodal fusion method is oversimplified, making it impossible to accurately capture depth information and ignore noise.
Using a method based on semantic drive and graph convolution knowledge distillation, the teacher model is used to transfer knowledge to the student model through the training and testing stages, and combined with bilateral filters, graph convolution and attention mechanisms, the auxiliary spatial feature extraction module and RGB refinement module realizes noise reduction and alignment of normalized digital surface model diagrams, and reduces in-class variance through semantic-driven knowledge distillation modules and structural distillation modules.
It achieves excellent semantic segmentation efficiency and accuracy with very few parameters and calculations, alleviates the problems of category imbalance and noise in remote sensing images, and improves image segmentation accuracy.
Smart Images

Figure CN119068362A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of semantic segmentation networks of deep learning, and more specifically to a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation. Background Art
[0002] Image semantic segmentation is a computer vision technology that aims to assign corresponding category labels to each pixel in an image, thereby providing a more sophisticated understanding of the image than object detection. Currently, this technology is widely used in remote sensing image analysis, unmanned driving, medical image analysis, drone monitoring, and intelligent interaction. Among them, because optical remote sensing images usually contain more categories of objects, with large intra-class variance, similarity between classes, and drastic scale changes of objects, semantic segmentation is challenging in the field of remote sensing and has always been a hot topic and difficulty in research.
[0003] Existing commonly used semantic segmentation technologies include support vector machines, random forest algorithms, and deep learning technologies. Among them, support vector machines and random forest algorithms mainly perform well in solving classification and regression problems, that is, they are more suitable for binary classification tasks. In addition, these traditional methods are usually highly complex and are only effective for models with small data volumes. In the field of remote sensing image semantic segmentation, the amount of data that needs to be processed is huge, so traditional classification algorithms are no longer suitable. However, deep learning technology has obvious advantages in processing large-scale and complex data types, because it can learn the hierarchical representation of data by building a multi-layer neural network, thereby learning more abstract semantic information to express the information in the image and greatly improving the accuracy of image pixel-level classification tasks.
[0004] With the development of deep learning technology, a series of new network architectures and methods have emerged. For example, the fully convolutional network abandons the fully connected layer in the traditional convolutional neural network and adopts a fully convolutional layer structure. It can accept input images of any size, maintain spatial information, and output segmented images of the same size. Subsequently, the encoder-decoder structure was derived on its basis, which is also a commonly used network architecture in current segmentation tasks. The encoder is usually used for feature extraction and dimensionality reduction, while the decoder is responsible for restoring spatial resolution and generating segmentation maps. Typical encoder-decoder structures such as U-Net and SegNet effectively combine low-level detail information and high-level semantic information through skip connections. In addition, in order to expand the receptive field without increasing the number of parameters, the hole convolution was proposed. By inserting "holes" between the convolution kernels, the resolution can be effectively maintained while increasing the receptive field. Secondly, in order to enable the network to focus on the important parts of the image and ignore unimportant information, the attention mechanism was proposed, which is particularly important in semantic segmentation because it can help the network better understand complex scenes and details.
[0005] Although the above methods and network architectures have made significant progress in semantic segmentation accuracy, the existing high-precision models are usually implemented at the cost of high memory consumption and high computational costs, which makes it difficult for these models to be implanted in mobile portable devices and used for practical development and application. In addition, most of the existing multimodal fusion methods oversimplify the modal fusion problem, that is, directly assuming that the depth map is accurate and can be perfectly aligned with the RGB pixels. However, current depth sensors are usually unable to accurately capture depth information, and the actual depth map usually contains a lot of noise.
[0006] Therefore, it is an urgent problem for those skilled in the art to propose a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation to solve the difficulties existing in the prior art. Summary of the invention
[0007] In view of this, the present invention provides a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation, which can achieve excellent semantic segmentation efficiency and accuracy with very few parameters and calculations.
[0008] In order to achieve the above object, the present invention provides the following technical solutions:
[0009] A remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation, characterized by comprising two processes: a training phase and a testing phase;
[0010] The training phase consists of the following steps:
[0011] S101, selecting Q original remote sensing images and a normalized digital surface model image nDSM and a true semantic segmentation image corresponding to each original remote sensing image, and forming a training set;
[0012] S102, construct a convolutional neural network;
[0013] S103, performing data enhancement on each original remote sensing image and the corresponding nDSM map in the training set as the original input image, inputting the image into the convolutional neural network for training, and obtaining the corresponding predicted remote sensing image;
[0014] S104, calculating the loss function value between the predicted image and the corresponding true semantic segmentation image;
[0015] S105, repeating step 103 and step 104 until the convolutional neural network converges and the loss function value is minimized, thereby obtaining a teacher model for convolutional neural network classification training;
[0016] S106, constructing a distillation framework;
[0017] S107, transfer the knowledge in the teacher model to the student model;
[0018] The testing phase includes the following steps:
[0019] S201, select the Pth group of remote sensing images to be tested in the test set and mark them as Ip, where if it is the Vaihingen data set, then 1≤p≤P, P=17; if it is the Potsdam data set, then 1≤p≤P, P=7;
[0020] S202, input various channels of the Pth group of remote sensing images to be detected into the trained model, and the model network outputs the remote sensing image semantic segmentation prediction image corresponding to the original remote sensing image. The obtained prediction image is recorded as
[0021] In the above method, optionally, in S101, the qth original remote sensing image in the training set is recorded as {Iq(i, j)}, and the real semantic segmentation image corresponding to {Iq(i, j)} in the training set is recorded as
[0022] The one-hot encoding technology is used to process the real semantic segmentation image corresponding to each original remote sensing image in the training set into a one-hot encoded image of size 256×256. The set of processed one-hot encoded images is recorded as
[0023] The remote sensing image is an RGB color image, Q is a positive integer, q is a positive integer, 1≤q≤Q, express The pixel value of the pixel with coordinate position (i, j) in the middle.
[0024] In the above method, optionally, the convolutional neural network in S102 includes a teacher network and a student network; the teacher network uses ResNet50 as a backbone network, and the student network uses MobileNetV-2 as a backbone network;
[0025] Convolutional neural networks consist of two parts: feature encoding and feature decoding;
[0026] The feature encoding operation includes: RGB image input layer, nDSM image input layer, ten convolution blocks, five RGB refinement modules RRM and five auxiliary spatial feature extraction modules ASFE;
[0027] The feature decoding operation consists of five cascaded convolutional blocks.
[0028] In the above method, optionally, the RGB image passes through a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, and a fifth convolution block in sequence;
[0029] The nDSM image passes through the sixth convolution block, the seventh convolution block, the eighth convolution block, the ninth convolution block, and the tenth convolution block in sequence;
[0030] The processed nDSM features are respectively input into the auxiliary spatial feature extraction module of the corresponding layer to obtain the aligned nDSM features and five spatial auxiliary weights. Then the spatial weights of the corresponding layer, the aligned nDSM features and the initially processed RGB features are input into the RGB refinement module. Under the guidance of the spatial auxiliary weights, the global context information is captured and the problem of class imbalance that is common in remote sensing images is alleviated.
[0031] The output of the RGB refinement module is fed to the cascaded eleventh convolution block, the twelfth convolution block, the thirteenth convolution block, the fourteenth convolution block, and the fifteenth convolution block in sequence for step-by-step pixel and feature recovery, and finally the five decoding feature outputs of the convolutional neural network are obtained.
[0032] In the above method, optionally, in S104, a cross entropy loss function is used to calculate the true semantic segmentation image. The collection of processed one-hot encoded images The set J consisting of the prediction graph pre The loss function value Loss(J pre , J ture ).
[0033] In the above method, optionally, in S106, the distillation framework consists of three parts: encoding feature structure knowledge distillation, decoding feature semantic knowledge distillation, and prediction graph response distillation;
[0034] The overall architecture of the distillation framework includes:
[0035] Encoded feature 1, encoded feature 2, encoded feature 3, encoded feature 4, and encoded feature 5 represent the outputs of the RGB refinement module in the teacher model, respectively;
[0036] Encoded feature six, encoded feature seven, encoded feature eight, encoded feature nine, and encoded feature ten respectively represent the outputs of the RGB refinement module in the student model;
[0037] Decoding feature 1, decoding feature 2, decoding feature 3, decoding feature 4, and decoding feature 5 represent the outputs of the auxiliary spatial feature extraction module in the teacher model;
[0038] Decoding feature six, decoding feature seven, decoding feature eight, decoding feature nine, and decoding feature ten respectively represent the outputs of the auxiliary spatial feature extraction module in the teacher model.
[0039] The above method optionally includes encoding feature structure knowledge distillation: the encoding features of the teacher and student corresponding layers are input as input sets into the structure distillation module based on inter-domain relations of graph convolution; different regions of different sizes of features in the input set are abstracted and the inter-domain abstract features are obtained, recorded as and The extracted inter-domain abstract features and Mapped to the graph space and applying graph convolution operation to reason about the inter-domain relationship, the inference feature is recorded as and In turn, we achieve intra-class compactness and inter-class separation; the inference feature is recorded as and The reverse mapping back to the feature map is recorded as and VID loss calculation using variational information distillation and The loss value between Where i = 5, the sum of the loss values of the five pairs of encoding features is used as the structural loss function value;
[0040] Decoding feature semantic knowledge distillation: The decoding features of the teacher and student layers are input into the semantic-driven knowledge distillation module as the input set; the semantic abstraction module SAM is used to perform semantic abstraction on the teacher decoding features in the input set, and the abstract semantic features are recorded as Will Injected into students to obtain the initial semantic enhancement features Through SAM The further abstracted semantic features are recorded as The input teacher features and student features are respectively compared with the corresponding abstract semantic features and Perform element-by-element multiplication to obtain the final semantic enhancement feature, recorded as and Variational information distillation (VID) loss is used to calculate each abstract semantic feature. and And each semantic enhancement feature and The loss value between Where i = 5, the sum of the loss values of the five pairs of decoded features is used as the semantic loss function value;
[0041] Prediction graph response distillation: For the prediction graph T of the teacher model pre And the prediction graph S of the student model pre , using the cross entropy loss function Cross Entropy Loss to calculate T pre and S preThe loss function value Loss(T pre , S pre );
[0042] The total loss of the distillation framework is the structure loss function value, the semantic loss function value and the response distillation loss function value Loss(T pre , S pre ) is the sum of .
[0043] In the above method, optionally, in S107, the knowledge in the teacher model is transferred to the student model: first, the weights in the trained teacher model are frozen, and then the student model is trained under the supervision of the teacher model using the constructed distillation framework, and steps S101-S106 are repeated M times until the student network converges and the loss function value is minimized, thereby obtaining the final student model;
[0044] During the training process, a total of Q loss function values are obtained. The loss function value with a small value is found among them, and the weight vector and bias term corresponding to this value are used as the optimal weight vector and optimal bias term of the student model, which are recorded as and Among them, M>1.
[0045] In the above method, optionally, in S202, the various channels of the remote sensing image to be detected include: an R channel component, a G channel component, a B channel component and a normalized digital surface model map nDSM component of Ip.
[0046] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation, which has the following beneficial effects:
[0047] 1) An auxiliary spatial feature extraction module (ASFE) with bilateral filter is proposed. This module is used to denoise the normalized digital surface model (nDSM) modal map. At the same time, a bidirectional guidance strategy is used to realize the alignment operation of RGB and nDSM and obtain spatially aligned bimodal features and spatial auxiliary weights. 2) An RGB refinement module (RRM) module is designed. This module processes the RGB features enhanced by spatial auxiliary weights by introducing graph convolution and reverse attention operations, thereby capturing global context information and alleviating the problem of class imbalance that is prevalent in remote sensing images. 3) A semantic-driven knowledge distillation module (SKD) is designed. This module guides the student model to learn the dark knowledge in the teacher features of the corresponding layer and the student features of the adjacent layer through a semantic injection-driven approach and combines traditional distillation and self-distillation strategies, and achieves the effect of inter-class separation. 4) A structural distillation of inter-domain relations (SDGC) based on graph convolution is designed. This module applies graph convolution to the extracted inter-domain relations and obtains abstract inter-domain structural features, thereby guiding the student model to learn structural relations and reducing intra-class variance. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0049] Figure 1 The overall implementation block diagram of a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation provided by the present invention;
[0050] Figure 2 It is an auxiliary spatial feature extraction module;
[0051] Figure 3 Refine the module for RGB;
[0052] Figure 4 This is the overall implementation diagram of the knowledge distillation framework;
[0053] Figure 5a is the first original remote sensing image;
[0054] Figure 5b In order to utilize the method of the present invention Figure 5a The original remote sensing image shown is predicted to obtain a predicted semantic segmentation image;
[0055] Figure 6a is the second original remote sensing image;
[0056] Figure 6b In order to utilize the method of the present invention Figure 6a The original remote sensing image shown is predicted to obtain a predicted semantic segmentation image;
[0057] Figure 7a This is the third original remote sensing image;
[0058] Figure 7b In order to utilize the method of the present invention Figure 7a The original remote sensing image shown is predicted to obtain a predicted semantic segmentation image;
[0059] Figure 8a This is the fourth original remote sensing image;
[0060] Figure 8b In order to utilize the method of the present invention Figure 8a The original remote sensing image shown is predicted to obtain a predicted semantic segmentation image. DETAILED DESCRIPTION
[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0062] See also Figure 1 As shown, a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation is characterized by including two processes: a training phase and a testing phase;
[0063] The training phase consists of the following steps:
[0064] S101, selecting Q original remote sensing images and a normalized digital surface model (nDSM) and a true semantic segmentation image corresponding to each original remote sensing image, and forming a training set;
[0065] S102, construct a convolutional neural network;
[0066] S103, performing data enhancement on each original remote sensing image and the corresponding nDSM map in the training set as the original input image, inputting the image into the convolutional neural network for training, and obtaining the corresponding predicted remote sensing image;
[0067] S104, calculating the loss function value between the predicted image and the corresponding true semantic segmentation image;
[0068] S105, repeating step 103 and step 104 until the convolutional neural network converges and the loss function value is minimized, thereby obtaining a teacher model for convolutional neural network classification training;
[0069] S106, constructing a distillation framework;
[0070] S107, transfer the knowledge in the teacher model to the student model;
[0071] The testing phase includes the following steps:
[0072] S201, select the Pth group of remote sensing images to be tested in the test set and mark them as Ip, where if it is the Vaihingen data set, then 1≤p≤P, P=17; if it is the Potsdam data set, then 1≤p≤P, P=7;
[0073] S202, input various channels of the Pth group of remote sensing images to be detected into the trained model, and the model network outputs the remote sensing image semantic segmentation prediction image corresponding to the original remote sensing image. The obtained prediction image is recorded as
[0074] Furthermore, in S101, the qth original remote sensing image in the training set is denoted as {I q (i, j)}, and the real semantic segmentation image corresponding to {Iq(i, j)} in the training set is denoted as
[0075] The one-hot encoding technology is used to process the real semantic segmentation image corresponding to each original remote sensing image in the training set into a one-hot encoded image of size 256×256. The set of processed one-hot encoded images is recorded as
[0076] Among them, the remote sensing image is an RGB color image, Q is a positive integer, such as Q = 784, q is a positive integer, 1≤q≤Q, express The pixel value of the pixel point with coordinate position (i, j) in the middle. The original remote sensing image is mainly composed of RGB images taken in different scenes. The RGB image records the spectral information of the three bands of red, green and blue. That is, each original scene image with significant objects contains the R channel component, G channel component and B channel component of the RGB image and the elevation component of the normalized digital surface model (nDSM).
[0077] Furthermore, the convolutional neural network in S102 includes a teacher network and a student network; the teacher network uses ResNet50 as the backbone network, and the student network uses MobileNetV-2 as the backbone network;
[0078] Convolutional neural networks consist of two parts: feature encoding and feature decoding;
[0079] The feature encoding operation includes: RGB image input layer, nDSM image input layer, ten convolutional blocks, five RGB refinement modules (RRMs), and five auxiliary spatial feature extraction modules (ASFEs);
[0080] The feature decoding operation consists of five cascaded convolutional blocks.
[0081] Further, refer to Figure 1 As shown, the RGB image passes through the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block in sequence;
[0082] The nDSM image passes through the sixth convolution block, the seventh convolution block, the eighth convolution block, the ninth convolution block, and the tenth convolution block in sequence;
[0083] The processed nDSM features are respectively input into the auxiliary spatial feature extraction module of the corresponding layer to obtain the aligned nDSM features and five spatial auxiliary weights. Then the spatial weights of the corresponding layer, the aligned nDSM features and the initially processed RGB features are input into the RGB refinement module. Under the guidance of the spatial auxiliary weights, the global context information is captured and the problem of class imbalance that is common in remote sensing images is alleviated.
[0084] The output of the RGB refinement module is fed to the cascaded eleventh convolution block, the twelfth convolution block, the thirteenth convolution block, the fourteenth convolution block, and the fifteenth convolution block in sequence for step-by-step pixel and feature recovery, and finally the five decoding feature outputs of the convolutional neural network are obtained.
[0085] Specifically, first, the input of the convolutional neural network is the RGB image and nDSM image of each picture (the width and height of the picture are W=256 and H=256 respectively, the RGB channels are R channel components, G channel components and B channel components, and the nDSM image is a three-channel image generated by copying along the channel dimension), and both are extracted through the MobileNetV-2 backbone network.
[0086] The structures of the two feature extraction flows are similar, specifically:
[0087] The first convolution block and the sixth convolution block are both composed of four inverse residual blocks connected in sequence, and both have 3 channels as input. The inverse residual block is composed of a convolution layer (Convolution, Conv), a normalization layer (BatchNorm) and an activation layer (Activation, Act, the activation function is Relu6) connected in sequence. The convolution kernel (kernel_size) of the convolution layer is 1×1, 3×3, 1×1, the number of convolution kernels (filters) is 24, the stride is 1, 2, 1, and the edge padding is 0, 1, and 0. After normalization, the feature map is output through the activation layer. The feature map sets output by the first and sixth convolution blocks are recorded as The width of the feature map is Height is
[0088] The second convolutional block and the seventh convolutional block are both composed of three inverse residual blocks connected in sequence, where the inverse residual block is composed of a convolutional layer, a normalization layer, and an activation layer connected in sequence. The convolutional kernel sizes of the convolutional layers are 1×1, 3×3, and 1×1, respectively, the number of convolutional kernels is 32, the step size is 1, 2, and 1, and the edge filling is 0, 1, and 0. After the normalization operation, the activation layer outputs the feature map. The feature map sets output by the second and seventh convolutional blocks are recorded as The width of the feature map is Height is
[0089] The third convolution block has the same structure as the eighth convolution block, which is composed of four inverse residual blocks connected in sequence. The inverse residual block is composed of a convolution layer, a normalization layer, and an activation layer connected in sequence. The convolution kernel sizes of the convolution layer are 1×1, 3×3, and 1×1, respectively, the number of convolution kernels is 64, the step sizes are 1, 2, and 1, respectively, and the edge filling is 0, 1, and 0. After the normalization operation, the activation layer outputs the feature map. The feature map sets output by the third and eighth convolution blocks are recorded as The width of the feature map is Height is
[0090] The fourth convolution block has the same structure as the ninth convolution block, which is composed of three inverse residual blocks connected in sequence. The inverse residual block is composed of a convolution layer, a normalization layer, and an activation layer connected in sequence. The convolution kernel sizes of the convolution layer are 1×1, 3×3, and 1×1, respectively, the number of convolution kernels is 96, the step sizes are 1, 2, and 1, respectively, and the edge filling is 0, 1, and 0. After the normalization operation, the feature map is output through the activation layer. The feature map sets output by the fourth and ninth convolution blocks are recorded as The width of the feature map is Height is
[0091] The fifth convolution block has the same structure as the tenth convolution block, which is composed of three inverse residual blocks connected in sequence. The inverse residual block is composed of a convolution layer, a normalization layer, and an activation layer connected in sequence. The convolution kernel sizes of the convolution layer are 1×1, 3×3, and 1×1, respectively, the number of convolution kernels is 160, the step sizes are 1, 2, and 1, respectively, and the edge padding is 0, 1, and 0. After the normalization operation, the activation layer outputs the feature map. The feature map sets output by the fifth and tenth convolution blocks are recorded as The width of the feature map is Height is
[0092] For auxiliary spatial feature extraction modules, such as Figure 2 As shown, the outputs of the first convolution block, the sixth convolution block, the second convolution block, the seventh convolution block, the third convolution block, the eighth convolution block, the fourth convolution block, the ninth convolution block, the fifth convolution block, and the tenth convolution block are respectively input into the first auxiliary spatial feature extraction module, the second auxiliary spatial feature extraction module, ..., until the fifth auxiliary spatial feature extraction module. Figure 2 As shown, input one and input two represent the features extracted from nDSM and RGB through the backbone network respectively. After passing through the bilateral filter and spatial self-calibration one, output six and weight six are obtained, and the two are multiplied pixel by pixel to obtain output seven; then output seven is input into spatial self-calibration two to obtain weight seven, and is multiplied pixel by pixel with input two to obtain output eight; secondly, output eight is passed through channel self-calibration two to obtain weight eight, and output seven and weight eight are multiplied to obtain output nine; then output nine is input into channel self-calibration one to obtain weight nine, and weight nine and output nine are multiplied pixel by pixel to obtain output ten; finally, output ten and input one are added pixel by pixel to obtain the final output of the auxiliary spatial feature extraction module, that is, output eleven.
[0093] For the RGB refinement module, such as Figure 3As shown. Input five represents the weight nine obtained by the auxiliary spatial feature extraction module as the spatial guiding feature, and input three and input four respectively represent the features extracted from RGB through the backbone network and the output eleven of the auxiliary spatial feature extraction module as the guided features for fusion and refinement operations. First, input three and input four are spliced to obtain output twelve, and then they are multiplied element by element with output thirteen to obtain output thirteen; then output thirteen is respectively input into the global branch and the local branch for processing. In the global branch, output thirteen is sequentially processed through graph convolution one, graph convolution two, graph convolution three and activation function one to obtain output fifteen, and then output fifteen is multiplied element by element with input three to obtain output sixteen. In the local branch, output thirteen first obtains weight ten through spatial self-calibration three, then weight ten is multiplied element by element with output thirteen to obtain output fourteen, and then output fourteen is obtained through channel self-calibration three to obtain weight eleven, and then weight eleven and weight eleven after reverse attention operation are respectively multiplied element by element with output fourteen to obtain output seventeen and output eighteen. Then, output sixteen, output seventeen and output eighteen are respectively passed through the sixteenth convolution block, the seventeenth convolution block and the eighteenth convolution block, and then concatenated along the channel dimension to obtain output nineteen; finally, output nineteen is passed through the eighteenth convolution block and residually connected with input three to obtain the final output of the RGB refinement module, that is, output twenty.
[0094] Furthermore, each original remote sensing image and the corresponding nDSM map in the training set are enhanced by brightness, contrast, color adjustment, random flipping, cropping, etc., and then used as the original input image, which is input into the convolutional neural network for training to obtain the corresponding predicted remote sensing image. The predicted map includes the semantic segmentation prediction map. The obtained predicted remote sensing image set is recorded as J pre .
[0095] Furthermore, in S104, a cross entropy loss function is used to calculate the true semantic segmentation image. The collection of processed one-hot encoded images The set J consisting of the prediction graph pre The loss function value Loss(J pre , J ture ).
[0096] Further, step S103 and step S104 are repeated V times until the convolutional neural network converges and the loss function value is minimized, thereby obtaining a teacher model for the convolutional neural network classification training. In the training process, a total of Q loss function values are obtained, and a loss function value with a small value is found among them. The weight vector and bias item corresponding to the value are used as the optimal weight vector and optimal bias item of the convolutional neural network classification training teacher model, which are denoted by W. best and bbest ; Among them, V>1
[0097] Further, see Figure 4 As shown, in S106, the distillation framework consists of three parts: encoding feature structure knowledge distillation, decoding feature semantic knowledge distillation, and prediction graph response distillation;
[0098] The overall architecture of the distillation framework includes:
[0099] Encoded feature 1, encoded feature 2, encoded feature 3, encoded feature 4, and encoded feature 5 represent the outputs of the RGB refinement module in the teacher model, respectively;
[0100] Encoded feature six, encoded feature seven, encoded feature eight, encoded feature nine, and encoded feature ten respectively represent the outputs of the RGB refinement module in the student model;
[0101] Decoding feature 1, decoding feature 2, decoding feature 3, decoding feature 4, and decoding feature 5 represent the outputs of the auxiliary spatial feature extraction module in the teacher model;
[0102] Decoding feature six, decoding feature seven, decoding feature eight, decoding feature nine, and decoding feature ten respectively represent the outputs of the auxiliary spatial feature extraction module in the teacher model.
[0103] Furthermore, the encoding feature structure knowledge distillation is performed: the encoding features of the corresponding layers of the teacher and the student are input as a set into the structural distillation module based on the inter-domain relationship of the graph convolution (such as encoding features 1 and 6 as a pair of inputs of the first structural distillation of the inter-domain relationship based on the graph convolution, such as encoding features 2 and 7 as a pair of inputs of the second structural distillation of the inter-domain relationship based on the graph convolution, and the subsequent operations are analogous); different regions of different sizes of features in the input set are abstracted and the inter-domain abstract features are obtained, which are recorded as and The extracted inter-domain abstract features and Mapped to the graph space and applying graph convolution operation to reason about the inter-domain relationship, the inference feature is recorded as and In turn, we achieve intra-class compactness and inter-class separation; the inference feature is recorded as and The reverse mapping back to the feature map is recorded as and The variational information distillation (VID) loss is used for calculation and The loss value between Where i = 5, the sum of the loss values of the five pairs of encoding features is used as the structural loss function value;
[0104] Decoding feature semantic knowledge distillation: The decoding features of the corresponding layers of the teacher and the student are input as a set into the semantic-driven knowledge distillation module (e.g., decoding features 1 and 6 are used as a pair of inputs for the first semantic-driven knowledge distillation, and decoding features 2 and 7 are used as a pair of inputs for the second semantic-driven knowledge distillation, and the subsequent operations are analogous). The semantic abstraction module (SAM) is used to perform semantic abstraction on the teacher decoding features in the input set, and the abstract semantic features are recorded as Will Injected into students to obtain the initial semantic enhancement features Through SAM The further abstracted semantic features are recorded as The input teacher features and student features are respectively compared with the corresponding abstract semantic features and Perform element-by-element multiplication to obtain the final semantic enhancement feature, recorded as and Variational Information Distillation (VID) loss is used to calculate each abstract semantic feature and And each semantic enhancement feature and The loss value between Where i = 5, the sum of the loss values of the five pairs of decoded features is used as the semantic loss function value;
[0105] Prediction graph response distillation: For the prediction graph T of the teacher model pre And the prediction graph S of the student model pre , using the cross entropy loss function (Cross Entropy Loss) to calculate T pre and S pre The loss function value Loss(T pre , S pre );
[0106] The total loss of the distillation framework is the structure loss function value, the semantic loss function value and the response distillation loss function value Loss(T pre , S pre ) is the sum of .
[0107] Furthermore, in S107, the knowledge in the teacher model is transferred to the student model: first, the weights in the trained teacher model are frozen, and then the student model is trained under the supervision of the teacher model using the constructed distillation framework, and steps S101-S106 are repeated M times until the student network converges and the loss function value is minimized, thereby obtaining the final student model;
[0108] During the training process, a total of Q loss function values are obtained. The loss function value with a small value is found among them, and the weight vector and bias term corresponding to this value are used as the optimal weight vector and optimal bias term of the student model, which are recorded as and Among them, M>1.
[0109] Further, in S202, various channels of the remote sensing image to be detected include: p R channel component, G channel component, B channel component and normalized digital surface model map nDSM component.
[0110] In a specific embodiment, the Python-based deep learning library Pytorch3.8 is used to build the architecture of the remote sensing image semantic segmentation network based on semantic drive and graph convolution knowledge distillation. The test set in the Potsdam and Vaihingen datasets is used to analyze the remote sensing image semantic segmentation effect using the method of the present invention.
[0111] The segmentation performance of the predicted semantic segmentation image is evaluated by using two commonly used objective parameters for evaluating semantic segmentation methods as evaluation indicators, namely, mean class accuracy (mAcc) and mean intersection over union (MIoU) of the segmented image and the label image. The method of the present invention is used to predict each remote sensing image in the Potsdam and Vaihingen datasets, and a predicted semantic segmentation image corresponding to each remote sensing image is obtained. The mean class accuracy mAcc and the ratio of the intersection over union (MIoU) of the segmented image and the label image reflecting the semantic segmentation effect of the method of the present invention are shown in Tables 1 and 2. It can be seen from the data listed in Tables 1 and 2 that the semantic segmentation results of the remote sensing images obtained by the method of the present invention are good, indicating that it is feasible and effective to obtain the predicted semantic segmentation images corresponding to the remote sensing images by using the method of the present invention.
[0112] Table 1 Evaluation results of the method of the present invention on the Vaihingen test set
[0113]
[0114] Table 2 Evaluation results of the method of the present invention on the Potsdam test set
[0115]
[0116] Figure 5a The first original remote sensing image is given; Figure 5b The method of the present invention is given Figure 5a The original remote sensing image shown is predicted to obtain a predicted semantic segmentation image; Figure 6a The second original remote sensing image is given; Figure 6b The method of the present invention is given Figure 6a The original remote sensing image shown is predicted to obtain a predicted semantic segmentation image; Figure 7a The third original remote sensing image is given; Figure 7b The method of the present invention is given Figure 7a The original remote sensing image shown is predicted to obtain a predicted semantic segmentation image; Figure 8a The fourth original remote sensing image is given; Figure 8b The method of the present invention is given Figure 8a The original remote sensing image shown in Figure 1 is used for prediction to obtain the predicted semantic segmentation image. Figure 5a and Figure 5b ,contrast Figure 6a and Figure 6b ,contrast Figure 6a and Figure 6b ,contrast Figure 8a and Figure 8b It can be seen that the segmentation accuracy of the predicted semantic segmentation image obtained by the method of the present invention is high.
[0117] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0118] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation, characterized in that: It includes two processes: training phase and testing phase; The training phase consists of the following steps: S101, selecting Q original remote sensing images and a normalized digital surface model image nDSM and a true semantic segmentation image corresponding to each original remote sensing image, and forming a training set; S102, construct a convolutional neural network; S103, performing data enhancement on each original remote sensing image and the corresponding nDSM map in the training set as the original input image, inputting the image into the convolutional neural network for training, and obtaining the corresponding predicted remote sensing image; S104, calculating the loss function value between the predicted image and the corresponding true semantic segmentation image; S105, repeatedly executing S103 and S104 until the convolutional neural network converges and the loss function value is minimized, thereby obtaining a teacher model for convolutional neural network classification training; S106, constructing a distillation framework; S107, transfer the knowledge in the teacher model to the student model; The testing phase includes the following steps: S201, select the Pth group of remote sensing images to be tested in the test set and mark them as I p , where if it is the Vaihingen data set, then 1≤p≤P, P=17; if it is the Potsdam data set, then 1≤p≤P, P=7; S202, input various channels of the Pth group of remote sensing images to be detected into the trained model, and the model network outputs the remote sensing image semantic segmentation prediction image corresponding to the original remote sensing image. The obtained prediction image is recorded as 2. According to claim 1, a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation is characterized in that: In S101, the qth original remote sensing image in the training set is denoted as {I q (i,j)}, and {I q (i,j)} corresponding to the real semantic segmentation image is recorded as The one-hot encoding technology is used to process the real semantic segmentation image corresponding to each original remote sensing image in the training set into a one-hot encoded image of size 256×256. The set of processed one-hot encoded images is recorded as The remote sensing image is an RGB color image, Q is a positive integer, q is a positive integer, 1≤q≤Q, express The pixel value of the pixel with coordinate position (i, j) in the middle.
3. According to claim 1, a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation is characterized in that: The convolutional neural network in S102 includes a teacher network and a student network; the teacher network uses ResNet50 as the backbone network, and the student network uses MobileNetV-2 as the backbone network; Convolutional neural networks consist of two parts: feature encoding and feature decoding; The feature encoding operation includes: RGB image input layer, nDSM image input layer, ten convolution blocks, five RGB refinement modules RRM and five auxiliary spatial feature extraction modules ASFE; The feature decoding operation consists of five cascaded convolutional blocks.
4. According to claim 3, a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation is characterized in that: The RGB image passes through the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block in sequence; The nDSM image passes through the sixth convolution block, the seventh convolution block, the eighth convolution block, the ninth convolution block, and the tenth convolution block in sequence; The processed nDSM features are respectively input into the auxiliary spatial feature extraction module of the corresponding layer to obtain the aligned nDSM features and five spatial auxiliary weights. Then the spatial weights of the corresponding layer, the aligned nDSM features and the initially processed RGB features are input into the RGB refinement module. Under the guidance of the spatial auxiliary weights, the global context information is captured and the problem of class imbalance that is common in remote sensing images is alleviated. The output of the RGB refinement module is fed to the cascaded eleventh convolution block, the twelfth convolution block, the thirteenth convolution block, the fourteenth convolution block, and the fifteenth convolution block in sequence for step-by-step pixel and feature recovery, and finally the five decoding feature outputs of the convolutional neural network are obtained.
5. According to claim 1, a remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation is characterized in that: In S104, the cross entropy loss function is used to calculate the true semantic segmentation image The collection of processed one-hot encoded images The set J consisting of the prediction graph pre The loss function value Loss(J pre ,J ture ).
6. The remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation according to claim 1, characterized in that: In S106, the distillation framework consists of three parts: encoding feature structure knowledge distillation, decoding feature semantic knowledge distillation, and prediction graph response distillation; The overall architecture of the distillation framework includes: Encoded feature one, encoded feature two, encoded feature three, encoded feature four, and encoded feature five represent the outputs of the RGB refinement module in the teacher model, respectively; Encoded feature six, encoded feature seven, encoded feature eight, encoded feature nine, and encoded feature ten respectively represent the outputs of the RGB refinement module in the student model; Decoding feature 1, decoding feature 2, decoding feature 3, decoding feature 4, and decoding feature 5 represent the outputs of the auxiliary spatial feature extraction module in the teacher model; Decoding feature six, decoding feature seven, decoding feature eight, decoding feature nine, and decoding feature ten respectively represent the outputs of the auxiliary spatial feature extraction module in the teacher model.
7. The remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation according to claim 6, characterized in that: Encoded feature structure knowledge distillation: The encoded features of the corresponding layers of the teacher and the student are input as the input set into the structural distillation module based on the inter-domain relationship of graph convolution; different regions of different sizes of features in the input set are abstracted and the inter-domain abstract features are obtained and recorded as and The extracted inter-domain abstract features and Mapped to the graph space and applying graph convolution operation to reason about the inter-domain relationship, the inference feature is recorded as and In turn, we achieve intra-class compactness and inter-class separation; the inference feature is recorded as and The reverse mapping back to the feature map is recorded as and Using variational information distillation VID loss calculation and The loss value between Where i = 5, the sum of the loss values of the five pairs of encoding features is used as the structural loss function value; Decoding feature semantic knowledge distillation: The decoding features of the corresponding layers of the teacher and the student are input into the semantic-driven knowledge distillation module as an input set; The semantic abstraction module SAM is used to perform semantic abstraction on the teacher decoding features in the input set, and the abstract semantic features are recorded as Will Injected into students to obtain the initial semantic enhancement features Through SAM The further abstracted semantic features are recorded as The input teacher features and student features are respectively compared with the corresponding abstract semantic features and Perform element-by-element multiplication to obtain the final semantic enhancement feature, recorded as and Variational information distillation (VID) loss is used to calculate each abstract semantic feature. and And each semantic enhancement feature and The loss value between Where i = 5, the sum of the loss values of the five pairs of decoded features is used as the semantic loss function value; Prediction graph response distillation: For the prediction graph T of the teacher model pre And the prediction graph S of the student model pre , using the cross entropy loss function to calculate T pre and S pre The loss function value Loss(T pre ,S pre ); The total loss of the distillation framework is the structure loss function value, the semantic loss function value and the response distillation loss function value Loss(T pre ,S pre ) is the sum of .
8. The remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation according to claim 1, characterized in that: In S107, the knowledge in the teacher model is transferred to the student model: first, the weights in the trained teacher model are frozen, and then the student model is trained under the supervision of the teacher model using the constructed distillation framework, and steps S101-S106 are repeated M times until the student network converges and the loss function value is minimized, thereby obtaining the final student model; During the training process, a total of Q loss function values are obtained. The loss function value with a small value is found among them, and the weight vector and bias term corresponding to this value are used as the optimal weight vector and optimal bias term of the student model, which are recorded as and Among them, M>
1.
9. The remote sensing image semantic segmentation method based on semantic drive and graph convolution knowledge distillation according to claim 1, characterized in that: In S202, various channels of the remote sensing image to be detected include: p R channel component, G channel component, B channel component and normalized digital surface model map nDSM component.
Citation Information
Cited By
Deep recursive medical image fusion method based on self-supervised generative distillation
CN120543393A
Training method and application of semantic segmentation model of point cloud sequence
CN120765946A
