Lightweight remote sensing scene classification method based on neural network

Abstract

The invention relates to the technical field of image processing, in particular to a lightweight remote sensing scene classification method based on a neural network, and the method comprises the steps: firstly obtaining a remote sensing image data set, and preprocessing the remote sensing image data set to obtain a training set and a test set; then, a teacher model including ConvNext and a student model remote sensing scene classification network of a ShuffleNetv2 network embedded with a mixed attention structure are constructed; inputting the training set into a teacher model and a student model at the same time to train the remote sensing scene classification network to obtain a remote sensing scene classification model; and finally, inputting the test set into the remote sensing scene classification model for classification prediction, and comparing an obtained prediction label with a real label of the test set to complete the test of the remote sensing scene classification model. According to the invention, the feature extraction capability is improved by using a mixed attention structure; and a knowledge distillation method is used to transmit dark knowledge of the teacher model to the student model, so that the precision of the neural network is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a lightweight remote sensing scene classification method based on a neural network. Background Art

[0002] At present, with the rapid development of remote sensing satellite and drone imaging technology, people can easily obtain various styles of remote sensing scene images, which are used in natural disaster detection, urban planning and other aspects. Among them, remote sensing scene classification refers to the use of a certain algorithm to determine which type of scene the image belongs to according to the unique true label corresponding to each scene image. In the past few decades, researchers at home and abroad have proposed a variety of methods for remote sensing scene classification. Early remote sensing scene classification methods were mainly based on artificially designed features. The use of manually designed features requires many professionals with remote sensing professional knowledge to design features based on the color, shape, texture, space and spectral information in the image. The most commonly used algorithms are: color histogram and scale-invariant feature conversion, etc. However, the above methods have many disadvantages: features need to be manually designed, requiring a lot of professional knowledge; artificially designed features can only extract low-level features; it takes a lot of manpower and the accuracy is not high.

[0003] Due to the above shortcomings and the popularity of deep learning, people have gradually turned their attention to the use of deep neural networks for remote sensing scene classification, and have achieved effective results. In 2017, the attention mechanism was proposed, and various network models and attention modules that combine Transform with CNN (convolutional neural network) were applied to image scene classification. However, the above models and modules still have some shortcomings: the model parameters are large and the amount of calculation is large. If the number of model layers is reduced, the number of parameters and the amount of calculation are reduced, the accuracy of the model will be reduced. The conventional attention mechanism module is embedded in the model, which has little effect on the improvement of the model and will increase the number of parameters. Therefore, it is a difficult problem to improve the accuracy of the model while reducing the number of parameters and the amount of calculation of the model as much as possible. Summary of the invention

[0004] In view of this, the present invention aims to provide a lightweight remote sensing scene classification method based on neural network, constructing a remote sensing scene classification network with a ConvNext teacher model and a student model of a ShuffleNetv2 network embedded with a hybrid attention structure, and the hybrid attention structure improves the feature extraction capability; using the knowledge distillation method, the dark knowledge of the teacher model is transferred to the student model, so as to improve the accuracy of the neural network.

[0005] To achieve the above object, the technical solution created by the present invention is implemented as follows: A lightweight remote sensing scene classification method based on a neural network comprises the following steps: S1: Obtain a remote sensing image dataset and preprocess the remote sensing image dataset to obtain a training set and a test set; S2: Build a remote sensing scene classification network that includes a ConvNext teacher model and a student model embedded in a ShuffleNetv2 network with a hybrid attention structure; S3: The remote sensing images of the training set in step S1 are simultaneously input into the teacher model and the student model of the remote sensing scene classification network constructed in step S2 to train the remote sensing scene classification network to obtain a remote sensing scene classification model; the teacher model obtains a soft label, and the student model obtains a hard label; the relative entropy between the soft label and the hard label is calculated, and the cross entropy between the hard label and the true label of the training set is calculated, and the result of the proportional sum of the relative entropy and the cross entropy is used as the loss function of the training; S4: Use the remote sensing images of the test set in step S1 to input into the remote sensing scene classification model in step S3 for classification prediction, compare the predicted labels with the true labels of the test set, and complete the test of the remote sensing scene classification model.

[0006] Furthermore, in step S2, the student model includes a shallow feature extraction module, a first-stage feature extraction module, a second-stage feature extraction module, a third-stage feature extraction module and a feature output module; wherein the remote sensing image is input into the shallow feature extraction module to extract shallow features; the shallow features are input into the first-stage feature extraction module to extract first-stage features; the first-stage features are input into the second-stage feature extraction module to extract second-stage features; the second-stage features are input into the third-stage feature extraction module to extract third-stage features; the third-stage features are input into the feature output module to obtain hard labels corresponding to the remote sensing images.

[0007] Furthermore, the shallow feature extraction module includes a two-dimensional convolution layer and a maximum pooling layer; wherein the remote sensing image is input into the two-dimensional convolution layer for convolution processing, and the features obtained after processing are input into the maximum pooling layer after ReLU activation operation to complete downsampling and obtain shallow features; the feature output module includes a two-dimensional convolution layer, a global pooling layer and a fully connected layer; in the third stage, the features are input into the two-dimensional convolution layer for convolution processing, and the features obtained after processing are input into the global pooling layer after ReLU activation operation to obtain a feature sequence string, and the feature sequence string is input into the fully connected layer for mapping to obtain a hard label.

[0008] Furthermore, in the first-stage feature extraction module, after the shallow features are extracted by the hybrid attention sub-module, the extracted features are input into the basic unit sub-module to output the first-stage features; in the second-stage feature extraction module, after the first-stage features are successively extracted by two hybrid attention sub-modules, the extracted features are input into two basic unit sub-modules to finally output the second-stage features; in the third-stage feature extraction module, after the second-stage features are extracted by the hybrid attention sub-module, the extracted features are input into the basic unit sub-module, and the basic unit sub-module outputs the third-stage features.

[0009] Furthermore, in the hybrid attention submodule, the input features are divided into a first feature sequence and a second feature sequence according to the number of channels; the first feature sequence first passes through a depthwise separable convolution layer, and the output features are normalized and then input into a two-dimensional convolution layer. At this time, the output features are sequentially normalized and ReLU activated to obtain first sequence features; the second feature sequence first passes through a hybrid attention structure to extract hybrid attention features containing channel information and spatial information, and the hybrid attention features are input into a depthwise separable convolution layer, and the output features are normalized and then input into a two-dimensional convolution layer. At this time, the output features are sequentially normalized and ReLU activated to obtain second sequence features; the first sequence features and the second sequence features are concatenated and channel shuffled before being output.

[0010] Furthermore, in the hybrid attention structure, the input features enter the channel attention structure and the spatial attention structure in parallel; in the channel attention structure, the input features enter the average pooling layer and the maximum pooling layer respectively for channel-wise pooling to obtain two channel pooling features; after the two channel pooling features are input into the fully connected layer, the corresponding channel attention sub-weights are obtained respectively; the result obtained by adding the corresponding elements of the two channel attention sub-weights is multiplied by the first weight coefficient, and the result obtained by the multiplication is subjected to a sigmoid activation operation to obtain the channel attention weight; the channel attention weight is multiplied by the corresponding element of the input feature to obtain the channel attention feature; In the spatial attention structure, the input features enter the average pooling layer and the maximum pooling layer respectively for spatial pooling to obtain two spatial pooling features; the two spatial pooling features are respectively input into the multi-scale dilated convolution path to obtain no less than two spatial attention sub-weights with different receptive fields; all the spatial attention sub-weights are concatenated and enter the two-dimensional convolution layer for dimensionality reduction; the reduced features are multiplied by the second weight coefficient, and then the multiplied features are subjected to a sigmoid activation operation to obtain the spatial attention weights; the spatial attention weights are multiplied by the corresponding elements of the input features to obtain the spatial attention features; the channel attention features and the corresponding elements of the spatial attention features are added to obtain the mixed attention features.

[0011] Furthermore, in the multi-scale dilated convolution path, the input features pass through at least one two-dimensional convolution layer and at least one dilated convolution layer in parallel to obtain corresponding spatial attention sub-weights, each of which contains a corresponding receptive field information.

[0012] Further, in step S3, the output of the teacher model is processed by the softmax function to obtain a soft label, and the output of the student model is processed by the softmax function to obtain a hard label; The relative entropy is: ; in, represents the relative entropy, Represents the i-th remote sensing image in the training set The corresponding soft tag, Represents the i-th remote sensing image in the training set The corresponding hard tag; The cross entropy is: ; in, represents the cross entropy, Represents the i-th remote sensing image in the training set The corresponding true label; The loss function is: ; in, represents the loss function, Represents the loss function ratio.

[0013] Furthermore, the preprocessing in step S1 includes the following steps: S11: standardize the remote sensing images in the remote sensing image dataset so that the remote sensing images are scaled to the same data interval and range; S12: Unify the input size of remote sensing images; S13: Randomly select some remote sensing images for data enhancement.

[0014] Compared with the prior art, the invention can achieve the following beneficial effects: (1) In the lightweight remote sensing scene classification method based on neural network created by the present invention, the ConvNext teacher model and the student model of the ShuffleNetv2 network embedded with the hybrid attention structure construct the remote sensing scene classification network, and the hybrid attention structure improves the feature extraction capability; the knowledge distillation method is used to transfer the dark knowledge of the teacher model to the student model, which not only reduces the number of model parameters, but also improves the accuracy of the model, so that the model can be deployed on mobile devices with limited computing resources; (2) In the lightweight remote sensing scene classification method based on neural network described in the invention, the hybrid attention structure is designed in the form of parallel spatial attention mechanism and channel attention mechanism modules, which improves the feature extraction efficiency and suppresses irrelevant information; at the same time, the original spatial attention module is improved, and the original convolution is replaced by a large-kernel hole convolution, which can better focus on global features, strengthen the local and global feature extraction capabilities, and solve the problem that it is difficult to evenly extract tiny features in high-space remote sensing scenes; (3) In the lightweight remote sensing scene classification method based on neural network described in the invention, a channel division process is added to the hybrid attention submodule, and a hybrid attention structure is introduced to replace the original 1×1 convolution in ShuffleNetv2, so that the model can accurately extract insignificant features. At the same time, the features cover more parts to be identified and are more evenly distributed, which enhances the learning and expression capabilities of the neural network and achieves higher remote sensing scene recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A flowchart of a lightweight remote sensing scene classification method based on a neural network according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of the student model described in the embodiment of the present invention; Figure 3 A schematic diagram of the structure of the hybrid attention submodule described in the embodiment of the present invention; Figure 4 A schematic diagram of the structure of the hybrid attention structure described in the embodiment of the present invention; Figure 5 A schematic diagram of the training process of the remote sensing scene classification network described in the embodiment of the present invention; Figure 6 To compare the thermal maps of remote sensing images obtained by different methods. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0017] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0018] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0019] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0020] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0021] like Figures 1 to 5 As shown, the lightweight remote sensing scene classification method based on neural network described in the embodiment of the present invention comprises the following steps: S1: Obtain a remote sensing image dataset and preprocess the remote sensing image dataset to obtain a training set and a test set. The remote sensing image dataset in the embodiment of the present invention comes from the AID public dataset, and the original size of the remote sensing image is 600×600. The preprocessing in step S1 includes the following steps: S11: Standardizing the remote sensing images in the remote sensing image data set so that the remote sensing images are scaled to the same data interval. In the embodiment of the present invention, the Min-Max method is specifically used to map the data to the specified data interval, so as to reduce the impact of the scale, features, and distribution differences of the images on the model, help the model to better learn features, and improve the performance of the model; S12: unifying the input size of the remote sensing image, specifically adjusting the remote sensing image from the original size of 600×600 to 224×224, the adjustment process includes image scaling and image cropping; S13: Randomly select some remote sensing images for data enhancement. The specific process is to perform operations such as cropping, scaling, rotating, flipping, and color space conversion (such as grayscale, color channel conversion, etc.) on some remote sensing images, which helps to enhance the robustness and generalization ability of the model obtained by subsequent training.

[0022] S2: Construct a remote sensing scene classification network; wherein the remote sensing scene classification network includes a teacher model of ConvNext (from the paper "A ConvNet for the 2020s" of the 2022 CVPR conference) and a student model of a ShuffleNetv2 network embedded with a hybrid attention structure. The present invention improves the original ShuffleNetV2 network (from the paper "ShuffleNetV2: Practical Guidelines for Efficient CNN Architecture Design" published in the journal "Computer Vision - ECCV 2018, Lecture Notes in Computer Science"), and the structure of the obtained student model is as follows Figure 2 As shown, it includes a shallow feature extraction module, a first-stage feature extraction module, a second-stage feature extraction module, a third-stage feature extraction module and a feature output module.

[0023] The shallow feature extraction module includes a two-dimensional convolution layer and a maximum pooling layer. The remote sensing image is input into the two-dimensional convolution layer for convolution processing, and the features obtained after the processing are input into the maximum pooling layer after the ReLU activation operation to complete downsampling and obtain shallow features. In an embodiment of the present invention, the image input into the shallow feature extraction module is a remote sensing image including three RGB channels (size is 224×224×3), and the remote sensing image is input into the two-dimensional convolution layer with a convolution kernel size of 3×3 for convolution processing to obtain features of size 24×112×112; the features obtained after the processing are input into the maximum pooling layer after the ReLU activation operation to complete downsampling pooling with a step size of 2 to obtain shallow features (size is 24×56×56). In the embodiment of the present invention, the number of repetitions of the first-stage feature extraction module, the second-stage feature extraction module, and the third-stage feature extraction module are all 2.

[0024] In the first-stage feature extraction module, after the shallow features are extracted by the hybrid attention submodule, the extracted features are input into the basic unit submodule to output the first-stage features; in the second-stage feature extraction module, after the first-stage features are continuously extracted by two hybrid attention submodules, the extracted features are input into two basic unit submodules to output the second-stage features; in the third-stage feature extraction module, after the second-stage features are extracted by the hybrid attention submodule, the extracted features are input into the basic unit submodule to output the third-stage features. In the embodiment of the present invention, the size of the first-stage features is 116×28×28, the size of the second-stage features is 232×14×14, and the size of the third-stage features is 464×7×7; and the basic unit submodules mentioned in the embodiment of the present invention are all basic units in the original ShuffleNetv2 network.

[0025] The embodiment of the present invention improves the original ShuffleNetV2 downsampling unit into a hybrid attention submodule designed with a hybrid attention structure. Figure 3As shown, in the hybrid attention submodule, the input features are divided into a first feature sequence and a second feature sequence according to the number of channels; the first feature sequence first passes through a depth-separable convolution layer, and the output features are normalized and then input into a two-dimensional convolution layer to strengthen the fusion of information between channels. At this time, the output features are sequentially normalized and ReLU activated to obtain first sequence features. The second feature sequence first passes through a hybrid attention structure to extract hybrid attention features containing channel information and spatial information, and the hybrid attention features are input into a depth-separable convolution layer, and the output features are normalized and then input into a two-dimensional convolution layer. At this time, the output features are sequentially normalized and ReLU activated to obtain second sequence features; the first sequence features and the second sequence features are feature concatenated and channel shuffled, and then the features are output. In an embodiment of the present invention, the convolution kernel size of the depth-separable convolution layer in the hybrid attention submodule is 3×3, the convolution step size is 2, and the convolution kernel size of the two-dimensional convolution layer is 1×1.

[0026] In the embodiment of the present invention, the convolution method in the original ShuffleNetV2 is changed to a depthwise separable convolution, which includes channel-by-channel convolution and point-by-point convolution, that is, the input feature map is first subjected to channel-by-channel convolution, and then point-by-point convolution through a 1×1 convolution kernel. The number of convolution kernels in the channel-by-channel convolution is equal to the number of channels of the input feature map, which reduces the number of convolution kernels compared to ordinary convolution. Point-by-point convolution obtains a feature map of the same scale as ordinary convolution, but greatly reduces the amount of calculation compared to ordinary convolution. The parameter quantity formula ratio of standard convolution to depthwise separable convolution is as follows: ; The calculation amount ratio of standard convolution and depth-separable convolution is as follows: ; Where: D K represents the convolution kernel size, M represents the number of channels of input features, N represents the number of channels of output features, and D F ×D F Indicates the size of the output feature map. From the above formula, we can see that although the calculation results are consistent, the ShuffleNetV2 using depthwise separable convolution greatly reduces the amount of calculation and the amount of parameters. On this basis, the number of repetitions of the feature extraction module in each stage of the ShuffleNetV2 model is adjusted, that is, the number of repetitions of the feature extraction module in the three stages is changed from 4, 8, 4 to 2, 2, 2, which reduces the basic unit in the original ShuffleNetv2 network. Reducing the network depth can effectively reduce the amount of calculation and the amount of parameters of the model, and can also avoid the decline in learning ability due to gradient descent.

[0027] Since the original CBAM (Convolutional Block Attention Module) uses the cascade method of CAM (Channel Attention Mechanism) and SAM (Spatial Attention Mechanism), that is, CAM first extracts channel features, and then SAM extracts spatial features, this will weaken SAM's extraction of spatial features of the input image. Therefore, the embodiment of the present invention improves CBAM and proposes a hybrid attention structure such as Figure 4 As shown, the input features enter the channel attention structure and the spatial attention structure in parallel.

[0028] In the channel attention mechanism (CAM), the input features enter the average pooling layer and the maximum pooling layer for channel-wise pooling to obtain two channel pooling features; after the two channel pooling features are input into the fully connected layer, the corresponding channel attention sub-weights are obtained. The result obtained by adding the corresponding elements of the two channel attention sub-weights is multiplied by the first weight coefficient, and then the result of the multiplication is subjected to a sigmoid activation operation to obtain the channel attention weight. The above process is represented by the following formula: ; in, represents the input features, represents the channel attention weight, represents the pooling operation of the average pooling layer in the channel direction, Represents the pooling operation of the maximum pooling layer in the channel direction, represents the fully connected operation of the multilayer perceptron, represents the first weight coefficient, represents the sigmoid activation operation, Indicates the addition of corresponding elements. The channel attention feature is obtained by multiplying the channel attention weight with the corresponding element of the input feature, that is: ; in, Represents channel attention features.

[0029] In the spatial attention structure, the input features enter the average pooling layer and the maximum pooling layer for spatial pooling, and obtain two spatial pooling features. The two spatial pooling features are respectively input into the multi-scale atrous convolution pathway (MAP) to obtain no less than two spatial attention sub-weights with different receptive fields. All the spatial attention sub-weights are concatenated and fused, and then the weights are multiplied by the second weight coefficient, and then the sigmoid activation operation is performed on the multiplied features to obtain the spatial attention weights. The above process is represented by the following formula: ; in, represents the spatial attention weight, represents the second weight coefficient, represents the operations performed in the multi-scale dilated convolution path, Indicates that the average pooling layer performs pooling operations in the spatial direction. Indicates that the maximum pooling layer performs pooling operations in the spatial direction. It represents a concatenation and fusion operation. In the embodiments of the present invention, all concatenation and fusion operations refer to that multiple input features are first concatenated channel by channel, and then a 1×1 convolution dimension reduction operation is performed on the concatenated features.

[0030] Specifically, in the multi-scale dilated convolution path, the input features are passed through a convolution layer with a convolution kernel size of 1×1 and two dilated convolution layers with a convolution kernel size of 3×3 in parallel to obtain three corresponding spatial attention sub-weights, each of which contains a corresponding receptive field information. In the embodiment of the present invention, the rate of intervals between parameters in the convolution kernels of the two dilated convolution layers is set to 1 and 2 respectively. The above process is represented by the following formula: ; in, Represents the input features Perform a convolution operation on the convolution layer with a convolution kernel size of 1×1. Represents the input features Perform a convolution operation on the dilated convolution layer with an interval rate of 1 and a convolution kernel size of 3×3. Represents the input features Perform a convolution operation on the dilated convolution layer with an interval rate of 2 and a convolution kernel size of 3×3. Multiply the spatial attention weight by the corresponding element of the input feature to obtain the spatial attention feature, that is: ; in, Represents spatial attention features.

[0031] Finally, the corresponding elements of the channel attention feature and the spatial attention feature are added to obtain the mixed attention feature, that is: ; in, Represents mixed attention features.

[0032] In the hybrid attention structure proposed by the present invention, the serial feature extraction in the original CBAM is changed to parallel extraction, and the spatial attention module is improved, so that the spatial information and channel information of the input features can be extracted simultaneously, so that the problem of weakened extraction ability of the spatial attention structure and loss of details can be effectively avoided during the extraction process. The first weight coefficient α and the second weight coefficient β are added to the channel attention structure and the spatial attention structure respectively, which can dynamically adjust the weight ratio between the channel attention structure and the spatial attention structure, achieve the effect of nonlinear adjustment, and enable the module to better extract the details of the input features. The initial values ​​of the first weight coefficient α and the second weight coefficient β are randomly selected, and then updated by the back propagation algorithm according to the calculated gradient of the loss function. The hybrid attention structure proposed by the present invention enables the model to accurately extract insignificant features. At the same time, the features cover more parts to be identified, and the distribution is more even, which enhances the learning ability and expression ability of the neural network and achieves a higher remote sensing scene recognition accuracy.

[0033] The feature output module includes a two-dimensional convolution layer, a global pooling layer, and a fully connected layer; the third-stage feature is input into the two-dimensional convolution layer for convolution processing, and the feature obtained after processing is input into the global pooling layer after ReLU activation operation to obtain a feature sequence string, and the feature sequence string is input into the fully connected layer for mapping to obtain a hard label. In the embodiment of the present invention, the third-stage feature of size 464×7×7 is input into the two-dimensional convolution layer with a convolution kernel size of 1×1 for convolution processing to obtain a feature of size 1024×7×7; the feature is input into the global pooling layer after ReLU activation operation to obtain a feature sequence string of size 1024×1×1; the feature sequence string is input into the fully connected layer for mapping to obtain a hard label.

[0034] S3: Input the remote sensing images of the training set in step S1 into the teacher model and student model of the remote sensing scene classification network constructed in step S2 to train the remote sensing scene classification network and obtain a remote sensing scene classification model. Figure 5As shown in the figure, the teacher model obtains soft labels and the student model obtains hard labels; the relative entropy between soft labels and hard labels is calculated, and the cross entropy between hard labels and the true labels of the training set is calculated. The result of proportional summation of relative entropy and cross entropy is used as the loss function of training. In step S3, the output of the teacher model is processed by the softmax function to obtain soft labels, and the output of the student model is processed by the softmax function to obtain hard labels. Relative entropy is expressed as: ; Among them, Loss KL represents relative entropy, P(x i ) represents the i-th remote sensing image x in the training set i The corresponding soft label, Q(x i ) represents the i-th remote sensing image x in the training set i The corresponding hard tag; The cross entropy is expressed as: ; Among them, Loss CE represents the cross entropy, Y(x i ) represents the i-th remote sensing image x in the training set i The corresponding true label; The loss function is: ; Among them, Loss represents the loss function, and γ represents the loss function ratio.

[0035] When the difference in the probability distribution of the output of the student model and the teacher model increases, the relative entropy between the soft label and the hard label will also increase. The training method uses the student model and the teacher model for knowledge distillation, and uses the proportional sum of relative entropy and cross entropy as the loss function. Compared with the traditional convolutional neural network training method, the temperature coefficient T is added, so that the output of the student model or the teacher model is in the form of , making the output of Softmax smoother and the model's distribution of different categories more even.

[0036] In the embodiment of the present invention, the dark knowledge in the teacher model is transferred to the student model. The learning rate of the experimental setting is 0.0005, the stochastic gradient descent (SGD) optimizer is used, the batch size (BatchSize) is set to 16, and the number of training rounds (epoch) is 100, that is, the training process is terminated after completing 100 training times, and the remote sensing scene classification model is obtained, which improves the accuracy of the model without changing the neural network structure.

[0037] S4: Use the remote sensing images of the test set in step S1 to input into the remote sensing scene classification model in step S3 for classification prediction, compare the predicted labels with the true labels of the test set, and complete the test of the remote sensing scene classification model.

[0038] In order to demonstrate the effectiveness of the lightweight remote sensing scene classification method based on neural network described in the embodiment of the present invention, the remote sensing scene classification method described in the embodiment of the present invention is compared with the original ShuffleNetV2 model and other classic models. The comparison models selected in the embodiments of the present invention also include: MobileViT network model (published in the paper "MobileViT: Light-Weight, General-Purpose, and Mobile-Friendly Vision Transformer" published at the ICLR conference in 2022), Mob-ile-NetV3 network model (published in the paper "Searching for MobileNetV3" published at the ICCV conference in 2022), Resnet34 network model (published in the paper "Deep Residual Learning for Image Recognition" published at the CVPR conference in 2016), ConvNext network model, EMTCAL network model (published in the journal "IEEE Transactions on Geoscience and Remote Sensing" and "EMTCAL: Efficient Multiscale Transformer and Cross-Level Attention Learning for Remote Sensing Scene Classification"), LAG-MANet network model (published in the journal "Journal of Geodaetics and Cartography" and "LAG-MANet Model for Remote Sensing Image Scene Classification"), and Hornet network model (from NeurIPS The paper "HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions" in the 2022 conference proceedings. The main comparison indicators are: accuracy, precision, recall, F1 score, model parameters (Params) and computational complexity (FLOPs). The comparison data is shown in Table 1: Table 1:

[0039] As can be seen from Table 1, the model provided by the embodiment of the present invention has a significant decrease in the number of parameters and the amount of calculation compared with the lightweight models such as MoblieViT, MobileNetV3, ShuffNetV2, EMTCAL, LAG-MANet, Hornet, etc. And the accuracy rate has been improved. Specifically, compared with pure convolutional networks such as ShuffleNetV2, MoblieNetV3, ResNet34, ConvNext, EMACAL, the accuracy rate, precision rate, recall rate and F1 score have been greatly improved. Compared with the models of MobileViT, Hornet, LAG-MANET, which are convolution and TransForm fusion architectures, there is also a significant improvement, thus proving the superiority and effectiveness of this model.

[0040] In order to demonstrate the superiority of the method provided by the embodiment of the present invention, the method provided by the embodiment of the present invention is compared with the remote sensing scene classification method of the original ShuffleNetV2 model using gradient weighted class activation mapping (Grad-CAM), and the most significant results considered by the model are displayed through a visualized heat map technology (Grad-CAM, from the 2022 International Journal of Computer Vision paper "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization"), and a heat map is drawn, as shown in FIG. Figure 6 shown. Figure 6 (a) shows the heat map generated by the remote sensing scene classification method of the original ShuffleNetV2 model. Figure 6 (b) in the figure shows a heat map generated by the method provided in an embodiment of the present invention. Through comparative analysis, it can be clearly observed that the scene classification model improved by the method provided in an embodiment of the present invention has a significant improvement in performance compared to the original model. A more coherent and concentrated heat map can help the model better understand the content of the image. Specifically, the heat map of the result obtained by the method provided in an embodiment of the present invention shows a higher coherence, which means that the model can pay more attention to key areas during identification and classification, thereby improving the accuracy of classification; in addition, the heat map is concentrated in the area that needs to be classified, which further indicates that the model can accurately locate the target object, which is crucial to improving the accuracy and robustness of the model. The lightweight remote sensing scene classification method based on neural network described in the embodiment of the present invention can be used not only in stand-alone software, but also in cloud software deployment, supporting software functions and module calls on the local and web ends.

[0041] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0042] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A lightweight remote sensing scene classification method based on neural network, characterized by: The following steps are involved: S1: Acquire a remote sensing image dataset, and preprocess the remote sensing image dataset to obtain a training set and a test set; S2: Construct a remote sensing scene classification network consisting of a ConvNext teacher model and a student model ShuffleNetv2 network embedded with a hybrid attention structure; S3: inputting the remote sensing images of the training set in step S1 into the teacher model and the student model of the remote sensing scene classification network constructed in step S2 at the same time, training the remote sensing scene classification network, and obtaining a remote sensing scene classification model; The teacher model obtains a soft label, and the student model obtains a hard label; Calculating the relative entropy between the soft label and the hard label, calculating the cross entropy between the hard label and the real label of the training set, and taking the proportional sum of the relative entropy and the cross entropy as the loss function of training; S4: Use the remote sensing images of the test set in step S1 to input into the remote sensing scene classification model in step S3 for classification prediction, compare the predicted labels with the true labels of the test set, and complete the test of the remote sensing scene classification model.

2. The lightweight remote sensing scene classification method based on neural network according to claim 1 is characterized in that: In step S2, the student model includes a shallow feature extraction module, a first-stage feature extraction module, a second-stage feature extraction module, a third-stage feature extraction module and a feature output module; wherein the remote sensing image is input into the shallow feature extraction module to extract shallow features; the shallow features are input into the first-stage feature extraction module to extract first-stage features; the first-stage features are input into the second-stage feature extraction module to extract second-stage features; the second-stage features are input into the third-stage feature extraction module to extract third-stage features; the third-stage features are input into the feature output module to obtain hard labels corresponding to the remote sensing image.

3. The lightweight remote sensing scene classification method based on neural network according to claim 2 is characterized in that: The shallow feature extraction module includes a two-dimensional convolution layer and a maximum pooling layer; wherein the remote sensing image is input into the two-dimensional convolution layer for convolution processing, and the features obtained after the processing are input into the maximum pooling layer after ReLU activation operation to complete downsampling, thereby obtaining the shallow features; The feature output module includes a two-dimensional convolution layer, a global pooling layer and a fully connected layer; the third-stage feature is input into the two-dimensional convolution layer for convolution processing, and the processed feature is input into the global pooling layer after ReLU activation operation to obtain a feature sequence string, and the feature sequence string is input into the fully connected layer for mapping to obtain the hard label.

4. The lightweight remote sensing scene classification method based on neural network according to claim 2 is characterized in that: In the first-stage feature extraction module, the shallow features are extracted by the hybrid attention submodule, and the extracted features are then input into the basic unit submodule to output the first-stage features; In the second-stage feature extraction module, the first-stage features are successively extracted by the two hybrid attention sub-modules, and the extracted features are then input into the two basic unit sub-modules, and finally the second-stage features are output; In the third-stage feature extraction module, after the second-stage features are extracted by the hybrid attention submodule, the extracted features are input into the basic unit submodule, and the basic unit submodule outputs the third-stage features.

5. The lightweight remote sensing scene classification method based on neural network according to claim 4 is characterized in that: In the hybrid attention submodule, the input features are divided into a first feature sequence and a second feature sequence according to the number of channels; The first feature sequence first passes through a depth-wise separable convolutional layer, and the output features are normalized and then input into a two-dimensional convolutional layer. At this time, the output features are sequentially normalized and ReLU activated to obtain the first sequence features; The second feature sequence first passes through the hybrid attention structure to extract hybrid attention features containing channel information and spatial information, the hybrid attention features are input into the depth-separable convolution layer, the output features are normalized and then input into the two-dimensional convolution layer, at this time, the output features are sequentially normalized and ReLU activated to obtain the second sequence features; After performing feature concatenation and channel shuffling on the first sequence features and the second sequence features, the features are output.

6. The lightweight remote sensing scene classification method based on neural network according to claim 5 is characterized by: In the hybrid attention structure, the input features enter the channel attention structure and the spatial attention structure in parallel; In the channel attention structure, the input features enter the average pooling layer and the maximum pooling layer respectively for channel-wise pooling to obtain two channel pooling features; After inputting the two channel pooling features into the fully connected layer, the corresponding channel attention sub-weights are obtained respectively; the result obtained by adding the corresponding elements of the two channel attention sub-weights is multiplied by the first weight coefficient, and then the multiplied result is subjected to a sigmoid activation operation to obtain the channel attention weight; the channel attention weight is multiplied by the corresponding element of the input feature to obtain the channel attention feature; In the spatial attention structure, the input features enter the average pooling layer and the maximum pooling layer respectively for pooling in the spatial direction to obtain two spatial pooling features; Input the two spatial pooling features into the multi-scale dilated convolution path respectively to obtain no less than two spatial attention sub-weights of different receptive fields; concatenate all the spatial attention sub-weights and enter the two-dimensional convolution layer for dimensionality reduction; multiply the reduced dimensionality feature by the second weight coefficient, and then perform a sigmoid activation operation on the multiplied feature to obtain the spatial attention weight; multiply the spatial attention weight by the corresponding element of the input feature to obtain the spatial attention feature; The channel attention feature and the corresponding elements of the spatial attention feature are added to obtain the mixed attention feature.

7. The lightweight remote sensing scene classification method based on neural network according to claim 6 is characterized by: In the multi-scale dilated convolution path, the input features pass through at least one two-dimensional convolution layer and at least one dilated convolution layer in parallel to obtain corresponding spatial attention sub-weights, each of which contains a corresponding receptive field information.

8. The lightweight remote sensing scene classification method based on neural network according to claim 1 is characterized by: In step S3, the output of the teacher model is processed by the softmax function to obtain the soft label, and the output of the student model is processed by the softmax function to obtain the hard label; the relative entropy is: ； in, represents the relative entropy, Represents the i-th remote sensing image in the training set The corresponding soft tag, Represents the i-th remote sensing image in the training set The corresponding hard tag; The cross entropy is: ； in, represents the cross entropy, Represents the i-th remote sensing image in the training set The corresponding true label; The loss function is: ； in, represents the loss function, Represents the loss function ratio.

9. The lightweight remote sensing scene classification method based on neural network according to claim 1, characterized in that: The preprocessing in step S1 includes the following steps: S11: performing standardization processing on the remote sensing images in the remote sensing image data set so that the remote sensing images are scaled to the same data interval and range; S12: unifying the input size of the remote sensing image; S13: Randomly select some remote sensing images for data enhancement.

Images