A remote sensing image classification system and method based on multi-head self-attention module

Abstract

The invention discloses a remote sensing image classification system and method based on a multi-head self-attention module. The framework network backbone of the classification model adopts the RestNet50 convolutional neural network. The classification model replaces the last three bottleneck layer modules of the RestNet50 convolutional neural network with the multi-head self-attention module. The model comprises: a scaling dot product attention module, which is used to perform scaling dot product attention processing on the detection image parameters input into the classification model; the multi-head self-attention module, which is used to perceive the most recognizable area in the detection image; a first normalization and residual connection module and a second normalization and residual connection module, which are used to keep the detection image parameters consistent in dimension after input into the classification model and before input into the classification model, so as to reduce the possibility of gradient disappearance in the classification model network; and a full connection module, which is used to extract features from the detection image.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a remote sensing image classification system and method based on a multi-head self-attention module. Background Art

[0002] In actual production and life, remote sensing images are used in many fields, among which remote sensing images play a vital role in urban planning, forest status assessment, ecological environment monitoring, etc. The main purpose of these applications is to accurately determine the main target categories contained in the image, such as buildings, forests or wetlands, by extracting and analyzing the features of the collected remote sensing images. At present, the main means of realizing remote sensing image classification is to use computer technology to simulate the visual perception ability of organisms, and automatically identify and classify the attributes of each pixel in the remote sensing image according to the similarities and differences of the features, such as spectral brightness, spatial structure characteristics, etc., according to a certain process or rule, so as to achieve the goal of extracting the ground object information contained in the image. In the early research on remote sensing image classification, researchers mainly chose the method of manually extracting image features to classify remote sensing images. However, with the continuous development of image acquisition technology, the methods of acquiring remote sensing images have gradually become characterized by fast acquisition speed and huge amount of information. In addition, due to the small amount of information contained in the manually extracted image features and low effectiveness, the traditional manual feature extraction method is difficult to meet the needs of the widespread application of remote sensing images. With the continuous deepening of research, researchers began to try to integrate image features extracted by different manual extraction methods. Among them, the commonly used image feature extraction methods are HOG, ORB, SIFT, etc. In the process of combining different manually extracted image features, it is necessary to constantly adjust the weight parameter size of the image features extracted by each extraction method to obtain better results. Although the given weight parameter size achieves higher accuracy in the classification of some remote sensing images, it is very difficult to use this method in a large-scale data set. Therefore, in order to solve the problems of slow speed and low accuracy in feature extraction and analysis of traditional image classification methods, fuzzy clustering classification and machine learning methods are comprehensively applied to image feature extraction and analysis methods.

[0003] In recent years, deep learning technology has been widely used in the field of computer vision. Among them, deep convolutional neural network (CNN) has significantly improved the efficiency and accuracy of image classification. Different from traditional classification methods, CNN has the characteristics of local perception and weight sharing. Local perception means that each neuron in each layer of CNN only needs to extract local pixel features from the input image of the current layer, and merge these local information in the deep layer of the network to obtain all the features of the image. At present, image recognition and classification algorithms based on CNN have made major breakthroughs in the field of computer vision.

[0004] In order to improve the classification efficiency and accuracy of remote sensing images based on existing classification methods, the present invention proposes a remote sensing image classification system and method based on a multi-head self-attention module; in view of the high accuracy of existing convolutional neural networks in remote sensing image classification, this system framework chooses to use ResNet50 as the backbone network, and replaces the middle layer of the last three bottleneck layers of ResNet50 with a multi-head self-attention module. Compared with the existing remote sensing image classification system, the present invention has significantly improved the accuracy, reduced the parameters of the framework, and improved the classification speed. Summary of the invention

[0005] The purpose of the present invention is to provide a remote sensing image classification system and method based on a multi-head self-attention module to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a remote sensing image classification method based on a multi-head self-attention module, characterized in that the workflow of the classification method is as follows:

[0007] S100: Upload a detection image, extract image parameters of the detection image and perform scaling dot product attention processing; obtain a first processing result;

[0008] S200: Connecting the plurality of first processing results obtained in step S100 to obtain a connection processing result;

[0009] S300: performing a normalization process and a residual connection on the connection processing result to obtain a second processing result;

[0010] S400: extracting feature information from the second processing result;

[0011] S500: performing a second normalization process and residual connection on the result obtained in step S400 to obtain a final image classification result;

[0012] The addition of step S200 to the above method is beneficial for capturing more feature information of the detection image when classifying the detection image, thereby improving the classification speed.

[0013] Furthermore, the image parameters of the detection images are initialized by normal random generation;

[0014] This initialization method makes the image classification more sensitive to the parameter set used in training and improves the classification accuracy of the system.

[0015] Further, the image parameters in step S100 include: a vector or matrix of a detected image, a key matrix of a detected image, and a value matrix of a detected image;

[0016] The key matrix of the detection image refers to the matrix of a certain area in the detection image; the value matrix of the detection image refers to the matrix obtained by performing dot multiplication of the vector or matrix of the detection image and the key matrix of the detection image;

[0017] The scaled dot product attention processing formula is as follows:

[0018]

[0019] Among them, Q represents the vector or matrix of the detection image; K represents the key matrix of the detection image, and K T represents the transposed matrix of the K matrix; V represents the value matrix of the detected image; represents the scaling factor;

[0020] When Softmax classifies data, if the input data is too large or too small, the recognition ability will be poor and the recognition rate will be low. The setting of this step is conducive to improving the classification ability of Softmax.

[0021] Furthermore, the process of scaling dot product attention processing in step S100 is as follows:

[0022] S101: First, the vector or matrix of the detection image, the key matrix of the detection image, and the value matrix of the detection image are subjected to multiple different linear mapping operations to obtain a matrix: Q·W i Q , K.W. i K 、V.W i V ;

[0023] S102: Matrix: Q·W i Q , K.W. i K 、V.W i V Substituting the scaled dot product attention processing formula into it, we get:

[0024]

[0025] Among them, Q.W i Q The vector or matrix representing the detected image after multiple linear mapping operations; K·W i K represents the key matrix of the detection image after multiple different linear mapping operations; V·W i V Represents the value matrix of the detected image after multiple different linear mapping operations; (K·W i K ) TIndicates K·W i K The transpose of a matrix; Represents the scaling factor.

[0026] Furthermore, the multi-head self-attention processing in step S200 includes the following process:

[0027] S210: The processing result obtained in step S100 is used as one head in the multi-head attention operation, that is:

[0028] head i =Attention(Q·W i Q , K.W. i K , V.W. i V )

[0029] Where i represents the i-th scaled dot product attention operation; head i represents the i-th head;

[0030] S220: Connect the results obtained after each scaling dot product attention process; that is:

[0031] MultiHead(Q,K,V)=Concat(head1,head2,...head i )

[0032] The processing of the multi-head self-attention module helps the classification model network capture richer features or information and alleviate the problem of gradient disappearance during the operation of the classification model.

[0033] Furthermore, both the first normalization process and the second normalization process adopt an image normalization method that keeps the aspect ratio unchanged. The normalization method process is as follows:

[0034] SS100: Get the height H of the detected image input , width W input , calculate the aspect ratio of the detection image, the formula is:

[0035]

[0036] SS200: Get the height H of the target image goal , width W goal , calculate the aspect ratio of the detection image, the formula is:

[0037]

[0038] SS300: If AspectRatio input=AspectRatio goal , use the nearest neighbor interpolation method to directly crop the detection image so that the size of the detection image is directly equal to the size of the target image; edge short =min(H input , W input )

[0039] SS400: If AspectRatio input ≠AspectRatio goal , calculate the length of both sides of the detection image, the formula is as follows:

[0040] edge short =min(H input , W input )

[0041] edge long =max(H input , W input )

[0042] Among them, edge short Indicates the length of the short side, edge long represents the length of the long side;

[0043] SS500: The short side length is used as a measure in the long side length to calculate the step length, and an image with a length equal to the step length value is intercepted. The detected image is converted into a small image with the same aspect ratio as the target image. The step length calculation formula is as follows:

[0044]

[0045] SS600: Perform geometric transformation on the small image obtained in step SS500 and normalize it so that the final image has the same size as the target image;

[0046] The process of normalization operation is mainly to convert the input parameters into the range of 0-1 to avoid too large difference in the order of magnitude of the input parameters. Normalizing the parameters is conducive to subsequent parameter processing and utilization.

[0047] Furthermore, the feature extraction in step S400 applies the following formula:

[0048] FNN(x)=max(0,W1·x+b1)·W2+b2

[0049] Wherein, b1 and b2 represent the first deviation parameter and the second deviation parameter respectively, and x is the second processing result obtained in step S300.

[0050] To implement the above method, the present invention also provides a remote sensing image classification system based on a multi-head self-attention module, characterized in that the classification system includes:

[0051] The scaled dot product attention module is used to perform scaled dot product attention processing on the detection image parameters of the input classification model;

[0052] Multi-head self-attention module, which is used to perceive the most recognizable areas in the detection image;

[0053] A first normalization and residual connection module and a second normalization and residual connection module, wherein the first normalization and residual connection module and the second normalization and residual connection module are used to keep the detection image parameters consistent in dimension after being input into the classification system and before being input into the classification system, thereby reducing the possibility of gradient disappearance in the network of the classification system;

[0054] Fully connected module, the fully connected module is used to extract features from the detection image;

[0055] The setting of the fully connected module can connect the input and output in this classification model, and the fully connected module has the characteristics of high throughput, high reliability and low latency.

[0056] Compared with the prior art, the beneficial effects achieved by the present invention are: the present invention selects to use ResNet50 as the backbone network in the system framework, and replaces the middle layer of the last three bottleneck layers of ResNet50 with a multi-head self-attention module. Compared with the existing remote sensing image classification system, the classification system proposed in this patent has significantly improved accuracy, reduced system parameters, and improved the system classification speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0058] Figure 1 It is a structural diagram of a remote sensing image classification system based on a multi-head self-attention module;

[0059] Figure 2 It is a workflow diagram of a remote sensing image classification method based on a multi-head self-attention module;

[0060] Figure 3 It is a schematic diagram of the comparison results of the accuracy of different parameter sets in different classification systems;

[0061] Figure 4 is a schematic diagram of the number of parameters included in different classification systems. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The "softmax" logistic regression model is a generalization of the logistic regression model for multi-classification problems, where the class label y can take on more than two values. Softmax regression models are useful for problems such as MNIST handwritten digit classification, where the goal is to identify 10 different individual digits. Softmax regression is supervised, but we will also introduce its combination with deep learning unsupervised learning methods later.

[0064] "FNN" feedforward neural network is the most basic and classic form of neural network. Its structure includes input layer, hidden layer and output layer. According to the number of hidden layers, it is divided into shallow network and deep network.

[0065] The loss function "Loss(y,k)" is used to measure the gap between the predicted category and the true category. The "cross entropy loss function" is a commonly used loss function. Cross entropy is used to judge the degree of closeness between the actual output and the expected output. It describes the distance between two probability distributions.

[0066] See also Figure 1-4 The present invention provides a technical solution: a remote sensing image classification method based on a multi-head self-attention module, characterized in that the workflow of the classification method is as follows:

[0067] S100: Upload a test image, extract image parameters of the test image and perform scaled dot product attention processing; obtain a first processing result; the image parameters of the test image are initialized by normal random generation; this initialization method makes the image classification more sensitive to the parameter set used in training, and improves the classification accuracy of the system;

[0068] The image parameters include: a vector or matrix of a detection image, a key matrix of a detection image, and a value matrix of a detection image;

[0069] The key matrix of the detection image refers to the matrix of a certain area in the detection image; the value matrix of the detection image refers to the matrix obtained by performing dot multiplication of the vector or matrix of the detection image and the key matrix of the detection image;

[0070] The scaled dot product attention processing formula is as follows:

[0071]

[0072] Among them, Q represents the vector or matrix of the detection image; K represents the key matrix of the detection image, and K T represents the transposed matrix of the K matrix; V represents the value matrix of the detected image; represents the scaling factor;

[0073] The processing procedure of the scaled dot product attention processing in step S100 is as follows:

[0074] S101: First, the vector or matrix of the detection image, the key matrix of the detection image, and the value matrix of the detection image are subjected to multiple different linear mapping operations to obtain a matrix: Q·W i Q , K.W. i K 、V.W i V ;

[0075] S102: Matrix: Q·W i Q , K.W. i K 、V.W i V Substituting into the scaled dot product attention processing formula we get:

[0076]

[0077] Among them, Q.W i Q The vector or matrix representing the detected image after multiple linear mapping operations; K·W i K represents the key matrix of the detection image after multiple different linear mapping operations; V·W i V Represents the value matrix of the detected image after multiple different linear mapping operations; (K·W i K ) T Indicates K·W i K The transpose of a matrix; represents the scaling factor;

[0078] S200: Connecting the plurality of first processing results obtained in step S100 to obtain a connection processing result;

[0079] Among them, multi-head self-attention processing includes the following processes:

[0080] S210: The processing result obtained in step S100 is used as one head in the multi-head attention operation, that is:

[0081] head i =Attention(Q·W i Q , K.W. i K , V.W. i V )

[0082] Where i represents the i-th scaled dot product attention operation; head i represents the i-th head;

[0083] S220: Connect the results obtained after each scaling dot product attention process; that is:

[0084] MultiHead(Q,K,V)=Concat(head1,head2,…head i )

[0085] S300: performing a normalization process and a residual connection on the connection processing result to obtain a second processing result;

[0086] Among them, the first normalization process and the second normalization process both adopt the image normalization method of keeping the aspect ratio unchanged, and the normalization method process is as follows:

[0087] SS100: Get the height H of the detected image input , width W input , calculate the aspect ratio of the detection image, the formula is:

[0088]

[0089] SS200: Get the height H of the target image goal , width W goal , calculate the aspect ratio of the detection image, the formula is:

[0090]

[0091] SS300: If AspectRatio input =AspectRatio goal , use the nearest neighbor interpolation method to directly crop the detection image so that the size of the detection image is directly equal to the size of the target image; edge short =min(H input , W input )

[0092] SS400: If AspectRatio input ≠AspectRatio goal , calculate the length of both sides of the detection image, the formula is as follows:

[0093] edge short =min(H input , W input )

[0094] edge long =max(H input , W input )

[0095] Among them, edge short Indicates the length of the short side, edge long represents the length of the long side;

[0096] SS500: The short side length is used as a measure in the long side length to calculate the step length, and an image with a length equal to the step length value is intercepted. The detected image is converted into a small image with the same aspect ratio as the target image. The step length calculation formula is as follows:

[0097]

[0098] SS600: Perform geometric transformation on the small image obtained in step SS500 and normalize it so that the size of the final image is consistent with the target image;

[0099] S400: extracting feature information from the second processing result;

[0100] Among them, feature extraction applies the following formula:

[0101] FNN(x)=max(0,W1·x+b1)·W2+b2

[0102] Wherein, b1 and b2 represent the first deviation parameter and the second deviation parameter respectively, and x is the second processing result obtained in step S300.

[0103] S500: performing a second normalization process and residual connection on the result obtained in step S400 to obtain a final image classification result.

[0104] To implement the above method, the present invention also provides a remote sensing image classification system based on a multi-head self-attention module, characterized in that: the framework network backbone of the classification system adopts the RestNet50 convolutional neural network, and the classification system replaces the last three bottleneck layer modules of the RestNet50 convolutional neural network with a multi-head self-attention module;

[0105] The classification systems include:

[0106] The scaled dot product attention module is used to perform scaled dot product attention processing on the detection image parameters of the input classification model;

[0107] Multi-head self-attention module, which is used to perceive the most recognizable areas in the detection image;

[0108] A first normalization and residual connection module and a second normalization and residual connection module, wherein the first normalization and residual connection module and the second normalization and residual connection module are used to keep the detection image parameters consistent in dimension after being input into the classification system and before being input into the classification system, thereby reducing the possibility of gradient disappearance in the network of the classification system;

[0109] Fully connected module, the fully connected module is used to extract features from the detection image;

[0110] The setting of the fully connected module can connect the input and output in this classification model, and the fully connected module has the characteristics of high throughput, high reliability and low latency.

[0111] The workflow of the classification system is as follows:

[0112] SS100: Upload the detection image to the classification system, first transfer the detection image to the scaled dot product attention module, the scaled dot product attention module extracts the image parameter matrix of the detection image and performs scaled dot product attention processing;

[0113] SSS200: The scaled dot product attention module inputs the processed image information matrix into the multi-head self-attention module, and the multi-head self-attention module concatenates the results obtained after each scaled dot product attention module processing;

[0114] SSS300: The multi-head self-attention module inputs the connection processing results into the first normalization and residual connection module for the first normalization processing and residual connection;

[0115] SSS400: The first normalization processing and residual connection module inputs the operation results into the fully connected module for feature information extraction;

[0116] SSS500: The fully connected module inputs the feature information into the second normalization processing and residual connection module for the second normalization processing and residual connection to obtain the final image classification result.

[0117] In the process of training the classification system, the cross entropy loss function is used to calculate the loss value of the classification system framework on the input parameter set;

[0118] The formula is as follows:

[0119]

[0120] Among them, the variable y is the category prediction probability output by the classification model based on the input parameters, y is a tensor, y[j] refers to the jth dimension of y, that is, the probability that the input image is predicted to be the jth category; the variable k is the actual category number of the input image.

[0121] like Figure 3 As shown, the system framework proposed in the present invention has achieved the highest classification accuracy (91.310%, 97.877%) on the RSSCN7 and EuroSAT data sets compared with several existing classification systems. And compared with the backbone network ResNet50 framework used in the system framework proposed in the present invention, the classification accuracy has also been significantly improved. On the PatternNet parameter set, the classification accuracy of the system framework proposed in the present invention is close to that of ResNet50, but compared with the ViT classification system, the classification accuracy of the system framework is lower; after analysis, it is found that since the input image size of the ViT classification system is 224×224, and the input size of the system framework proposed in this patent is 200×200, the downsampling of the image causes the image to lose some features, making the classification accuracy of the framework proposed in the present invention lower than that of the ViT system. It can be concluded that the system framework proposed in the present invention can more accurately perceive the most discriminative areas in the image, making the classification accuracy more accurate.

[0122] like Figure 4 As shown, the classification system framework proposed in the present invention contains 20.8×106 parameters, and the ResNet50 framework contains 25.5×106 parameters. After using ResNet50 as the backbone network and replacing part of its structure with a multi-head self-attention module, the number of parameters has been significantly reduced, reducing the overhead of video memory and the amount of computation during runtime; and the system framework proposed in the present invention has the best classification effect on the RSSCN7 and EuroSAT parameter sets, and both the number of parameters and the classification speed have been optimized. Compared with the existing remote sensing image classification system framework, the system framework proposed in the present invention can achieve higher-precision classification effects in a GPU-accelerated environment, reduce the parameters contained in the system framework, and reduce the occupancy of video memory.

[0123] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0124] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image classification method based on a multi-head self-attention module, characterized in that: The workflow of the classification method is as follows: S100: Upload a detection image, extract image parameters of the detection image and perform scaling dot product attention processing; Obtaining a first processing result; the image parameter initialization of the detection image adopts a normal random generation method; S200: Connecting the plurality of first processing results obtained in step S100 to obtain a connection processing result; S300: performing a normalization process and a residual connection on the connection processing result to obtain a second processing result; S400: extracting feature information from the second processing result; S500: performing a second normalization process and residual connection on the result obtained in step S400 to obtain a final image classification result; The image parameters in step S100 include: a vector or matrix of a detected image, a key matrix of a detected image, and a value matrix of a detected image; The key matrix of the detection image refers to the matrix of a certain area in the detection image; the value matrix of the detection image refers to the matrix obtained by performing dot multiplication of the vector or matrix of the detection image and the key matrix of the detection image; The scaled dot product attention processing formula is as follows: Among them, Q represents the vector or matrix of the detection image; K represents the key matrix of the detection image, and represents the transposed matrix of the K matrix; V represents the value matrix of the detected image; represents the scaling factor; The processing process of the scaled dot product attention processing in step S100 is as follows: S101: First, the vector or matrix of the detection image, the key matrix of the detection image, and the value matrix of the detection image are subjected to multiple different linear mapping operations to obtain a matrix: , , ; S102: transform the matrix: , , Substituting the scaled dot product attention processing formula into the formula, we get: in, A vector or matrix representing the detected image after multiple different linear mapping operations; The key matrix represents the detection image after multiple different linear mapping operations; A value matrix representing the detected image after multiple different linear mapping operations; Indicates the The transpose of a matrix; represents the scaling factor; The first normalization process and the second normalization process both adopt an image normalization method that keeps the aspect ratio unchanged. The normalization method process is as follows: SS100: Get the height of the detected image ,width , calculate the aspect ratio of the detection image, the formula is: SS200: Get the height of the target image ,width , calculate the aspect ratio of the detection image, the formula is: SS300: If , use the nearest neighbor interpolation method to directly crop the detection image so that the size of the detection image is directly equal to the size of the target image; SS400: If , calculate the length of both sides of the detection image, the formula is as follows: in, Indicates the length of the short side, represents the length of the long side; SS500: The step length is calculated by taking the short side length as a measure in the long side length, and an image with a length equal to the step length value is intercepted, and the detection image is converted into a small image with the same aspect ratio as the target image. The step length is calculated as follows: SS600: Perform geometric transformation on the small image obtained in step SS500 and normalize it so that the final image is consistent with the size of the target image.

2. The remote sensing image classification method based on a multi-head self-attention module according to claim 1, characterized in that: The multi-head self-attention processing in step S200 includes the following process: S210: Using the processing result obtained in step S100 as one head in the multi-head attention operation, that is: Among them, i represents the i-th scaled dot product attention operation; represents the i-th head; S220: Connect the results obtained after each scaling dot product attention process; that is, .

3. The remote sensing image classification method based on a multi-head self-attention module according to claim 1, characterized in that: The feature extraction in step S400 uses the following formula: in, , represent the first deviation parameter and the second deviation parameter respectively, and x is the second processing result obtained in step S300.

4. A remote sensing image classification system applied to a remote sensing image classification method based on a multi-head self-attention module as described in any one of claims 1 to 3, characterized in that: The classification system includes: A scaled dot product attention module, the scaled dot product attention module is used to perform scaled dot product attention processing on the detection image parameters input to the classification model; A multi-head self-attention module, the multi-head self-attention module is used to perceive the most recognizable area in the detection image; A first normalization and residual connection module and a second normalization and residual connection module, wherein the first normalization and residual connection module and the second normalization and residual connection module are used to keep the detection image parameters consistent in dimension after being input into the classification system and before being input into the classification system, so as to reduce the possibility of gradient disappearance in the network of the classification system; A fully connected module, wherein the fully connected module is used to extract features from the detection image.

Images