A multi-feature fusion SAR image sea ice classification method based on a self-attention mechanism

Abstract

The present application relates to the technical field of remote sensing scene classification, and provides a multi-feature fusion SAR image sea ice classification method based on a self-attention mechanism.The method can effectively extract the global features of the optimal texture feature image by using the self-attention mechanism, specifically as follows: the contribution degrees of different features are determined according to the classification accuracy of the image texture features of a support vector machine; the first three texture features with high contribution degrees to the classification accuracy are combined to form a three-channel image, which is used for training a Transform Encoder model; and the global features of the sea ice SAR image extracted by the self-attention mechanism of the model are used for classification, so that the final classification result is obtained.The method can improve the classification accuracy of the SAR image sea ice by using the self-attention mechanism to extract the global features of the optimal image texture, and is easy to implement and scientific and reasonable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing scene classification, and particularly relates to a multi-feature fusion SAR image sea ice classification method based on a self-attention mechanism. BACKGROUND

[0002] Sea ice is an important part of the climate system, has an important influence on the global climate and environment, and is one of the most sensitive factors reflecting global climate change. On the other hand, the freezing and drifting of sea ice has a destructive impact on marine shipping, offshore oil and gas exploration, and other offshore production activities, and even causes serious disasters, causing incalculable economic losses to the coastal economy.

[0003] Satellite remote sensing technology is the most efficient way of sea ice monitoring, has the advantages of large range, real-time, sustainability, and low cost, and synthetic aperture radar (SAR) is not affected by climate and time and has become the main way of sea ice monitoring.

[0004] Previous studies have shown through experiments that different texture features can assist in sea ice classification, for example, after extracting mean, angular second moment, entropy, contrast, and contrast difference matrix texture features, using random forest or support vector machine method to classify the sea ice scene. However, the importance of different texture features is not utilized, and the self-attention mechanism is not used to extract global features. SUMMARY

[0005] Based on the above background technology, the purpose of the present application is to provide a multi-feature fusion SAR image sea ice classification method based on a self-attention mechanism, which can perform feature optimization on the texture features of the image, and use the self-attention mechanism to extract the global features of the image, thereby further improving the sea ice classification accuracy of the SAR image.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A multi-feature fusion SAR image sea ice classification method based on a self-attention mechanism, characterized by the following steps:

[0008] 1) Process the SAR remote sensing image data, i.e. the public data set SI-STSAR-7, and adjust the original image to 224*224 size.

[0009] 2) Extract five texture features of the adjusted image based on the gray level co-occurrence matrix: mean, angular second moment, entropy, contrast, and contrast difference matrix; the specific calculation formulas of each texture feature are as follows:

[0010] Mean (Mean):

[0011]

[0012] Angular Second Moment (ASM):

[0013]

[0014] Entropy (ENT):

[0015]

[0016] Constrast (Con):

[0017]

[0018] Inverse Differential Moment (IDM):

[0019]

[0020] where P(i, j) represents the number or frequency of two pixels with gray levels i and j respectively appearing at a given spatial distance and direction;

[0021] 3) The five texture feature maps of mean, angular second moment, entropy, constrast and inverse differential moment obtained are taken as input features, all of which are split into training set and test set at a ratio of 8:2, and are classified using support vector machine method, and according to the classification result, three of the five texture feature maps with higher accuracy are selected to form a three-channel feature map.

[0022] 4) The three-channel feature map obtained is divided into sub-images of fixed size, each sub-image is called a patch, one patch corresponds to one token, and then the dimension thereof is adjusted to be appropriate for the input size of the model, and the encoding of each patch, i.e. the image embedding vector, is obtained, which is specifically as follows:

[0023] Taking an input IMG(batch, 3, 224, 224) and a patch resolution of 16*16 as an example, the specific process of the sub-image division process is as follows: first, each image is divided into 14*14 small blocks, that is, from IMG(batch, 3, 16*14, 16*14) to IMG(batch, 3, 14*14, 16*16), and then converted into IMG(batch, 14*14, 16*16*3), the dimension of each patch is 16*16*3, which is equivalent to being divided into 14*14 patches, that is, 14*14 tokens, and the length of each token vector is 16*16*3. In addition, a token of a class needs to be added, which is actually initializing a trainable parameter [1, 16*16*3], which is spliced with the token sequence. The dimension of each token changes from [14*14, 16*16*3] to [14*14+1, 16*16*3];

[0024] wherein batch is the batch size, that is, the number of images input at a time;

[0025] After dividing the sub-image, the embedding vector of the image is obtained, and the dimension thereof is adjusted. The image embedding vector is adjusted to 1024 by a fully connected layer, so that it will not be too long as the input of the model.

[0026] 5) The obtained three-channel feature image is encoded in a two-dimensional position using a sincos encoding method to obtain a two-dimensional position encoding vector. The 2D sincos encoding method is used, and the encoding formula is:

[0027]

[0028]

[0029] wherein PE is a two-dimensional matrix, the row represents a sub-image, and the column represents a position vector; pos represents the position of the sub-image in the image; d model represents the dimension of the position vector; i represents the position of the position vector;

[0030] The two-dimensional encoding method is to represent half of the two-dimensional position encoding dimension using horizontal sincos encoding and the other half using vertical sincos encoding. After encoding, the position encoding vector is added to each element of the image embedding vector obtained in the previous step, and the input vector X of the model is obtained.

[0031] 6) Construct a Transformer Encoder model, the structure of the Transformer Encoder model is composed of a plurality of Transformer Encoder Blocks stacked 6 times, the Transformer Encoder Block is mainly composed of multi-head self-attention calculation and a feedforward neural network layer FNN, residual connection is used in the middle, which is used for the extraction of attention of each part of the image, the calculation formula of the multi-head self-attention is as follows:

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] MultiHead(Q,K,V)=Concat(head1,…,head h )w 0

[0038] Wherein, Q, K, V are input vectors of the model, d is the dimension of the input vector Q, head i is the vector of the i-th head, w Q , w K , w V are translation model parameters, Concat is a vector connection function, Attention is an attention mechanism function, and softmax is a normalized exponential function;The calculation formula of the feedforward neural network layer FNN is as follows:

[0039] FNN(x)=max(0,xW1+b1)W2+b2

[0040] Wherein, x is a hidden layer vector, W1, W2, b1 and b2 are parameters of the model;

[0041] The input vector X is sent into the Transformer Encoder network model, the multi-head self-attention calculation layer in the model extracts image space features, the model output features are obtained through the feedforward neural network layer to obtain a classification probability vector, the classification result obtained by the model classifier unit is compared with the real result, parameter adjustment is carried out, and the model is trained until the model loss function value is less than a given threshold, finally, the vector output by the model is converted into a probability representation through the classifier unit during decoding, and image classification is completed.

[0042] The present application has the beneficial effects that:

[0043] The method considers the importance of different texture features of the image and performs feature optimization; three selected texture features with high contribution to classification accuracy are combined to form a three-channel feature image, which is introduced into a Transformer Encoder model for training, and the training is completed after the model converges, and the classification result of the input image is obtained. The method uses a self-attention mechanism to extract the global features of the optimized image texture, which can improve the accuracy of SAR image sea ice classification, is scientific and reasonable, and is easy to implement. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A step flowchart for implementing the present application is shown in the following table:

[0045] Figure 2 SAR sea ice images and five extracted texture features. DETAILED DESCRIPTION

[0046] The present application will be further described below in conjunction with the accompanying drawings of the specification:

[0047] In combination Figure 1 A multi-feature fusion SAR image sea ice classification method based on a self-attention mechanism includes the following steps:

[0048] 1) Process the SAR remote sensing image data, i.e., the public data set SI-STSAR-7, and adjust the original image to 224*224 size.

[0049] 2) Extract five texture features of the adjusted image based on the gray level co-occurrence matrix: mean, angular second moment, entropy, contrast, and contrast difference matrix; the specific calculation formulas of each texture feature are as follows:

[0050] Mean (Mean):

[0051]

[0052] Angular Second Moment (ASM):

[0053]

[0054] Entropy (ENT):

[0055]

[0056] Constrast (Con):

[0057]

[0058] Inverse Differential Moment (IDM):

[0059]

[0060] where P(i,j) represents the frequency or occurrence of two pixels with gray levels i and j respectively at a given spatial distance and direction.

[0061] 3) The five texture feature maps of mean, angular second moment, entropy, contrast, and inverse differential moment obtained are all split into training set and data set at a ratio of 8:2, and support vector machine method is used for classification, Kappa coefficient is calculated according to the confusion matrix of the classification result, Kappa coefficients obtained from the classification results of the five texture feature maps are compared, and the three with higher accuracy are selected to form a three-channel feature map, and the calculation formula of Kappa coefficient is as follows:

[0062]

[0063]

[0064] where p0 is the sum of the number of correctly classified samples of each class divided by the total number of samples, a1, a2, …, ai are the number of real samples of each class, b1, b2, …, bi are the number of predicted samples of each class, and n is the total number of samples.

[0065] 4) In step 4), the three-channel feature map obtained is divided into sub-images of fixed size, each sub-image is called a patch, and each patch corresponds to a token, and then the dimensions are adjusted to be suitable for the input size of the model, and the encoding of each patch, i.e. image embedding vector, is obtained, which is specifically:

[0066] Taking input IMG(batch, 3, 224, 224) and each patch resolution of 16*16 as an example, the specific process of the sub-image division process is as follows: first, each image is divided into 14*14 small blocks, i.e. from IMG(batch, 3, 16*14, 16*14) to IMG(batch, 3, 14*14, 16*16), and then converted to IMG(batch, 14*14, 16*16*3), the dimension of each patch is 16*16*3, which is equivalent to being divided into 14*14 patches, i.e. 14*14 tokens, and the length of each token vector is 16*16*3. In addition, a token of a class is needed, which is actually initializing a trainable parameter [1, 16*16*3], which is concatenated with the token sequence. The dimension of each token is changed from [14*14, 16*16*3] to [14*14+1, 16*16*3].

[0067] where batch is the batch size, i.e. the number of images input at once.

[0068] After dividing the sub-image, the embedding vector of the image is obtained, and its dimension is adjusted. The image embedding vector is adjusted to 1024 dimensions through a fully connected layer, so that it will not be too long as the input of the model.

[0069] 5) The obtained three-channel feature image is encoded using the sincos encoding method for two-dimensional position encoding, and the encoding formula is:

[0070]

[0071]

[0072] where PE is a two-dimensional matrix, the row represents the sub-image, and the column represents the position vector; pos represents the position of the sub-image in the image; d model represents the dimension of the position vector; i represents the position of the position vector;

[0073] The two-dimensional encoding method is to use the horizontal sincos encoding to represent half of the two-dimensional position encoding, and the other half is represented by the vertical sincos encoding. After encoding, the position encoding vector is added to each element of the image embedding vector in the previous step, and the input vector X of the model is obtained.

[0074] 6) Construct the Transformer Encoder model, the structure of the Transformer Encoder model is composed of multiple Transformer Encoder Blocks stacked 6 times, and the Transformer Encoder Block is mainly composed of multi-head self-attention calculation and feedforward neural network layer FNN, using residual connection in the middle, used for attention extraction of each part of the image. The calculation formula of multi-head self-attention is as follows:

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] MultiHead(Q,K,V)=Concat(head1,…,head h )w 0

[0081] wherein Q, K, V are input vectors of the model, d is the dimension of input vector Q, head i is the vector of the i-th head, w Q , w K , w V is the translation model parameter, Concat is the vector connection function, Attention is the attention mechanism function, and softmax is the normalization exponential function; the calculation formula of the feedforward neural network layer FNN is as follows:

[0082] FNN(x) = max(0, xW1 + b1)W2 + b2

[0083] wherein x is the hidden layer vector, W1, W2, b1, b2 are parameters of the model.

[0084] The input vector X is input into the Transformer Encoder model, the classification probability vector is obtained by the model output feature through the feedforward neural network layer, the classification result obtained by the model classifier unit is compared with the real result, the model is optimized, the model training is repeated, and the process is repeated until the model loss function value is less than 10 -5 , and the training is ended; after the training is ended, the model can output the classification result corresponding to the input image after one decoding.

[0085] The method uses the self-attention mechanism, extracts the global features of the preferred texture feature image, and improves the precision of SAR image sea ice classification.

[0086] Of course, the above description is not a limitation of the present application, and the present application is not limited to the above examples, and the changes, modifications, additions or replacements made by the person skilled in the art within the essential scope of the present application should also belong to the protection scope of the present application.

Claims

1. A multi-feature fusion SAR image sea ice classification method based on self-attention mechanism, characterized in that... Includes the following steps: 1) Process SAR remote sensing image data and adjust the original image size to 224*224; 2) Extract five texture features of the adjusted image based on the gray-level co-occurrence matrix: mean, second angular moment, entropy, contrast, and inverse difference matrix; 3) The five texture feature maps obtained—mean, second moment of angle, entropy, contrast, and inverse difference matrix—are used as input features. The support vector machine method is used for classification. Based on the classification results, the three texture feature maps with the higher accuracy are selected to form a three-channel feature map. 4) Divide the obtained three-channel feature image into sub-images of fixed size, connect the sub-images and adjust their dimensions to form an image embedding vector; 5) The obtained three-channel feature image is encoded in two dimensions using sincos encoding to obtain a two-dimensional positional encoding vector. This vector is then added to the image embedding vector to obtain the input vector X. 6) Construct a Transformer Encoder model. Feed the input vector X into the Transformer Encoder network model. The multi-head self-attention calculation layer in the model extracts the spatial features of the image. Parameters are tuned and trained until the model loss function is less than a given threshold. Finally, during decoding, the classifier unit converts the vector output by the model into a probability representation to complete image classification. In step 3), the five texture feature maps obtained in step 2) are split into training and testing sets in an 8:2 ratio. The support vector machine method is used for classification. Based on the Kappa coefficient of the classification results, the three texture feature maps with higher accuracy are selected to form a three-channel feature map. In step 4), the obtained three-channel feature map is divided into sub-images of fixed size. Each sub-image is called a patch, and each patch corresponds to a token. The dimensions are then adjusted to be suitable for the input size of the model to obtain the encoding of each patch, i.e., the image embedding vector. After dividing the sub-images, the image embedding vector is obtained, and its dimensions are adjusted. The image embedding vector is passed through a fully connected layer to adjust the dimensions to the required length.

2. The sea ice classification method for multi-feature fusion SAR images based on self-attention mechanism according to claim 1, characterized in that: In step 5), a 2D sincos encoding method is used. The 2D encoding method means that half of the two-dimensional position encoding is represented by horizontal sincos encoding and the other half by vertical sincos encoding. After the encoding is completed, the position encoding vector is added to the corresponding elements of the image embedding vector in the previous step to obtain the input vector X of the model.

3. The sea ice classification method for multi-feature fusion SAR images based on self-attention mechanism according to claim 1, characterized in that... In step 6), the Transformer Encoder model is used for image classification, as follows: A Transformer Encoder model is constructed, consisting of multiple Transformer Encoder Blocks stacked L times. Each Transformer Encoder Block mainly comprises multi-head self-attention computation and a feedforward neural network layer (FNN), with residual connections used in between for extracting attention from different parts of the image. The multi-head self-attention computation is as follows: ； ； ； ； ； ； in, For the model input vector, For input vectors Dimensions For the first A vector of heads, For translation model parameters, For vector concatenation functions, For attention mechanism functions, It is a normalized exponential function; The calculation of the feedforward neural network layer FNN is as follows: ； in, For hidden layer vectors, These are the parameters of the model; The input vector X is fed into the Transformer Encoder model. The model output features are processed by a feedforward neural network layer to obtain a classification probability vector. The classification result obtained by the model classifier unit is compared with the true result to optimize the model. This process is repeated to train the model until the model loss function value is less than a given threshold, at which point training ends. After training, the model can output the classification result of the corresponding input image after one decoding.

Images