Satellite image semantic segmentation method and system based on full convolutional neural network and transformer

Abstract

The application is based on a satellite image semantic segmentation method and system of a full convolutional neural network and a transformer, and the method comprises the following steps: 1) performing defogging and denoising processing on a satellite image; 2) extracting satellite image features by using a multi-layer, i.e., depth down-sampling convolutional neural network; 3) optimizing satellite image features by using a transformer with a self-attention mechanism; and 4) realizing semantic segmentation of the satellite image by using up-sampling. The application uses an auto-encoder to perform defogging and denoising on an image, so that the satellite image is clearer. While the convolutional network extracts satellite image features, a self-attention network of a transformer is designed, and the satellite image is further extracted by using weight distribution. Finally, the feature map of the self-attention layer is up-sampled to obtain a segmentation of the original image size. Moreover, the prediction of the last layer which is rich in global information and the prediction of the shallower layer which is rich in local information are combined through a skip-level structure, so that the global prediction is obeyed while the local prediction is performed, and accurate semantic segmentation is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of satellite image pattern segmentation and recognition based on deep learning, and particularly relates to an effective segmentation and recognition strategy, namely a satellite image semantic segmentation method and system based on a full convolutional neural network and a transformer. BACKGROUND

[0002] The processing of satellite remote sensing images plays an extremely important role in national economic construction and national defense construction. The satellite images obtained after processing of the remote sensing images contain a large amount of information, and the data volume is large and the data types are complex and diverse. Traditional target detection and recognition methods are difficult to adapt to massive data, rely on manual annotation, are very time-consuming and labor-intensive, and are strongly dependent on professional knowledge and the characteristics of the data itself.

[0003] The method based on deep learning provides a very effective feature extraction framework, so that it is widely used in the recognition and classification of satellite remote sensing images. The pixel-level semantic segmentation of the image is to one-to-one correspond the pixels of the image to the classification of the image, and is an algorithm for identifying each pixel of the image to understand the pixel level of the image.

[0004] The template matching method and the background modeling method used in the current common ship recognition method have high uncertainty and unsatisfactory effect. At the same time, due to the relatively small size of the ships in the satellite image, the cloud and fog obstruction, the land background interference and other problems, the automatic recognition of the ships in the satellite image still faces many challenges. At the same time, in the existing semantic segmentation convolutional network model, the step of pooling is too large, which is easy to ignore the details of the satellite image, and the network depth is too deep, which is not convenient for training. The existing network training has a sufficient learning ability of the stacked encoder-decoder structure, which can bring good positioning recovery, but the posterior feature completely depends on the priori, lacks the diversity and flexibility of the feature, and the size of the experience receptive field is not proportional to the depth of the network. SUMMARY

[0005] In view of the deficiencies of the semantic segmentation technology in the existing image recognition, the application proposes a satellite image semantic segmentation method and system combining a full convolutional neural network and a transformer. The application first uses an auto-encoder to de-fog the satellite image, then extracts the image features by a convolutional neural network, further optimizes the features of the satellite image, and the full convolution algorithm can expand the image by any multiple to realize semantic segmentation.

[0006] Convolutional neural networks exploit the prior knowledge of strong local pixel correlation within images, while transformer architectures do not contain built-in inductive priors on the local interactions, so they can freely learn complex relationships between their inputs through self-attention mechanisms. To complete the task of semantic segmentation, a powerful backbone network is needed as an editor. Unlike image classification, which identifies individual objects, semantic segmentation requires processing objects of different sizes in a single image, which requires the network to interact with multi-scale information. In existing network designs for semantic segmentation, convolutional neural networks achieve feature extraction of images through the learning of convolution kernels. The ability of self-attention mechanisms to extract important information in image semantic segmentation has received little attention. The present application combines convolutional neural networks and transformer technology and uses deconvolution upsampling to achieve semantic segmentation of satellite images, thereby achieving image recognition.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] The satellite image semantic segmentation method based on full convolutional neural network (down-sampling convolution and up-sampling deconvolution neural network) and transformer relates to a network for training end-to-end, pixel-to-pixel semantic segmentation, comprising the following specific steps:

[0009] Step 1), performing defogging and denoising processing on the satellite image;

[0010] Step 2), extracting satellite image features using a multi-layer, i.e. deep, down-sampling convolutional neural network;

[0011] Step 3), optimizing satellite image features using a transformer with a self-attention mechanism;

[0012] Step 4), achieving semantic segmentation of the satellite image using up-sampling deconvolution (full convolution is achieved by this point).

[0013] Preferably, in step 1), a denoising autoencoder is designed, the places where the original image pixel signal is covered by clouds are regarded as noise, the signal (pixel value) containing noise is input into the denoising autoencoder, the error between the output and the original signal is calculated, and the random gradient descent algorithm is used to adjust the weight to minimize the error, thereby achieving defogging of the satellite image.

[0014] Further preferably, in step 1), an autoencoder is used to remove haze and denoise the satellite image. The denoising autoencoder treats pixels covered by clouds as noise, and this noisy signal is input into the denoising autoencoder. An autoencoder is a type of artificial neural network (ANN) used in semi-supervised and unsupervised learning. It uses input information as a learning target and performs representation learning on the input information, identifying patterns between adjacent pixels.

[0015] The improved autoencoder recognition network is designed with two hidden layers to enhance learning ability and image dehazing effect. It receives input and converts it into an efficient internal representation. The generative network with two hidden layers outputs something similar to the input data, that is, converts the internal representation into output. The basic mathematical process is:

[0016] The pixel matrix on the satellite image is converted into a vector as the input of the automatic encoder recognition network. Its input is x = [x(1), x(2) ... x(d)], d is the dimension, and the encoder projects x from the input layer to the first hidden layer h1 through the mapping function f, h1 = [h1(1), h1(2) ... h1(d h1 )]. f is: h1=f(x)=ReLu(W1x+b1). Where W1 is h1(d h1 )×d weight matrix, b1 is the bias vector. Then h1 is projected from the first hidden layer to the second hidden layer h2 through the mapping function f, h2=[h2(1),h2(2)……h2(d h2 )]. f is: h2=f(x)=ReLu(W2 h2+b2). ​​Where W2 is h2(d h2 )×h1 weight matrix, b1 is the bias vector. The activation function ReLu of the decoder is the rectified linear unit function. Through the mapping function Map h2 represented by the hidden layer to the output layer ,in The function is , using stochastic gradient descent algorithm to adjust the weights W1, W2, Adjustment is performed by using the error back propagation algorithm to back propagate the gradient information of the error loss function and update all model parameters at the same time. Here the loss function is assumed to be

[0017]

[0018] The updated network weight parameters are

[0019]

[0020] p is the learning rate, i represents the i-th neuron. The output can also be reconstructed by limiting the number of hidden layer units As much as possible similar to the input x, so the image is completed by defogging and denoising.

[0021] Preferably, step 2) is specifically as follows: the output vector of the autoencoder is changed into a matrix as the input of the deep convolutional network, and then the backbone network of the deep convolutional neural network is used, the backbone network including two convolutional layers and two pooling layers, and the pooling layer adopts the average pooling rule. For an m x m image, the convolution calculation formula C = A * B is specifically as follows

[0022]

[0023] The convolution between the image and the filter (convolution kernel) is calculated. The parameters in the convolution kernel are obtained through learning as weight coefficients.

[0024] As a further preferred, the relevant design parameters are as follows:

[0025] The first layer is a convolutional layer, the input is the pixel matrix of the defogged picture (length, width, and channel number (red, green, and blue)), and the size of the 3-channel image is 32 x 32; 6 filter kernels with a size of 5 x 5 x 3 are used, no full 0 padding is needed, the stride is 1, and the output feature map has a size of 28 x 28 x 6.

[0026] In the second layer of the pooling layer, the input feature map has a size of 28 x 28 x 6, the filter size is 2 x 2, and the stride is 2, so that the output feature map has a size of 14 x 14 x 6.

[0027] In the third layer of the convolutional layer, the input feature map is the output feature map of the second layer of the pooling layer, so the size is 14 x 14 x 6; here the filter size is 5 x 5, the depth is 16, no full 0 padding is needed, the stride is 1, and the output feature map has a size of 10 x 10 x 16.

[0028] In the fourth layer of the pooling layer, the input feature map is the output feature map of the third layer of the convolutional layer, and the size is 10 x 10 x 16; the filter size is 2 x 2, and the stride is 2, so that the output feature map has a size of 5 x 5 x 16.

[0029] The convolution kernel parameter learning is realized through the following formula:

[0030]

[0031] β is the learning rate, and k is the number of learning iterations.

[0032] Preferably, in step 3), in order to improve the image recognition ability of image semantic segmentation, the preferred scheme uses a data enhancement technique, that is, in the encoder stage, a self-attention mechanism transformer network is introduced, which is beneficial to the training of the subsequent machine learning model deep architecture, thereby avoiding overfitting and increasing the generalization ability. In this step, the vector output by the backbone network is further extracted by the transformer with self-attention mechanism, and in the attention mechanism (self-attention), Q is the image query feature generated by the input vector and the search matrix, K represents the key value generated by the network according to the input vector, and V represents the matching value generated by the network according to the input vector, and here, the dot product is used to calculate the similarity of the two matrices Q and K. The self-attention mechanism calculation process is that, for the input vector x, multiply by three coefficient matrices W q 、W k 、W v (q, k, v have the same meaning as Q, K, V), to obtain Then calculate the similarity A = K T Q of Q and K, and perform a softmax operation or a Relu operation on the A matrix to obtain A', here , d is the dimension of the vector. Using the obtained A', calculate the output vector of each input vector of the corresponding self-attention layer , that is, the self-attention value is obtained, which is the weighted sum of the value vectors (information), and the weight is the correlation or similarity degree between each sample and each piece of information.

[0033] The transformer network designed in the application is a residual network with two attention layers. The output vector is used as the input of the deconvolutional neural network. The deconvolution is performed by upsampling to obtain the original size of the semantic segmentation image. The deconvolution network involved here is essentially a transpose convolutional network without full connection layer, which can adapt to any size input. This deconvolution layer increases the image size, outputs a finer result, and through the skip-level structure combining the results of different depth layers, the robustness and accuracy are ensured.

[0034] Because in the convolutional network, the convolution operation and the pooling operation make the size of the feature map smaller, in order to obtain the dense pixel prediction of the original image size, the application will perform a bilinear interpolation upsampling operation on the obtained feature map. It is easy to realize by transposed convolution with a fixed convolution kernel, and the parameters of the convolution kernel in the application are obtained by learning and training.

[0035] Preferably, step 4): up-sampling the feature map of the last self-attention layer to obtain a segmentation of the original image size, combining the prediction of the last layer rich in global information and the prediction of the shallower layer rich in local information through the skip structure due to the fewer pixels seen, making local prediction while complying with global prediction, realizing accurate semantic segmentation.

[0036] Specific process: the relationship between the output image size and the input:

[0037] output = stride(input-1) + kernel_size-2padding

[0038] In the formula, output is the size of the output image, input is the size of the input image, kernel_size is the size of the convolution kernel (filter), stride is the step, and padding is the 0 padding operation.

[0039] Bilateral interpolation is to perform first-order linear interpolation on two horizontal directions, and then perform first-order linear interpolation on the vertical direction. In this way, if the values of four adjacent points Q 11 (x1,y1), Q 12 (x1,y2), Q 21 (x2,y1), and Q 22 (x2,y2) are known, the value of the intermediate interpolation P(x,y) is:

[0040] f(x,y) = w 11 f(Q 11 )+w 22 f(Q 12 )+w 33 f(Q 21 )+w 22 f(Q 22 )

[0041] Here w 11 , w 22 , w 33 , and w 22 are

[0042]

[0043] represent weight coefficients.

[0044] With the above formula, the prediction of the last attention layer is up-sampled by bilinear interpolation with a step of stride = 8 and padding = 1 to obtain an image of the original size (32x32), and is fused (added) with the prediction from the pooling 2 layer (up-sampling of the pooling 2 layer with stride = 4 and padding = 6 to obtain an image of the original size (32x32)) to obtain a semantic segmentation image.

[0045] The application further discloses a satellite image semantic segmentation system based on a full convolutional neural network and a transformer, which comprises an auto-encoder, an encoder and a decoder.

[0046] The auto-encoder is used for defogging and denoising a satellite image.

[0047] The encoder is used for extracting satellite image features by using a multi-layer, i.e., deep, down-sampling convolutional neural network, and optimizing the satellite image features by using a transformer with a self-attention mechanism.

[0048] The decoder is used for realizing semantic segmentation of the satellite image by up-sampling.

[0049] Compared with the prior art, the application has the following beneficial effects:

[0050] The satellite image semantic segmentation method based on the full convolutional neural network and the transformer has the following innovative effects: the auto-encoder is used for defogging and denoising an image with clouds and fog, so that the satellite image becomes clearer; the convolutional network is used for extracting satellite image features, and a transformer self-attention network is designed to further extract the satellite image by using weight distribution; the feature map of the last attention layer is up-sampled to obtain a segmentation image of the original image size; and the prediction of the last layer with rich global information and the prediction of the shallower layer with rich local information are combined through a skip-level structure to realize accurate semantic segmentation while complying with global prediction and local prediction. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The application relates to a framework diagram of a satellite image semantic segmentation method based on a full convolutional neural network and a transformer.

[0052] Figure 2 The application relates to a framework diagram of an auto-encoder.

[0053] Figure 3 The application relates to a self-attention transformer network framework diagram.

[0054] Figure 4 The application relates to a loss function convergence diagram for training.

[0055] Figure 5 The application relates to a system block diagram of a satellite image semantic segmentation system based on a full convolutional neural network and a transformer. DETAILED DESCRIPTION

[0056] To more clearly illustrate the embodiments of the present invention, specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings and other embodiments can be obtained based on these drawings without inventive efforts.

[0057] The embodiment of the present invention provides a satellite image semantic segmentation method based on a fully convolutional neural network and a transformer, which realizes end-to-end training and pixel-by-pixel image semantic segmentation.

[0058] The automatic encoder can be used to remove fog and denoise satellite images. Through the denoising automatic encoder, the original image pixel signal with cloud cover is regarded as noise. This noisy signal (pixel value) is input into the denoising automatic encoder, and the error between the output and the original signal is calculated. The stochastic gradient descent algorithm is then used to adjust the weights to minimize the error, thus achieving defogging and denoising of satellite images.

[0059] The encoder uses a deep convolutional neural network and a converter with a self-attention mechanism to extract satellite image features, and the decoder uses upsampling convolution to achieve semantic segmentation of satellite images.

[0060] The process and principle of the satellite image semantic segmentation method of this embodiment are described in detail below.

[0061] 1) Dehaze and denoise satellite images.

[0062] Cloud-covered pixels are treated as noise by a denoising autoencoder. This noisy signal is then fed into the denoising autoencoder. The autoencoder is a type of artificial neural network (ANN) used in semi-supervised and unsupervised learning. It uses the input information as a learning target and performs representation learning on it, identifying patterns between neighboring pixels.

[0063] Basic mathematical process:

[0064] The pixel matrix on the satellite image is converted into a vector as the input of the automatic encoder recognition network, whose input is

[0065] x=[x(1),x(2)……x(d)], d is the dimension, the encoder projects x from the input layer to the first hidden layer h1 through the mapping function f,

[0066] h1=[h1(1),h1(2)……h1(dh1 )],

[0067] f is:

[0068] h1=f(x)=ReLu(W1x+b1),

[0069] where W1 is a h1(d h1 )×d weight matrix, and b1 is a bias vector.

[0070] h2=[h2(1),h2(2)……h2(d h2 )],

[0071] f is: h2=f(x)=ReLu(W2 h2+b2),

[0072] where W2 is a h2(d h2 )×h1 weight matrix, and b1 is a bias vector. The activation function ReLu of the decoder is a rectified linear unit function.

[0073] The hidden layer representation h2 is mapped to the output layer by the mapping function where the function is

[0074] The weight values W1, W2, are adjusted using the stochastic gradient descent algorithm, that is, the gradient information of the error loss function is back propagated using the error back propagation algorithm, and all model parameters are updated, and the loss function is assumed to be

[0075]

[0076] The updated network weight parameters are

[0077]

[0078] ρ is the learning rate, and i represents the i-th neuron. The output is reconstructed by limiting the number of hidden layer units as much as possible to be similar to the input x, so that the image is defogged and denoised.

[0079] 2) Deep convolutional network extracts image features

[0080] The output vector of the autoencoder is changed into a matrix as the input of the deep convolutional network, and then the backbone network (two convolutional layers and two pooling layers) of the deep convolutional neural network is used, and the average pooling rule is used in the pooling layer. For an m×m image, the convolution calculation formula C=A*B used in the present application is specific to

[0081]

[0082] The network parameters are as follows:

[0083] In the first layer of the convolution layer, the input is the pixel matrix of the defogged picture (length, width, channel number (red, green and blue)), and the size is 32*32*3; 6 filter sizes of 5*5*3 are used, and full 0 padding is not used, so the stride is 1, and the output feature map has a size of 28*28*6.

[0084] In the second layer of the pooling layer, the input feature map has a size of 28*28*6, the filter size is 2*2, and the stride is 2, so the output feature map has a size of 14*14*6.

[0085] In the third layer of the convolution layer, the input feature map is the output feature map of the second layer of the pooling layer, so the size is 14*14*6; here the filter size is 5*5, the depth is 16, full 0 padding is not used, the stride is 1, and the output feature map has a size of 10*10*16.

[0086] In the fourth layer of the pooling layer, the input feature map is the output feature map of the third layer of the convolution layer, and the size is 10*10*16; the filter size is 2*2, and the stride is 2, so the output feature map has a size of 5*5*16.

[0087] The convolution kernel parameter learning is realized through the following formula

[0088]

[0089] Beta is the learning rate, and k is the learning iteration number.

[0090] 3) Self-attention mechanism optimizes image features

[0091] In order to improve the image recognition ability of image semantic segmentation, the application uses a data enhancement technology, that is, in the encoder stage, a self-attention mechanism converter network is introduced, which is beneficial to the training of the subsequent machine learning model deep architecture, thereby avoiding overfitting and increasing the generalization ability. Here, the vector output by the backbone network will be further extracted by the converter with the self-attention mechanism, Q in the attention mechanism (self-attention) is the image query feature generated by the input vector and the search matrix, K represents the key value generated by the network according to the input vector, and V represents the matching value generated by the network according to the input vector, here, the dot product is used to calculate the similarity of the two matrices Q and K. The self-attention mechanism calculation process is that, for the input vector x, multiply by three coefficient matrices W q 、W k 、W v , to obtain Then the similarity A=K of Q and K is calculated T Q, the A matrix is subjected to a softmax operation or a Relu operation to obtain A', and here , d is the dimension of the vector. Using the obtained A', the similarity of each input vector to the output vector x (=V·A') of the corresponding self-attention layer is calculated, that is, the self-attention value is obtained, which is a weighted sum of value vectors (information), and the weight is the distribution of the correlation or similarity between each sample and each piece of information.

[0092] The converter network designed in the application is a residual network with two attention layers. The output vector is used as the input of the deconvolutional neural network. Here, the deconvolution is performed by upsampling to obtain a semantic segmentation image of the original size. The deconvolutional network involved here is essentially a transpose convolutional network without a full connection layer, which can adapt to any size input. This deconvolutional layer increases the image size, outputs a more refined result, and ensures robustness and accuracy through the skip-level structure of combining the results of different depth layers.

[0093] 4) Deconvolutional network for semantic segmentation

[0094] The feature map of the last self-attention layer is upsampled to obtain a segmentation of the original image size. The prediction of the last layer rich in global information is combined with the prediction of the shallower layer rich in local information through the skip-level structure to perform local prediction while complying with global prediction, thereby achieving accurate semantic segmentation.

[0095] Specific process: the relationship between the output image size and the input after upsampling:

[0096] output = stride (input-1) + kernel_size-2padding

[0097] In the formula, output is the size of the output image, input is the size of the input image, kernel_size is the size of the convolution kernel (filter), stride is the step, and padding is the 0 padding operation.

[0098] The bidirectional interpolation is first linearly interpolated in the two horizontal directions, and then linearly interpolated in the vertical direction. Thus, if the values of Q 11 (x1,y1), Q 12 (x1,y2), Q 21 (x2,y1), and Q 22 (x2,y2) are known, the value of the middle P(x,y) is:

[0099] f(x,y) = w 11 f(Q 11)+w 12 f(Q 12 )+w 21 f(Q 21 )+w 22 f(Q 22 )

[0100] Here

[0101] is a weight coefficient.

[0102] By using the above formula, the prediction of the last attention layer is up-sampled by bilinear interpolation with a step size of stride=8 and padding=1 to obtain an image of the original size (32*32), and is fused (added) with the prediction from the pooling 2 layer (the up-sampling of the pooling 2 layer has a step size of 4 and padding=6 to obtain an image of the original size (32*32)), to obtain a semantic segmentation image.

[0103] The process of realizing semantic segmentation is briefly summarized as follows:

[0104] 1. The automatic encoder performs defogging and denoising processing on the satellite image.

[0105] Specifically, the places where the original image pixel signal is covered by clouds are regarded as noise, the signal (pixel value) containing noise is input into the denoising automatic encoder, the error between the output and the original signal is calculated, and then the random gradient descent algorithm is used to adjust the weight value, so that the error reaches the minimum, and the satellite image is defogged.

[0106] 2. The deep convolutional neural network extracts satellite image features.

[0107] 3. The satellite image features are optimized by using a transformer with a self-attention mechanism.

[0108] 4. The semantic segmentation of the satellite image is realized by using a bilinear up-sampling deconvolutional network.

[0109] As Figure 5 shown, the application also discloses a satellite image semantic segmentation system embodiment based on a full convolutional neural network and a transformer, which comprises an automatic encoder, an encoder and a decoder.

[0110] The automatic encoder performs defogging and denoising processing on the satellite image.

[0111] The encoder adopts a multi-layer, i.e. deep down-sampling convolutional neural network to extract satellite image features, and uses a transformer with a self-attention mechanism to optimize the satellite image features.

[0112] The decoder realizes the semantic segmentation of the satellite image by using up-sampling.

[0113] Other contents of this embodiment can refer to the above satellite image semantic segmentation method embodiment.

[0114] The preferred embodiments and principles of the present application are described in detail above, and for those skilled in the art, according to the ideas provided by the present application, there will be changes in the specific implementation manner, and these changes should also be considered as the protection scope of the present application.

Claims

1. A satellite image semantic segmentation method based on a full convolutional neural network and a transformer, characterized in that, The steps are as follows: Step 1), using an automatic encoder to process the satellite image for defogging and denoising; Step 2), using a multi-layer, i.e., depth down-sampling convolutional neural network to extract satellite image features; Step 3), using a transformer with self-attention mechanism to optimize satellite image features; Step 4), using up-sampling deconvolution to realize semantic segmentation of the satellite image; Step 1) is specified as follows: the pixel matrix on the satellite image is converted into a vector as the input of the auto-encoder identification network, the input is x = [x(1), x(2) … x(d)], d is the dimension, the encoder projects x from the input layer to the first hidden layer h1 through the mapping function f, h1 = [h1(1), h1(2) … h1(d h1 )], f is: x = [x(1), x(2) … x(d)] → f(x) = h1 = [h1(1), h1(2) … h1(d)] h1 = f(x) = Relu(W1x + b1); wherein, W1 is a h1(d h1 ) x d weight matrix, b1 is a bias vector; h1 is projected from the first hidden layer to the second hidden layer h2 through a mapping function f, h2 = [h2(1), h2(2) … h2(d h2 )], f is: h2 = f(x) = Relu(W2h1 + b2); wherein, W2 is a h2(d h2 ) x h1 weight matrix; the activation function ReLu of the decoder is a rectified linear unit function; h2 represented by the hidden layer is mapped to the output layer h3 through a mapping function f, h3 = f(x) = Relu(W3h2 + b3); wherein, W3 is a h3(d ) x h2 weight matrix; the activation function ReLu of the decoder is a rectified linear unit function; the output of the decoder is h3 = [h3(1), h3(2) … h3(d )], wherein, h3 is the output of the decoder, and the function is The weight values W1, W2, are adjusted by using a stochastic gradient descent algorithm, that is, the gradient information of the error loss function is back propagated by using an error back propagation algorithm, and all model parameters are updated, and the loss function is set as: The updated network weight parameters are: p is the learning rate, i represents the i-th neuron, k is the iteration number, is the derivative symbol.

2. The satellite image semantic segmentation method based on a full convolutional neural network and a transformer according to claim 1, characterized in that, Step 2) is as follows: The output vector of the automatic encoder is changed into a matrix as the input of the deep convolutional network; the backbone network of the deep convolutional neural network is used, and for m x m images, the convolution calculation formula is as follows: Wherein, the subscripts i and j are pixel positions; The convolution kernel parameter learning is realized through the following formula: β is the learning rate, and k is the learning iteration number.

3. The satellite image semantic segmentation method based on a full convolutional neural network and a transformer according to claim 2, characterized in that, Step 3) is as follows: The vector output by the backbone network is subjected to feature extraction by using a transformer with a self-attention mechanism, wherein Q in the self-attention mechanism is an image query feature generated by inputting a vector into a search matrix, K represents a key value generated by the network according to the input vector, and V represents a matching value generated by the network according to the input vector, and the similarity between the two matrices Q and K is calculated by using a dot product; the self-attention mechanism calculation process is as follows: for an input vector x, the vector is multiplied by three corresponding coefficient matrices W q , W k , and W v , respectively The similarity A between Q and K is calculated T Q, and the A matrix is subjected to a softmax operation or a Relu operation to obtain A′, d is the dimension of the vector; the obtained A′ is used to calculate the output vector of each input vector corresponding to the self-attention layer , i.e., to obtain a self-attention value which is a weighted sum of value vectors, and the weight is the distribution of the correlation or similarity between each sample and each piece of information.

4. The satellite image semantic segmentation method based on a full convolutional neural network and a transformer according to claim 3, characterized in that, Step 4) is as follows: The relationship between the output image size and the input is as follows: output = stride(input-1) + kernel_size-2padding In the formula, output is the size of the output image, input is the size of the input image, kernel_size is the size of the convolution kernel, stride is the step, and padding is the 0 operation; The bidirectional interpolation is to perform first-order linear interpolation in two horizontal directions and then perform first-order linear interpolation in the vertical direction. Thus, if the pixel values Q 11 (x1,y1), Q 12 (x1,y2), Q 21 (x2,y1), Q 22 (x2,y2) of four adjacent points in the horizontal and vertical directions are known, the value of the intermediate interpolation P(x, y) is: f(x,y) = w 11 f(Q 11 ) + w 22 f(Q 12 ) + w 33 f(Q 21 ) + w 22 f(Q 22 ) Here w 11 , w 22 , w 33 , w 22 are: are representative of weight coefficients; Using the above formula, the prediction of the last attention layer is up-sampled by bilinear interpolation with a step of stride = 8 and padding = 1 to obtain an image of the original size, and is fused with the prediction from the pooling 2 layer to obtain a semantic segmentation image.

5. A satellite image semantic segmentation system based on fully convolutional neural network and transformer for performing the method of claim 1, characterized in that, It comprises an automatic encoder, an encoder and a decoder; The automatic encoder is used for defogging and denoising of the satellite image; The encoder is used to extract satellite image features by using a multi-layer, i.e., depth down-sampling convolutional neural network; and the satellite image features are optimized by using a transformer with a self-attention mechanism; The decoder is used to realize semantic segmentation of the satellite image by using up-sampling.

Images