Sand storm identification method for remote sensing image based on CTFusionNet

Abstract

The invention discloses a remote sensing image sandstorm identification method based on a CTFusionNet, and belongs to the field of remote sensing image processing, and the method comprises the following steps: inputting a to-be-identified remote sensing image into a trained CTFusionNet, and obtaining a sandstorm identification result of the to-be-identified remote sensing image; wherein the CTFusionNet comprises a CNN (Convolutional Neural Network) branch, a lower sampling layer, an embedded layer, a Transform branch, an upper sampling layer and a feature superposition fusion layer which are sequentially connected from input to output; the first classifier is connected with the output end of the CNN branch; and the second classifier is connected with the output end of the feature superposition fusion layer. The identification accuracy of the sand storm in the remote sensing image can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and more particularly to a sandstorm recognition method for remote sensing images based on CTFusionNet. Background Art

[0002] Currently, sandstorm identification is a crucial step in sand control and desertification prevention. Satellite remote sensing technology, with its rich spectral information and real-time monitoring capabilities, has become the main research method in the field of sandstorm identification. Traditional fixed threshold methods are unable to adapt to different background environments, resulting in low sandstorm identification accuracy.

[0003] Therefore, how to provide a sandstorm identification method that can improve the identification accuracy of sandstorms in remote sensing images is a problem that those skilled in the art urgently need to solve. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a sandstorm recognition method based on remote sensing images of CTFusionNet.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A sandstorm recognition method based on remote sensing images using CTFusionNet includes the following steps:

[0007] Inputting the remote sensing image to be identified into the trained CTFusionNet to obtain a sandstorm recognition result of the remote sensing image to be identified;

[0008] Among them, CTFusionNet includes a CNN branch, a downsampling layer, an embedding layer, a Transformer branch, an upsampling layer and a feature stacking and fusion layer connected in sequence from input to output; a first classifier connected to the output end of the CNN branch; and a second classifier connected to the output end of the feature stacking and fusion layer.

[0009] Preferably, the remote sensing image to be identified is input into the CNN branch to obtain the local feature map Local feature map Local feature map Local feature map and local feature maps

[0010] The local feature map Local feature map Local feature map Local feature map Local feature map Input them into the downsampling layer and embedding layer respectively to obtain the intermediate feature map F';

[0011] The local feature map Input into the first classifier to obtain the first classification result;

[0012] The intermediate feature map F' is input into the Transformer branch and upsampling layer in sequence to obtain the global feature map

[0013] The local feature map and global feature maps After being input into the feature superposition fusion layer, it is input into the second classifier to obtain the second classification result;

[0014] The classification result with a higher confidence level between the first classification result and the second classification result is used as the sandstorm recognition result of the remote sensing image to be recognized.

[0015] Preferably, the CNN branch includes a first RP module, a second RP module, a third RP module, a fourth RP module and a fifth RP module connected in sequence from input to output;

[0016] The Transformer branch includes a first multi-head attention layer, a second multi-head attention layer, a third multi-head attention layer, a fourth multi-head attention layer, a fifth multi-head attention layer, a sixth multi-head attention layer, a seventh multi-head attention layer, an eighth multi-head attention layer, a ninth multi-head attention layer, a tenth multi-head attention layer, an eleventh multi-head attention layer, a twelfth multi-head attention layer, a multi-layer perceptron layer, and an element-by-element addition unit, which are sequentially connected from input to output;

[0017] The remote sensing image to be identified is processed by the first RP module to obtain a local feature map

[0018] Local feature map After processing by the second RP module, the local feature map is obtained

[0019] Local feature map After processing by the third RP module, a local feature map is obtained.

[0020] Local feature map After processing by the fourth RP module, a local feature map is obtained.

[0021] Local feature map After processing by the fifth RP module, a local feature map is obtained.

[0022] Local feature map Performing classification using the first classifier to obtain the first classification result;

[0023] Local feature map Local feature map Local feature map Local feature map Local feature map After being processed by the downsampling layer and the embedding layer respectively, the intermediate feature map F' is obtained;

[0024] The intermediate feature map F' is processed by the first multi-head attention layer, the second multi-head attention layer, the third multi-head attention layer, the fourth multi-head attention layer, the fifth multi-head attention layer, the sixth multi-head attention layer, the seventh multi-head attention layer, the eighth multi-head attention layer, the ninth multi-head attention layer, the tenth multi-head attention layer, the eleventh multi-head attention layer, the twelfth multi-head attention layer and the multi-layer perceptron layer in sequence to obtain the global feature map

[0025] Intermediate feature map F' and global feature map After being input into the element-by-element addition unit of the Transformer branch, the global feature map is obtained.

[0026] Global feature map After processing the upsampling layer, the global feature map is obtained

[0027] Local feature map and global feature maps After being processed by the feature superposition fusion layer, it is input into the second classifier to obtain the second classification result.

[0028] Preferably, the first RP module, the second RP module, the third RP module, the fourth RP module and the fifth RP module have the same structure, and all include a first 1*1 convolution layer, a 3*3 convolution layer and a 5*5 convolution layer, a channel dimension splicing layer, a second 1*1 convolution layer and an element-by-element addition unit;

[0029] The input of the first 1*1 convolutional layer, the input of the 3*3 convolutional layer, and the input of the 5*5 convolutional layer are connected to form the input of the RP module.

[0030] The output of the first 1*1 convolutional layer, the output of the 3*3 convolutional layer, and the output of the 5*5 convolutional layer are all connected to the input of the channel dimension splicing layer;

[0031] The output of the channel dimension concatenation layer is connected to the input of the second 1*1 convolutional layer;

[0032] The output of the second 1*1 convolutional layer is connected to the input of the element-by-element addition unit of the RP module;

[0033] The input of the element-by-element addition unit of the RP module is connected to the input of the first 1*1 convolutional layer;

[0034] The output terminal of the element-by-element addition unit of the RP module constitutes the output terminal of the RP module.

[0035] Preferably, the first 1*1 convolutional layer is connected to the input end of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence;

[0036] The 3*3 convolution layer is connected to the input end of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence;

[0037] The 5*5 convolution layer is connected to the input end of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence;

[0038] The second 1*1 convolutional layer is connected to the input end of the element-by-element addition unit of the RP module through a normalization layer and a ReLU activation function layer in sequence.

[0039] Preferably, the convolution step sizes of the first 1*1 convolution layer, the 3*3 convolution layer, the 5*5 convolution layer, and the second 1*1 convolution layer are all 1.

[0040] Preferably, a trained CTFusionNet is obtained based on the following steps:

[0041] Obtain a training set; wherein the training set consists of remote sensing images, each pixel of each remote sensing image is labeled, and each pixel is a training sample, and the label is sandstorm or non-sandstorm;

[0042] The CTFusionNet is trained using the training set to obtain a trained CTFusionNet. During the training process, the network parameters of the CTFusionNet are updated by minimizing the total loss function.

[0043] Preferably, the expression of the total loss function is:

[0044] L total =L CE +L SCBCE :

[0045] Where, L total Represents the total loss function; L CE represents the cross entropy loss of the first classifier; L SCBCE represents the selective confidence cross entropy loss for the second classifier.

[0046] Preferably, L CEThe expression is:

[0047]

[0048] Where H represents the height of the remote sensing image; W represents the width of the remote sensing image; Y ij represents the true label of the pixel in the i-th row and j-th column of the remote sensing image; P ij It represents the confidence of the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier, that is, the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier is a sandstorm.

[0049] Preferably, L SCBCE The expression is:

[0050]

[0051]

[0052] Where m ij represents the mask of the pixel in row i and column j of the remote sensing image; σ represents the preset confidence threshold; P ij ' represents the confidence of the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier, that is, the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier is a sandstorm.

[0053] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a sandstorm recognition method for remote sensing images based on CTFusionNet, which can improve the recognition accuracy of sandstorms in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0055] Figure 1 A schematic diagram of CTFusionNet provided in an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of an RP module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] The embodiment of the present invention discloses a sandstorm recognition method based on remote sensing images using CTFusionNet, comprising the following steps:

[0059] Inputting the remote sensing image to be identified into the trained CTFusionNet to obtain a sandstorm recognition result of the remote sensing image to be identified;

[0060] Among them, CTFusionNet includes a CNN branch, a downsampling layer, an embedding layer, a Transformer branch, an upsampling layer and a feature stacking and fusion layer connected in sequence from input to output; a first classifier connected to the output end of the CNN branch; and a second classifier connected to the output end of the feature stacking and fusion layer.

[0061] In one embodiment, the remote sensing image to be identified is input into the CNN branch to obtain a local feature map Local feature map Local feature map Local feature map and local feature maps

[0062] The local feature map Local feature map Local feature map Local feature map Local feature map Input them into the downsampling layer and embedding layer respectively to obtain the intermediate feature map F';

[0063] The local feature map Input into the first classifier to obtain the first classification result;

[0064] The intermediate feature map F' is input into the Transformer branch and upsampling layer in sequence to obtain the global feature map

[0065] The local feature map and global feature maps After being input into the feature superposition fusion layer, it is input into the second classifier to obtain the second classification result;

[0066] The classification result with a higher confidence level between the first classification result and the second classification result is used as the sandstorm recognition result of the remote sensing image to be recognized.

[0067] In one embodiment, if Figure 1 As shown, the CNN branch includes a first RP module, a second RP module, a third RP module, a fourth RP module and a fifth RP module connected in sequence from input to output;

[0068] The Transformer branch includes a first multi-head attention layer, a second multi-head attention layer, a third multi-head attention layer, a fourth multi-head attention layer, a fifth multi-head attention layer, a sixth multi-head attention layer, a seventh multi-head attention layer, an eighth multi-head attention layer, a ninth multi-head attention layer, a tenth multi-head attention layer, an eleventh multi-head attention layer, a twelfth multi-head attention layer, a multi-layer perceptron layer, and an element-by-element addition unit, which are sequentially connected from input to output;

[0069] The remote sensing image to be identified is processed by the first RP module to obtain a local feature map

[0070] Local feature map After processing by the second RP module, the local feature map is obtained

[0071] Local feature map After processing by the third RP module, a local feature map is obtained.

[0072] Local feature map After processing by the fourth RP module, a local feature map is obtained.

[0073] Local feature map After processing by the fifth RP module, a local feature map is obtained.

[0074] Local feature map Performing classification using the first classifier to obtain the first classification result;

[0075] Local feature map Local feature map Local feature map Local feature map Local feature map After being processed by the downsampling layer and the embedding layer respectively, the intermediate feature map F' is obtained;

[0076] It can be understood that the downsampling layer is used to downsample the high-resolution feature map to a lower spatial resolution;

[0077] The downsampling layer uses a convolutional layer or a pooling layer;

[0078] The intermediate feature map F' output by the embedding layer can be regarded as a series of "tokens". Assume that the feature map F obtained after downsampling down It will be converted into a low-dimensional representation suitable for Transformer processing through the embedding layer. Assume that the weight matrix of the embedding layer is W embed , then the intermediate feature map F' can be expressed as [N,D], where N represents the number of "tokens" in the feature map and D represents the feature dimension after embedding. The intermediate feature map F' can be expressed as:

[0079] F'=W embed *F down ;

[0080] Where W embed is the weight matrix of the embedding layer, responsible for mapping each token from the original feature space to the embedding space, generating a feature representation suitable for the Transformer. The intermediate feature map F' can be viewed as a set of tokens, each representing the features of a local region in the image. These tokens serve as input to the Transformer branch to obtain global features.

[0081] The intermediate feature map F' is processed by the first multi-head attention layer, the second multi-head attention layer, the third multi-head attention layer, the fourth multi-head attention layer, the fifth multi-head attention layer, the sixth multi-head attention layer, the seventh multi-head attention layer, the eighth multi-head attention layer, the ninth multi-head attention layer, the tenth multi-head attention layer, the eleventh multi-head attention layer, the twelfth multi-head attention layer and the multi-layer perceptron layer in sequence to obtain the global feature map

[0082] Intermediate feature map F' and global feature map After being input into the element-by-element addition unit of the Transformer branch, the global feature map is obtained.

[0083] Global feature map After processing the upsampling layer, the global feature map is obtained

[0084] It can be understood that the upsampling layer is used to convert the low-resolution global feature map Restore to high-resolution global feature map

[0085] Local feature map and global feature maps After being processed by the feature superposition fusion layer, it is input into the second classifier to obtain the second classification result.

[0086] It should be noted that the size of the remote sensing image to be identified is H*W*C, where H, W, and C represent the height, width, and number of channels of the remote sensing image to be identified, respectively.

[0087] Local feature map The dimensions are H*W*C;

[0088] Local feature map The dimensions are H*W*C;

[0089] Local feature map The dimensions are H*W*C;

[0090] Local feature map The dimensions are H*W*C;

[0091] Local feature map The dimensions are H*W*C;

[0092] The size of the intermediate feature map F' is H'*W'*5C; where H' and W' are 5 local feature maps (i.e. local feature maps Local feature map Local feature map Local feature map Local feature map ) width and height after downsampling;

[0093] Global feature map The size is H'*W'*5C;

[0094] Global feature map The size is H'*W'*5C;

[0095] Global feature map The dimensions are H*W*C.

[0096] In one embodiment, if Figure 2 As shown in the figure, the structures of the first RP module, the second RP module, the third RP module, the fourth RP module and the fifth RP module are the same, and all include a first 1*1 convolution layer, a 3*3 convolution layer and a 5*5 convolution layer, a channel dimension splicing layer, a second 1*1 convolution layer and an element-by-element addition unit;

[0097] The input of the first 1*1 convolutional layer, the input of the 3*3 convolutional layer, and the input of the 5*5 convolutional layer are connected to form the input of the RP module.

[0098] The output of the first 1*1 convolutional layer, the output of the 3*3 convolutional layer, and the output of the 5*5 convolutional layer are all connected to the input of the channel dimension splicing layer;

[0099] The output of the channel dimension concatenation layer is connected to the input of the second 1*1 convolutional layer;

[0100] The output of the second 1*1 convolutional layer is connected to the input of the element-by-element addition unit of the RP module;

[0101] The input of the element-by-element addition unit of the RP module is connected to the input of the first 1*1 convolutional layer;

[0102] The output terminal of the element-by-element addition unit of the RP module constitutes the output terminal of the RP module.

[0103] In one embodiment, the convolution steps of the first 1*1 convolution layer, the 3*3 convolution layer, the 5*5 convolution layer, and the second 1*1 convolution layer are all 1.

[0104] In this embodiment:

[0105] 1) The specific processing process of the first RP module is as follows:

[0106] Input the remote sensing image to be identified (size is H*W*C) into the first 1*1 convolution layer in the first RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0107] Input the remote sensing image to be identified (size is H*W*C) into the 3*3 convolution layer in the first RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0108] Input the remote sensing image to be identified (size is H*W*C) into the 5*5 convolution layer in the first RP module to obtain the local feature map (Dimensions are H*W*C);

[0109] The local feature map Local feature map and local feature maps Input the channel dimension splicing layer in the first RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0110] The spliced ​​feature map Input to the second 1*1 convolution layer in the first RP module to obtain the local feature map (Dimensions are H*W*C);

[0111] The remote sensing image to be identified is combined with the local feature map Input into the element-by-element addition unit in the first RP module to obtain the local feature map (Dimensions are H*W*C);

[0112] 2) The specific processing process of the second RP module is as follows:

[0113] The local feature map (size is H*W*C) is input to the first 1*1 convolution layer in the second RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0114] The local feature map (size is H*W*C) is input to the 3*3 convolution layer in the second RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0115] The local feature map (size is H*W*C) is input to the 5*5 convolution layer in the second RP module to obtain the local feature map (Dimensions are H*W*C);

[0116] The local feature map Local feature map and local feature maps Input to the channel dimension splicing layer in the second RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0117] The spliced ​​feature map Input to the second 1*1 convolution layer in the second RP module to obtain the local feature map (Dimensions are H*W*C);

[0118] The local feature map With local feature map Input to the element-by-element addition unit in the second RP module to obtain the local feature map (Dimensions are H*W*C);

[0119] 3) The specific processing process of the third RP module is as follows:

[0120] The local feature map (size is H*W*C) is input to the first 1*1 convolution layer in the third RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0121] The local feature map (size is H*W*C) is input to the 3*3 convolution layer in the third RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0122] The local feature map (size is H*W*C) is input to the 5*5 convolution layer in the third RP module to obtain the local feature map (Dimensions are H*W*C);

[0123] The local feature map Local feature map and local feature maps Input the channel dimension splicing layer in the third RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0124] The spliced ​​feature map Input to the second 1*1 convolution layer in the third RP module to obtain the local feature map (Dimensions are H*W*C);

[0125] The local feature map With local feature map Input into the element-by-element addition unit in the third RP module to obtain the local feature map (Dimensions are H*W*C);

[0126] 4) The specific processing process of the fourth RP module is as follows:

[0127] The local feature map (size is H*W*C) is input to the first 1*1 convolution layer in the fourth RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0128] The local feature map (size is H*W*C) is input to the 3*3 convolution layer in the fourth RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0129] The local feature map (size is H*W*C) is input to the 5*5 convolution layer in the fourth RP module to obtain the local feature map (Dimensions are H*W*C);

[0130] The local feature map Local feature map and local feature maps Input to the channel dimension splicing layer in the fourth RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0131] The spliced ​​feature map Input to the second 1*1 convolution layer in the fourth RP module to obtain the local feature map (Dimensions are H*W*C);

[0132] The local feature map With local feature map Input into the element-by-element addition unit in the fourth RP module to obtain the local feature map (Dimensions are H*W*C);

[0133] 5) The specific processing process of the fifth RP module is as follows:

[0134] The local feature map (size is H*W*C) is input to the first 1*1 convolution layer in the fifth RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0135] The local feature map (size is H*W*C) is input to the 3*3 convolution layer in the fifth RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0136] The local feature map (size is H*W*C) is input to the 5*5 convolution layer in the fifth RP module to obtain the local feature map (Dimensions are H*W*C);

[0137] The local feature map Local feature map and local feature maps Input to the channel dimension splicing layer in the fifth RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0138] The spliced ​​feature map Input to the second 1*1 convolution layer in the fifth RP module to obtain the local feature map (Dimensions are H*W*C);

[0139] The local feature map With local feature map Input to the element-by-element addition unit in the fifth RP module to obtain the local feature map (Dimensions are H*W*C);

[0140] It is understandable that:

[0141] The first 1×1 convolutional layer of the present invention is responsible for extracting local fine features, while the 3×3 convolutional layer and the 5×5 convolutional layer capture the contextual features of the medium range and larger areas respectively;

[0142] The present invention avoids downsampling by setting the stride of all convolutional layers to 1, thereby ensuring that the spatial resolution of the feature map is preserved at each layer, further providing rich detail information for subsequent sandstorm identification tasks.

[0143] In summary, to comprehensively capture features at different scales, the CNN branch employs a multi-scale convolution strategy. Each RP module contains three convolution kernels of different sizes: 1×1, 3×3, and 5×5. Through the combination of these convolution kernels, the CNN branch is able to simultaneously capture local, medium-range, and broader contextual information while maintaining spatial resolution.

[0144] The element-by-element addition unit design in the RP module of the present invention not only effectively mitigates vanishing gradients but also enhances the representational capabilities of feature maps while maintaining spatial resolution. This means that the RP module of the present invention can better capture deep features, improving the model's performance when processing complex images, particularly in extracting details from remote sensing images.

[0145] The present invention performs dimensionality reduction through a second 1×1 convolutional layer with a convolution step size of 1, reducing computational effort while maintaining the resolution of the feature map. This approach effectively integrates features of different scales while avoiding detail loss due to feature map size changes.

[0146] The refined feature processing of the RP module of the present invention can effectively extract rich local and global feature information without sacrificing spatial resolution, thereby providing a more accurate and robust feature representation for subsequent sandstorm identification tasks, thereby improving the segmentation performance of the model in complex scenarios.

[0147] In one embodiment, the first 1*1 convolutional layer is connected to the input end of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence;

[0148] The 3*3 convolution layer is connected to the input end of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence;

[0149] The 5*5 convolution layer is connected to the input end of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence;

[0150] The second 1*1 convolutional layer is connected to the input end of the element-by-element addition unit of the RP module through a normalization layer and a ReLU activation function layer in sequence.

[0151] In this embodiment,

[0152] 1) The specific processing process of the first RP module is as follows:

[0153] Input the remote sensing image to be identified (size is H*W*C) into the first 1*1 convolution layer in the first RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0154] Input the remote sensing image to be identified (size is H*W*C) into the 3*3 convolution layer in the first RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0155] Input the remote sensing image to be identified (size is H*W*C) into the 5*5 convolution layer in the first RP module to obtain the local feature map (Dimensions are H*W*C);

[0156] The local feature map Local feature map and local feature maps Input the channel dimension splicing layer in the first RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0157] The spliced ​​feature map Input to the second 1*1 convolution layer in the first RP module to obtain the local feature map (Dimensions are H*W*C);

[0158] The remote sensing image to be identified is combined with the local feature map Input into the element-by-element addition unit in the first RP module to obtain the local feature map (Dimensions are H*W*C);

[0159] 2) The specific processing process of the second RP module is as follows:

[0160] The local feature map (size is H*W*C) is input to the first 1*1 convolution layer in the second RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0161] The local feature map (size is H*W*C) is input to the 3*3 convolution layer in the second RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0162] The local feature map (size is H*W*C) is input to the 5*5 convolution layer in the second RP module to obtain the local feature map (Dimensions are H*W*C);

[0163] The local feature map Local feature map and local feature maps Input to the channel dimension splicing layer in the second RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0164] The spliced ​​feature map Input to the second 1*1 convolution layer in the second RP module to obtain the local feature map (Dimensions are H*W*C);

[0165] The local feature map With local feature map Input to the element-by-element addition unit in the second RP module to obtain the local feature map (Dimensions are H*W*C);

[0166] 3) The specific processing process of the third RP module is as follows:

[0167] The local feature map (size is H*W*C) is input to the first 1*1 convolution layer in the third RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0168] The local feature map (size is H*W*C) is input to the 3*3 convolution layer in the third RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0169] The local feature map (size is H*W*C) is input to the 5*5 convolution layer in the third RP module to obtain the local feature map (Dimensions are H*W*C);

[0170] The local feature map Local feature map and local feature maps Input the channel dimension splicing layer in the third RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0171] The spliced ​​feature map Input to the second 1*1 convolution layer in the third RP module to obtain the local feature map (Dimensions are H*W*C);

[0172] The local feature map With local feature map Input into the element-by-element addition unit in the third RP module to obtain the local feature map (Dimensions are H*W*C);

[0173] 4) The specific processing process of the fourth RP module is as follows:

[0174] The local feature map (size is H*W*C) is input to the first 1*1 convolution layer in the fourth RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0175] The local feature map (size is H*W*C) is input to the 3*3 convolution layer in the fourth RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0176] The local feature map (size is H*W*C) is input to the 5*5 convolution layer in the fourth RP module to obtain the local feature map (Dimensions are H*W*C);

[0177] The local feature map Local feature map and local feature maps Input to the channel dimension splicing layer in the fourth RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0178] The spliced ​​feature map Input to the second 1*1 convolution layer in the fourth RP module to obtain the local feature map (Dimensions are H*W*C);

[0179] The local feature map With local feature map Input into the element-by-element addition unit in the fourth RP module to obtain the local feature map (Dimensions are H*W*C);

[0180] 5) The specific processing process of the fifth RP module is as follows:

[0181] The local feature map (size is H*W*C) is input to the first 1*1 convolution layer in the fifth RP module to obtain the local feature map (Dimensions are H*W*C / 4);

[0182] The local feature map (size is H*W*C) is input to the 3*3 convolution layer in the fifth RP module to obtain the local feature map (Dimensions are H*W*C / 2);

[0183] The local feature map (size is H*W*C) is input to the 5*5 convolution layer in the fifth RP module to obtain the local feature map (Dimensions are H*W*C);

[0184] The local feature map Local feature map and local feature maps Input to the channel dimension splicing layer in the fifth RP module to obtain the splicing feature map (Dimensions are H*W*(C+C / 2+C / 4));

[0185] The spliced ​​feature map Input to the second 1*1 convolution layer in the fifth RP module to obtain the local feature map (Dimensions are H*W*C);

[0186] The local feature map With local feature map Input to the element-by-element addition unit in the fifth RP module to obtain the local feature map (Dimensions are H*W*C);

[0187] In one embodiment, a trained CTFusionNet is obtained based on the following steps:

[0188] Obtain a training set; wherein the training set consists of remote sensing images, each pixel of each remote sensing image is labeled, and each pixel is a training sample, and the label is sandstorm or non-sandstorm;

[0189] The CTFusionNet is trained using the training set to obtain a trained CTFusionNet. During the training process, the network parameters of the CTFusionNet are updated by minimizing the total loss function.

[0190] In one embodiment, the total loss function is expressed as:

[0191] L total =L CE +L SCBCE ;

[0192]

[0193] Where, Ltotal Represents the total loss function; L CE represents the cross entropy loss of the first classifier; L SCBCE represents the selective confidence cross entropy loss of the second classifier; H represents the height of the remote sensing image; W represents the width of the remote sensing image; Y ij represents the true label of the pixel in the i-th row and j-th column of the remote sensing image; P ij represents the confidence level of the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier, that is, the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier is a sandstorm; m ij represents the mask of the pixel in the i-th row and j-th column of the remote sensing image; σ represents a preset confidence threshold (in a certain embodiment, σ is 0.5); P ij ' represents the confidence of the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier, that is, the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier is a sandstorm.

[0194] Finally, the CTFusionNet of the present invention is compared with RF, UNet, DeepLabV3+, HRNet, Swin Transformer, and CTFusionNet, as shown in Table 1:

[0195] Table 1

[0196]

[0197] From Table 1 we can see that:

[0198] For OA values: CTFusionNet performed best at 0.88, followed by SwinTransformer at 0.87, HRNet and DeeplabV3+ both performed well at 0.80, while RF and UNet were relatively low at 0.77 and 0.63 respectively.

[0199] It can be seen from this that the CTFusionNet of the present invention has strong overall accuracy.

[0200] Regarding the Kappa value: The Kappa value is used to measure the correlation between the prediction results and the true labels. The CTFusionNet value is 0.78, which is slightly better than the 0.75 of SwinTransformer, while the RF and UNet values ​​are only 0.48 and 0.52, respectively. Their Kappa values ​​are low, indicating that the correlation between the prediction results of the two models and the true labels is weak.

[0201] It can be seen from this that the CTFusionNet of the present invention has better classification consistency;

[0202] For the IoU value (intersection over union): CTFusionNet and SwinTransformer are 0.69 and 0.62 respectively, outperforming other models in terms of IoU, indicating that these two models are more accurate in predicting the target area. However, other models, UNet and HRNet, are slightly lower, both at 0.57, while RF is lower at only 0.52.

[0203] It can be seen from this that the CTFusionNet of the present invention is superior to other models in processing the overlap of target areas.

[0204] In summary, the present invention can significantly improve the recognition accuracy of sandstorms in remote sensing images.

[0205] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0206] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A sandstorm recognition method based on remote sensing images using CTFusionNet, characterized in that: The following steps are involved: Inputting the remote sensing image to be identified into the trained CTFusionNet to obtain a sandstorm recognition result of the remote sensing image to be identified; Among them, CTFusionNet includes a CNN branch, a downsampling layer, an embedding layer, a Transformer branch, an upsampling layer and a feature stacking and fusion layer connected in sequence from input to output; a first classifier connected to the output end of the CNN branch; and a second classifier connected to the output end of the feature stacking and fusion layer.

2. The sandstorm identification method based on remote sensing images using CTFusionNet according to claim 1, characterized in that: Input the remote sensing image to be identified into the CNN branch to obtain the local feature map Local feature map Local feature map Local feature map and local feature maps The local feature map Local feature map Local feature map Local feature map Local feature map Input them into the downsampling layer and embedding layer respectively to obtain the intermediate feature map F'; The local feature map Input into the first classifier to obtain the first classification result; The intermediate feature map F' is input into the Transformer branch and upsampling layer in sequence to obtain the global feature map The local feature map and global feature maps After being input into the feature superposition fusion layer, it is input into the second classifier to obtain the second classification result; The classification result with a higher confidence level between the first classification result and the second classification result is used as the sandstorm recognition result of the remote sensing image to be recognized.

3. The sandstorm identification method based on remote sensing images of CTFusionNet according to claim 2 is characterized in that: The CNN branch includes a first RP module, a second RP module, a third RP module, a fourth RP module and a fifth RP module connected in sequence from input to output; The Transformer branch includes a first multi-head attention layer, a second multi-head attention layer, a third multi-head attention layer, a fourth multi-head attention layer, a fifth multi-head attention layer, a sixth multi-head attention layer, a seventh multi-head attention layer, an eighth multi-head attention layer, a ninth multi-head attention layer, a tenth multi-head attention layer, an eleventh multi-head attention layer, a twelfth multi-head attention layer, a multi-layer perceptron layer, and an element-by-element addition unit, which are sequentially connected from input to output; The remote sensing image to be identified is processed by the first RP module to obtain a local feature map Local feature map After processing by the second RP module, the local feature map is obtained Local feature map After processing by the third RP module, a local feature map is obtained. Local feature map After processing by the fourth RP module, a local feature map is obtained. Local feature map After processing by the fifth RP module, a local feature map is obtained. Local feature map Performing classification using the first classifier to obtain the first classification result; Local feature map Local feature map Local feature map Local feature map Local feature map After being processed by the downsampling layer and the embedding layer respectively, the intermediate feature map F' is obtained; The intermediate feature map F' is processed by the first multi-head attention layer, the second multi-head attention layer, the third multi-head attention layer, the fourth multi-head attention layer, the fifth multi-head attention layer, the sixth multi-head attention layer, the seventh multi-head attention layer, the eighth multi-head attention layer, the ninth multi-head attention layer, the tenth multi-head attention layer, the eleventh multi-head attention layer, the twelfth multi-head attention layer and the multi-layer perceptron layer in sequence to obtain the global feature map Intermediate feature map F' and global feature map After being input into the element-by-element addition unit of the Transformer branch, the global feature map is obtained. Global feature map After processing the upsampling layer, the global feature map is obtained Local feature map and global feature maps After being processed by the feature superposition fusion layer, it is input into the second classifier to obtain the second classification result.

4. The sandstorm identification method based on remote sensing images of CTFusionNet according to claim 3 is characterized in that: The first RP module, the second RP module, the third RP module, the fourth RP module, and the fifth RP module have the same structure, including a first 1*1 convolutional layer, a 3*3 convolutional layer, a 5*5 convolutional layer, a channel dimension splicing layer, a second 1*1 convolutional layer, and an element-by-element addition unit; The input of the first 1*1 convolutional layer, the input of the 3*3 convolutional layer, and the input of the 5*5 convolutional layer are connected to form the input of the RP module. The output of the first 1*1 convolutional layer, the output of the 3*3 convolutional layer, and the output of the 5*5 convolutional layer are all connected to the input of the channel dimension splicing layer; The output of the channel dimension concatenation layer is connected to the input of the second 1*1 convolutional layer; The output of the second 1*1 convolutional layer is connected to the input of the element-by-element addition unit of the RP module; The input of the element-by-element addition unit of the RP module is connected to the input of the first 1*1 convolutional layer; The output terminal of the element-by-element addition unit of the RP module constitutes the output terminal of the RP module.

5. The sandstorm identification method based on remote sensing images using CTFusionNet according to claim 4, characterized in that: The first 1*1 convolutional layer is connected to the input end of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence; The 3*3 convolution layer is connected to the input end of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence; The 5*5 convolution layer is connected to the input end of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence; The second 1*1 convolutional layer is connected to the input end of the element-by-element addition unit of the RP module through a normalization layer and a ReLU activation function layer in sequence.

6. The sandstorm identification method based on remote sensing images of CTFusionNet according to claim 5 is characterized in that: The convolution step size of the first 1*1 convolution layer, 3*3 convolution layer, 5*5 convolution layer, and the second 1*1 convolution layer are all 1.

7. The sandstorm identification method based on remote sensing images of CTFusionNet according to claim 1, characterized in that: Get the trained CTFusionNet based on the following steps: Obtain a training set; wherein the training set consists of remote sensing images, each pixel of each remote sensing image is labeled, and each pixel is a training sample, and the label is sandstorm or non-sandstorm; The CTFusionNet is trained using the training set to obtain a trained CTFusionNet. During the training process, the network parameters of the CTFusionNet are updated by minimizing the total loss function.

8. The sandstorm identification method based on remote sensing images of CTFusionNet according to claim 7 is characterized in that: The expression of the total loss function is: L total =L CE +L SCBCE : Where, L total Represents the total loss function; L CE represents the cross entropy loss of the first classifier; L SCBCE represents the selective confidence cross entropy loss for the second classifier.

9. The sandstorm identification method based on remote sensing images of CTFusionNet according to claim 8, characterized in that: L CE The expression is: Where H represents the height of the remote sensing image; W represents the width of the remote sensing image; Y ij represents the true label of the pixel in the i-th row and j-th column of the remote sensing image; P ij It represents the confidence of the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier, that is, the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier is a sandstorm.

10. The sandstorm identification method based on remote sensing images of CTFusionNet according to claim 8, characterized in that: L SCBCE The expression is: Where m ij represents the mask of the pixel in row i and column j of the remote sensing image; σ represents the preset confidence threshold; P ij ' represents the confidence of the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier, that is, the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier is a sandstorm.

Images