A method for identifying dust storms from remote sensing images based on CTFusionNet

Abstract

This invention discloses a method for identifying dust storms in remote sensing images based on CTFusionNet, belonging to the field of remote sensing image processing. The method includes the following steps: inputting the remote sensing image to be identified into a trained CTFusionNet to obtain the dust storm identification result of the remote sensing image; wherein, CTFusionNet includes a CNN branch, a downsampling layer, an embedding layer, a Transformer branch, an upsampling layer, and a feature overlay fusion layer connected sequentially from input to output; a first classifier connected to the output of the CNN branch; and a second classifier connected to the output of the feature overlay fusion layer. This invention can improve the accuracy of dust storm identification in remote sensing images.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and more specifically to a method for identifying sandstorms in remote sensing images based on CTFusionNet. Background Technology

[0002] Currently, dust storm identification is a crucial step in desertification control and prevention. Satellite remote sensing technology, with its rich spectral information and real-time monitoring capabilities, has become the main research method in the field of dust storm identification. Traditional fixed threshold methods suffer from low accuracy in dust storm identification because they cannot adapt to different background environments.

[0003] Therefore, how to provide a method for identifying dust storms that can improve the accuracy of dust storm identification in remote sensing images is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method for identifying sandstorms from remote sensing images based on CTFusionNet.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for identifying dust storms from remote sensing images based on CTFusionNet includes the following steps:

[0007] The remote sensing image to be identified is input into the trained CTFusionNet to obtain the sandstorm identification result of the remote sensing image to be identified;

[0008] CTFusionNet includes a CNN branch, a downsampling layer, an embedding layer, a Transformer branch, an upsampling layer, and a feature overlay fusion layer connected sequentially from input to output; a first classifier connected to the output of the CNN branch; and a second classifier connected to the output of the feature overlay fusion layer.

[0009] Preferably, the remote sensing image to be identified is input into a CNN branch to obtain local feature maps. Local feature map Local feature map Local feature map and local feature maps

[0010] Local feature map Local feature map Local feature map Local feature map Local feature map The data are sequentially input into the downsampling layer and the embedding layer to obtain the intermediate feature map F'.

[0011] Local feature map The input is fed into the first classifier to obtain the first classification result;

[0012] The intermediate feature map F' is sequentially fed into the Transformer branch and the upsampling layer to obtain the global feature map.

[0013] Local feature map and global feature map The input is fed into the feature overlay and fusion layer and then into the second classifier to obtain the second classification result.

[0014] The classification result with higher confidence between the first classification result and the second classification result is used as the dust storm identification result of the remote sensing image to be identified.

[0015] Preferably, the CNN branch includes a first RP module, a second RP module, a third RP module, a fourth RP module, and a fifth RP module connected sequentially from input to output;

[0016] The Transformer branch includes a first multi-head attention layer, a second multi-head attention layer, a third multi-head attention layer, a fourth multi-head attention layer, a fifth multi-head attention layer, a sixth multi-head attention layer, a seventh multi-head attention layer, an eighth multi-head attention layer, a ninth multi-head attention layer, a tenth multi-head attention layer, an eleventh multi-head attention layer, a twelfth multi-head attention layer, a multilayer perceptron layer, and an element-wise addition unit, all connected sequentially from input to output.

[0017] The remote sensing image to be identified is processed by the first RP module to obtain a local feature map.

[0018] Local feature map After processing by the second RP module, a local feature map is obtained.

[0019] Local feature map After processing by the third RP module, a local feature map is obtained.

[0020] Local feature map After processing by the fourth RP module, a local feature map is obtained.

[0021] Local feature map After processing by the fifth RP module, a local feature map is obtained.

[0022] Local feature map The first classification result is obtained after classification by the first classifier;

[0023] Local feature map Local feature map Local feature map Local feature map Local feature map After being processed by the downsampling layer and the embedding layer respectively, the intermediate feature map F' is obtained;

[0024] The intermediate feature map F' is processed sequentially through the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, eleventh, and twelfth multi-head attention layers and the multilayer perceptron layer to obtain the global feature map.

[0025] Intermediate feature map F' and global feature map After being processed by the element-wise addition unit in the Transformer branch, the global feature map is obtained.

[0026] Global feature map After upsampling, the global feature map is obtained.

[0027] Local feature map and global feature map After being processed by the feature overlay and fusion layer, the result is input into the second classifier to obtain the second classification result.

[0028] Preferably, the first RP module, the second RP module, the third RP module, the fourth RP module and the fifth RP module have the same structure, each including a first 1*1 convolutional layer, a 3*3 convolutional layer and a 5*5 convolutional layer, a channel dimension splicing layer, a second 1*1 convolutional layer and an element-wise addition unit;

[0029] The input terminals of the first 1*1 convolutional layer, the 3*3 convolutional layer, and the 5*5 convolutional layer are connected to form the input terminals of the RP module.

[0030] The outputs of the first 1*1 convolutional layer, the 3*3 convolutional layer, and the 5*5 convolutional layer are all connected to the input of the channel dimension splicing layer.

[0031] The output of the channel dimension splicing layer is connected to the input of the second 1*1 convolutional layer;

[0032] The output of the second 1*1 convolutional layer is connected to the input of the element-wise addition unit of the RP module;

[0033] The input of the element-wise addition unit of the RP module is connected to the input of the first 1*1 convolutional layer;

[0034] The output of the element-by-element summation unit of the RP module constitutes the output of the RP module.

[0035] Preferably, the first 1*1 convolutional layer is connected to the input of the channel dimension splicing layer in sequence through a normalization layer and a ReLU activation function layer;

[0036] The 3*3 convolutional layer is connected to the input of the channel dimension splicing layer in sequence through a normalization layer and a ReLU activation function layer;

[0037] The 5*5 convolutional layer is connected to the input of the channel dimension splicing layer in sequence through a normalization layer and a ReLU activation function layer.

[0038] The second 1*1 convolutional layer is connected to the input of the element-wise addition unit of the RP module through a normalization layer and a ReLU activation function layer in sequence.

[0039] Preferably, the convolution stride of the first 1*1 convolutional layer, the 3*3 convolutional layer, the 5*5 convolutional layer, and the second 1*1 convolutional layer is 1.

[0040] Preferably, the trained CTFusionNet is obtained based on the following steps;

[0041] Obtain the training set; the training set consists of remote sensing images, each pixel of each remote sensing image is labeled, one pixel is a training sample, and the label is sandstorm or non-sandstorm.

[0042] CTFusionNet is trained using the training set to obtain a trained CTFusionNet; during the training process, the network parameters of CTFusionNet are updated by minimizing the total loss function.

[0043] Preferably, the expression for the total loss function is:

[0044] L total =L CE +L SCBCE :

[0045] In the formula, L total L represents the total loss function; CE L represents the cross-entropy loss of the first classifier; SCBCE This represents the selective confidence cross-entropy loss of the second classifier.

[0046] Preferably, L CEThe expression is:

[0047]

[0048] In the formula, H represents the height of the remote sensing image; W represents the width of the remote sensing image; Y ij P represents the true label of the pixel in the i-th row and j-th column of the remote sensing image; ij This represents the confidence level of the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier, that is, the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier is a sandstorm.

[0049] Preferably, L SCBCE The expression is:

[0050]

[0051]

[0052] In the formula, m ij σ represents the mask of the pixel in the i-th row and j-th column of the remote sensing image; σ represents the preset confidence threshold; P ij ' represents the confidence level of the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier, that is, the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier is a sandstorm.

[0053] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a method for identifying sandstorms in remote sensing images based on CTFusionNet, which can improve the accuracy of sandstorm identification in remote sensing images. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of CTFusionNet provided in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the RP module provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] This invention discloses a method for identifying sandstorms from remote sensing images based on CTFusionNet, comprising the following steps:

[0059] The remote sensing image to be identified is input into the trained CTFusionNet to obtain the sandstorm identification result of the remote sensing image to be identified;

[0060] CTFusionNet includes a CNN branch, a downsampling layer, an embedding layer, a Transformer branch, an upsampling layer, and a feature overlay fusion layer connected sequentially from input to output; a first classifier connected to the output of the CNN branch; and a second classifier connected to the output of the feature overlay fusion layer.

[0061] In one embodiment, the remote sensing image to be identified is input into a CNN branch to obtain a local feature map. Local feature map Local feature map Local feature map and local feature maps

[0062] Local feature map Local feature map Local feature map Local feature map Local feature map The data are sequentially input into the downsampling layer and the embedding layer to obtain the intermediate feature map F'.

[0063] Local feature map The input is fed into the first classifier to obtain the first classification result;

[0064] The intermediate feature map F' is sequentially fed into the Transformer branch and the upsampling layer to obtain the global feature map.

[0065] Local feature map and global feature map The input is fed into the feature overlay and fusion layer and then into the second classifier to obtain the second classification result.

[0066] The classification result with higher confidence between the first classification result and the second classification result is used as the dust storm identification result of the remote sensing image to be identified.

[0067] In one embodiment, such as Figure 1 As shown, the CNN branch includes a first RP module, a second RP module, a third RP module, a fourth RP module, and a fifth RP module connected sequentially from input to output;

[0068] The Transformer branch includes a first multi-head attention layer, a second multi-head attention layer, a third multi-head attention layer, a fourth multi-head attention layer, a fifth multi-head attention layer, a sixth multi-head attention layer, a seventh multi-head attention layer, an eighth multi-head attention layer, a ninth multi-head attention layer, a tenth multi-head attention layer, an eleventh multi-head attention layer, a twelfth multi-head attention layer, a multilayer perceptron layer, and an element-wise addition unit, all connected sequentially from input to output.

[0069] The remote sensing image to be identified is processed by the first RP module to obtain a local feature map.

[0070] Local feature map After processing by the second RP module, a local feature map is obtained.

[0071] Local feature map After processing by the third RP module, a local feature map is obtained.

[0072] Local feature map After processing by the fourth RP module, a local feature map is obtained.

[0073] Local feature map After processing by the fifth RP module, a local feature map is obtained.

[0074] Local feature map The first classification result is obtained after classification by the first classifier;

[0075] Local feature map Local feature map Local feature map Local feature map Local feature map After being processed by the downsampling layer and the embedding layer respectively, the intermediate feature map F' is obtained;

[0076] It is understandable that downsampling layers are used to downsample high-resolution feature maps to a lower spatial resolution.

[0077] The downsampling layer uses a convolutional layer or a pooling layer;

[0078] The intermediate feature map F' output by the embedding layer can be viewed as a series of "tokens". Assume the feature map F obtained after downsampling... down It will be transformed into a low-dimensional representation suitable for Transformer processing through the embedding layer. Assume the weight matrix of the embedding layer is W. embed Then the intermediate feature map F' can be represented as [N,D], where N represents the number of "tokens" in the feature map and D represents the feature dimension after embedding.

[0079] F' = W embed *F down ;

[0080] In the formula, W embed This is the weight matrix of the embedding layer, responsible for mapping each "token" from the original feature space to the embedding space, generating a feature representation suitable for the Transformer. The intermediate feature map F' can be viewed as a set of "tokens," each representing a feature of a local region in the image. These "tokens" will serve as input to the Transformer branch to obtain global features.

[0081] The intermediate feature map F' is processed sequentially through the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, eleventh, and twelfth multi-head attention layers and the multilayer perceptron layer to obtain the global feature map.

[0082] Intermediate feature map F' and global feature map After being processed by the element-wise addition unit in the Transformer branch, the global feature map is obtained.

[0083] Global feature map After upsampling, the global feature map is obtained.

[0084] It is understandable that the upsampling layer is used to convert low-resolution global feature maps. Restore to a high-resolution global feature map

[0085] Local feature map and global feature map After being processed by the feature overlay and fusion layer, the result is input into the second classifier to obtain the second classification result.

[0086] It should be noted that the dimensions of the remote sensing image to be identified are H*W*C; where H, W, and C represent the height, width, and number of channels of the remote sensing image to be identified, respectively.

[0087] Local feature map The dimensions are H*W*C;

[0088] Local feature map The dimensions are H*W*C;

[0089] Local feature map The dimensions are H*W*C;

[0090] Local feature map The dimensions are H*W*C;

[0091] Local feature map The dimensions are H*W*C;

[0092] The size of the intermediate feature map F' is H'*W'*5C; where H' and W' are 5 local feature maps (i.e., local feature maps). Local feature map Local feature map Local feature map Local feature map The width and height after downsampling;

[0093] Global feature map The dimensions are H'*W'*5C;

[0094] Global feature map The dimensions are H'*W'*5C;

[0095] Global feature map The dimensions are H*W*C.

[0096] In one embodiment, such as Figure 2 As shown, the first RP module, the second RP module, the third RP module, the fourth RP module and the fifth RP module have the same structure, all including a first 1*1 convolutional layer, a 3*3 convolutional layer and a 5*5 convolutional layer, a channel dimension splicing layer, a second 1*1 convolutional layer and an element-wise addition unit;

[0097] The input terminals of the first 1*1 convolutional layer, the 3*3 convolutional layer, and the 5*5 convolutional layer are connected to form the input terminals of the RP module.

[0098] The outputs of the first 1*1 convolutional layer, the 3*3 convolutional layer, and the 5*5 convolutional layer are all connected to the input of the channel dimension splicing layer.

[0099] The output of the channel dimension splicing layer is connected to the input of the second 1*1 convolutional layer;

[0100] The output of the second 1*1 convolutional layer is connected to the input of the element-wise addition unit of the RP module;

[0101] The input of the element-wise addition unit of the RP module is connected to the input of the first 1*1 convolutional layer;

[0102] The output of the element-by-element summation unit of the RP module constitutes the output of the RP module.

[0103] In one embodiment, the convolution stride of the first 1*1 convolutional layer, the 3*3 convolutional layer, the 5*5 convolutional layer, and the second 1*1 convolutional layer is all 1.

[0104] In this embodiment:

[0105] 1) The specific processing procedure of the first RP module is as follows:

[0106] The remote sensing image to be identified (with dimensions H*W*C) is input into the first 1*1 convolutional layer of the first RP module to obtain local feature maps. (Dimensions are H*W*C / 4);

[0107] The remote sensing image to be identified (with dimensions H*W*C) is input into a 3*3 convolutional layer in the first RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0108] The remote sensing image to be identified (with dimensions H*W*C) is input into a 5*5 convolutional layer in the first RP module to obtain local feature maps. (Dimensions are H*W*C);

[0109] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the first RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0110] splicing feature maps The input is fed into the second 1*1 convolutional layer in the first RP module to obtain local feature maps. (Dimensions are H*W*C);

[0111] Compare the remote sensing image to be identified with local feature maps The input is fed into the element-wise addition unit in the first RP module to obtain the local feature map. (Dimensions are H*W*C);

[0112] 2) The specific processing procedure of the second RP module is as follows:

[0113] Local feature map The data (with dimensions H*W*C) is input into the first 1*1 convolutional layer of the second RP module to obtain the local feature map. (Dimensions are H*W*C / 4);

[0114] Local feature map The data (with dimensions H*W*C) is input into a 3*3 convolutional layer in the second RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0115] Local feature map The data (with dimensions H*W*C) is input into a 5*5 convolutional layer in the second RP module to obtain local feature maps. (Dimensions are H*W*C);

[0116] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the second RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0117] splicing feature maps The input is fed into the second 1x1 convolutional layer in the second RP module to obtain local feature maps. (Dimensions are H*W*C);

[0118] Local feature map With local feature map The input is fed into the element-wise addition unit in the second RP module to obtain the local feature map. (Dimensions are H*W*C);

[0119] 3) The specific processing procedure of the third RP module is as follows:

[0120] Local feature map The data (with dimensions H*W*C) is input into the first 1*1 convolutional layer of the third RP module to obtain the local feature map. (Dimensions are H*W*C / 4);

[0121] Local feature map The data (with dimensions H*W*C) is input into a 3*3 convolutional layer in the third RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0122] Local feature map The data (with dimensions H*W*C) is input into a 5*5 convolutional layer in the third RP module to obtain local feature maps. (Dimensions are H*W*C);

[0123] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the third RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0124] splicing feature maps The input is fed into the second 1x1 convolutional layer in the third RP module to obtain local feature maps. (Dimensions are H*W*C);

[0125] Local feature map With local feature map The input is fed into the element-wise addition unit in the third RP module to obtain the local feature map. (Dimensions are H*W*C);

[0126] 4) The specific processing procedure of the fourth RP module is as follows:

[0127] Local feature map The input (with dimensions H*W*C) is fed into the first 1*1 convolutional layer in the fourth RP module to obtain the local feature map. (Dimensions are H*W*C / 4);

[0128] Local feature map The data (with dimensions H*W*C) is input into a 3*3 convolutional layer in the fourth RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0129] Local feature map The data (with dimensions H*W*C) is input into a 5*5 convolutional layer in the fourth RP module to obtain local feature maps. (Dimensions are H*W*C);

[0130] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the fourth RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0131] splicing feature maps The input is fed into the second 1x1 convolutional layer in the fourth RP module to obtain local feature maps. (Dimensions are H*W*C);

[0132] Local feature map With local feature map The input is fed into the element-wise addition unit in the fourth RP module to obtain the local feature map. (Dimensions are H*W*C);

[0133] 5) The specific processing procedure of the fifth RP module is as follows:

[0134] Local feature map The data (with dimensions H*W*C) is input into the first 1*1 convolutional layer of the fifth RP module to obtain the local feature map. (Dimensions are H*W*C / 4);

[0135] Local feature map The data (with dimensions H*W*C) is input into a 3*3 convolutional layer in the fifth RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0136] Local feature map The data (with dimensions H*W*C) is input into a 5*5 convolutional layer in the fifth RP module to obtain local feature maps. (Dimensions are H*W*C);

[0137] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the fifth RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0138] splicing feature maps The input is fed into the second 1x1 convolutional layer in the fifth RP module to obtain local feature maps. (Dimensions are H*W*C);

[0139] Local feature map With local feature map The input is fed into the element-wise addition unit in the fifth RP module to obtain the local feature map. (Dimensions are H*W*C);

[0140] It is understandable that:

[0141] In this invention, the first 1×1 convolutional layer is responsible for extracting local fine features, while the 3×3 convolutional layer and the 5×5 convolutional layer capture contextual features of medium and larger areas, respectively.

[0142] This invention avoids downsampling by setting the stride of all convolutional layers to 1, thereby ensuring that the spatial resolution of the feature map is preserved in each layer, and further providing rich detailed information for subsequent sandstorm identification tasks.

[0143] In summary, to comprehensively capture features at different scales, the CNN branch employs a multi-scale convolution strategy. Each RP module contains three different sized convolutional kernels: 1×1, 3×3, and 5×5. Through the combination of these kernels, the CNN branch can simultaneously capture local, intermediate, and broader contextual information while maintaining spatial resolution.

[0144] The element-wise addition unit design in the RP module of this invention not only effectively alleviates gradient vanishing but also enhances the representational power of the feature map while maintaining spatial resolution. In other words, the RP module of this invention can better capture deep features, improving the model's performance when processing complex images, especially in detail extraction from remote sensing images.

[0145] This invention performs dimensionality reduction using a second 1×1 convolutional layer, while simultaneously designing the stride of this second 1×1 convolutional layer to be 1. This reduces computational cost while maintaining the resolution of the feature map. This approach effectively fuses features at different scales while avoiding the loss of detail caused by changes in feature map size.

[0146] The RP module of this invention performs refined feature processing, which can effectively extract rich local and global feature information without sacrificing spatial resolution. This provides a more accurate and robust feature representation for subsequent sandstorm identification tasks, thereby improving the model's segmentation performance in complex scenes.

[0147] In one embodiment, the first 1*1 convolutional layer is connected to the input of the channel dimension splicing layer through a normalization layer and a ReLU activation function layer in sequence;

[0148] The 3*3 convolutional layer is connected to the input of the channel dimension splicing layer in sequence through a normalization layer and a ReLU activation function layer;

[0149] The 5*5 convolutional layer is connected to the input of the channel dimension splicing layer in sequence through a normalization layer and a ReLU activation function layer.

[0150] The second 1*1 convolutional layer is connected to the input of the element-wise addition unit of the RP module through a normalization layer and a ReLU activation function layer in sequence.

[0151] In this embodiment,

[0152] 1) The specific processing procedure of the first RP module is as follows:

[0153] The remote sensing image to be identified (with dimensions H*W*C) is input into the first 1*1 convolutional layer of the first RP module to obtain local feature maps. (Dimensions are H*W*C / 4);

[0154] The remote sensing image to be identified (with dimensions H*W*C) is input into a 3*3 convolutional layer in the first RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0155] The remote sensing image to be identified (with dimensions H*W*C) is input into a 5*5 convolutional layer in the first RP module to obtain local feature maps. (Dimensions are H*W*C);

[0156] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the first RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0157] splicing feature maps The input is fed into the second 1*1 convolutional layer in the first RP module to obtain local feature maps. (Dimensions are H*W*C);

[0158] Compare the remote sensing image to be identified with local feature maps The input is fed into the element-wise addition unit in the first RP module to obtain the local feature map. (Dimensions are H*W*C);

[0159] 2) The specific processing procedure of the second RP module is as follows:

[0160] Local feature map The data (with dimensions H*W*C) is input into the first 1*1 convolutional layer of the second RP module to obtain the local feature map. (Dimensions are H*W*C / 4);

[0161] Local feature map The data (with dimensions H*W*C) is input into a 3*3 convolutional layer in the second RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0162] Local feature map The data (with dimensions H*W*C) is input into a 5*5 convolutional layer in the second RP module to obtain local feature maps. (Dimensions are H*W*C);

[0163] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the second RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0164] splicing feature maps The input is fed into the second 1x1 convolutional layer in the second RP module to obtain local feature maps. (Dimensions are H*W*C);

[0165] Local feature map With local feature map The input is fed into the element-wise addition unit in the second RP module to obtain the local feature map. (Dimensions are H*W*C);

[0166] 3) The specific processing procedure of the third RP module is as follows:

[0167] Local feature map The data (with dimensions H*W*C) is input into the first 1*1 convolutional layer of the third RP module to obtain the local feature map. (Dimensions are H*W*C / 4);

[0168] Local feature map The data (with dimensions H*W*C) is input into a 3*3 convolutional layer in the third RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0169] Local feature map The data (with dimensions H*W*C) is input into a 5*5 convolutional layer in the third RP module to obtain local feature maps. (Dimensions are H*W*C);

[0170] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the third RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0171] splicing feature maps The input is fed into the second 1x1 convolutional layer in the third RP module to obtain local feature maps. (Dimensions are H*W*C);

[0172] Local feature map With local feature map The input is fed into the element-wise addition unit in the third RP module to obtain the local feature map. (Dimensions are H*W*C);

[0173] 4) The specific processing procedure of the fourth RP module is as follows:

[0174] Local feature map The input (with dimensions H*W*C) is fed into the first 1*1 convolutional layer in the fourth RP module to obtain the local feature map. (Dimensions are H*W*C / 4);

[0175] Local feature map The data (with dimensions H*W*C) is input into a 3*3 convolutional layer in the fourth RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0176] Local feature map The data (with dimensions H*W*C) is input into a 5*5 convolutional layer in the fourth RP module to obtain local feature maps. (Dimensions are H*W*C);

[0177] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the fourth RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0178] splicing feature maps The input is fed into the second 1x1 convolutional layer in the fourth RP module to obtain local feature maps. (Dimensions are H*W*C);

[0179] Local feature map With local feature map The input is fed into the element-wise addition unit in the fourth RP module to obtain the local feature map. (Dimensions are H*W*C);

[0180] 5) The specific processing procedure of the fifth RP module is as follows:

[0181] Local feature map The data (with dimensions H*W*C) is input into the first 1*1 convolutional layer of the fifth RP module to obtain the local feature map. (Dimensions are H*W*C / 4);

[0182] Local feature map The data (with dimensions H*W*C) is input into a 3*3 convolutional layer in the fifth RP module to obtain local feature maps. (Dimensions are H*W*C / 2);

[0183] Local feature map The data (with dimensions H*W*C) is input into a 5*5 convolutional layer in the fifth RP module to obtain local feature maps. (Dimensions are H*W*C);

[0184] Local feature map Local feature map and local feature maps The input is fed into the channel-dimensional stitching layer in the fifth RP module to obtain the stitched feature map. (Dimensions are H*W*(C+C / 2+C / 4));

[0185] splicing feature maps The input is fed into the second 1x1 convolutional layer in the fifth RP module to obtain local feature maps. (Dimensions are H*W*C);

[0186] Local feature map With local feature map The input is fed into the element-wise addition unit in the fifth RP module to obtain the local feature map. (Dimensions are H*W*C);

[0187] In one embodiment, the trained CTFusionNet is obtained based on the following steps;

[0188] Obtain the training set; the training set consists of remote sensing images, each pixel of each remote sensing image is labeled, one pixel is a training sample, and the label is sandstorm or non-sandstorm.

[0189] CTFusionNet is trained using the training set to obtain a trained CTFusionNet; during the training process, the network parameters of CTFusionNet are updated by minimizing the total loss function.

[0190] In one embodiment, the expression for the total loss function is:

[0191] L total =L CE +L SCBCE ;

[0192]

[0193] In the formula, Ltotal L represents the total loss function; CE L represents the cross-entropy loss of the first classifier; SCBCE Y represents the selective confidence cross-entropy loss of the second classifier; H represents the height of the remote sensing image; W represents the width of the remote sensing image; Y represents the height of the remote sensing image. ij P represents the true label of the pixel in the i-th row and j-th column of the remote sensing image; ij This represents the confidence level of the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier, i.e., the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the first classifier is a sandstorm; m ij σ represents the mask of the pixel in the i-th row and j-th column of the remote sensing image; σ represents the preset confidence threshold (in one embodiment, σ is 0.5); P ij ' represents the confidence level of the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier, that is, the predicted probability that the pixel in the i-th row and j-th column of the remote sensing image output by the second classifier is a sandstorm.

[0194] Finally, the CTFusionNet of this invention was compared with RF, UNet, DeepLabV3+, HRNet, Swin Transformer, and CTFusionNet, as shown in Table 1:

[0195] Table 1

[0196]

[0197] As shown in Table 1:

[0198] Regarding the OA value: CTFusionNet performed best with 0.88, followed by SwinTransformer with 0.87, HRNet and DeeplabV3+ both with 0.80, which also performed well, while RF and UNet were relatively low with 0.77 and 0.63 respectively.

[0199] Therefore, it can be seen that the CTFusionNet of the present invention has strong overall accuracy.

[0200] Regarding the Kappa value: The Kappa value is used to measure the correlation between the prediction results and the true labels. CTFusionNet has a Kappa value of 0.78, which is slightly better than SwinTransformer's 0.75. RF and UNet have Kappa values ​​of only 0.48 and 0.52, respectively. Their lower Kappa values ​​indicate that the correlation between the prediction results of these two models and the true labels is weak.

[0201] Therefore, it can be seen that the CTFusionNet of the present invention has better classification consistency.

[0202] Regarding the IoU (Intersection over Union) value: CTFusionNet and SwinTransformer scored 0.69 and 0.62 respectively, outperforming other models in IoU, indicating that these two models are more accurate in predicting the target region. However, other models, UNet and HRNet, scored slightly lower at 0.57, while RF scored even lower at only 0.52.

[0203] Therefore, it can be seen that the CTFusionNet of this invention is superior to other models in handling the overlap of the target region.

[0204] In summary, this invention can significantly improve the accuracy of identifying sandstorms in remote sensing images.

[0205] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0206] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dust storm identification method based on CTFusionNet remote sensing image, characterized in that, The method comprises the following steps: inputting a to-be-identified remote sensing image into the trained CTFusionNet to obtain a sandstorm identification result of the to-be-identified remote sensing image; The CTFusionNet comprises a CNN branch, a down-sampling layer, an embedding layer, a Transformer branch, an up-sampling layer and a feature superposition fusion layer connected in sequence from input to output; and a first classifier connected to the output end of the CNN branch; and a second classifier connected to the output end of the feature superposition fusion layer; Input the remote sensing image to be identified into the CNN branch to obtain a local feature map , the local feature map , the local feature map , the local feature map , and the local feature map ; input the local feature map , the local feature map , the local feature map , the local feature map , the local feature map to the down-sampling layer and the embedding layer respectively in sequence to obtain the intermediate feature map ; the local feature maps are input to the first classifier to obtain a first classification result; input the intermediate feature map input to the Transformer branch and the upsampling layer in sequence to obtain a global feature map ; input the local feature map and the global feature map to a feature superposition fusion layer and then to a second classifier to obtain a second classification result; The classification result with higher confidence in the first classification result and the second classification result is taken as the sandstorm identification result of the to-be-identified remote sensing image.

2. The dust storm identification method based on CTFusionNet of the remote sensing image according to claim 1, characterized in that, The CNN branch comprises a first RP module, a second RP module, a third RP module, a fourth RP module and a fifth RP module connected in sequence from input to output; The Transformer branch comprises a first multi-head attention layer, a second multi-head attention layer, a third multi-head attention layer, a fourth multi-head attention layer, a fifth multi-head attention layer, a sixth multi-head attention layer, a seventh multi-head attention layer, an eighth multi-head attention layer, a ninth multi-head attention layer, a tenth multi-head attention layer, an eleventh multi-head attention layer, a twelfth multi-head attention layer, a multi-layer perception layer and an element-wise addition unit connected in sequence from input to output; The remote sensing image to be identified is processed by the first RP module to obtain a local feature map ; local feature map obtaining a local feature map through the second RP module ; local feature map After the third RP module processing, a local feature map is obtained ; local feature map After the fourth RP module processing, a local feature map is obtained ; local feature map After the fifth RP module processing, a local feature map is obtained ; Local feature map Classifying through the first classifier to obtain the first classification result; local feature map , local feature map , local feature map , local feature map , local feature map , local feature map ​ intermediate feature map after sequentially passing through the first multi-head attention layer, the second multi-head attention layer, the third multi-head attention layer, the fourth multi-head attention layer, the fifth multi-head attention layer, the sixth multi-head attention layer, the seventh multi-head attention layer, the eighth multi-head attention layer, the ninth multi-head attention layer, the tenth multi-head attention layer, the eleventh multi-head attention layer, the twelfth multi-head attention layer, and the multi-layer perception layer, a global feature map is obtained ; intermediate feature map and global feature map After the element-wise addition unit processing of the input to the Transformer branch, the global feature map is obtained ; global feature map After the up-sampling layer processing, a global feature map is obtained ; local feature maps and global feature maps input to the second classifier after being processed by the feature superposition fusion layer, to obtain the second classification result.

3. The method of claim 2, wherein the method is based on a CTFusionNet. The first RP module, the second RP module, the third RP module, the fourth RP module and the fifth RP module have the same structure and each comprises a first 1*1 convolution layer, a 3*3 convolution layer and a 5*5 convolution layer, a channel dimension concatenation layer, a second 1*1 convolution layer and an element-wise addition unit; The input end of the first 1*1 convolution layer, the input end of the 3*3 convolution layer and the input end of the 5*5 convolution layer are connected together to form the input end of the RP module; The output end of the first 1*1 convolution layer, the output end of the 3*3 convolution layer and the output end of the 5*5 convolution layer are connected to the input end of the channel dimension concatenation layer; The output end of the channel dimension concatenation layer is connected to the input end of the second 1*1 convolution layer; The output end of the second 1*1 convolution layer is connected to the input end of the element-wise addition unit of the RP module; The input end of the element-wise addition unit of the RP module is connected to the input end of the first 1*1 convolution layer; The output end of the element-wise addition unit of the RP module forms the output end of the RP module.

4. The sandstorm identification method for remote sensing images based on CTFusionNet according to claim 3, wherein: The first 1*1 convolution layer is connected to the input end of the channel dimension concatenation layer through a normalization layer and a ReLU activation function layer in sequence; The 3*3 convolution layer is connected to the input end of the channel dimension concatenation layer through a normalization layer and a ReLU activation function layer in sequence; The 5*5 convolution layer is connected to the input end of the channel dimension concatenation layer through a normalization layer and a ReLU activation function layer in sequence; The second 1*1 convolution layer is connected to the input end of the element-wise addition unit of the RP module through a normalization layer and a ReLU activation function layer in sequence.

5. The method of claim 4, wherein the method is based on a CTFusionNet. The convolution step length of the first 1*1 convolution layer, the 3*3 convolution layer, the 5*5 convolution layer and the second 1*1 convolution layer is 1. 6.The dust storm identification method based on CTFusionNet of remote sensing image according to claim 1, characterized in that, The trained CTFusionNet is obtained based on the following steps; Obtain a training set; wherein the training set is composed of remote sensing images, each pixel of each remote sensing image is provided with a label, one pixel is one training sample, and the label is a sandstorm or a non-sandstorm; The trained CTFusionNet is obtained by training the CTFusionNet using the training set; wherein the network parameters of the CTFusionNet are updated by minimizing the total loss function during the training process.

7. The method of claim 6, wherein the method is based on a CTFusionNet. The expression of the total loss function is: ： wherein represents the total loss function; represents the cross-entropy loss of the first classifier; represents the selective confidence cross-entropy loss of the second classifier.

8. The method of claim 7, wherein the method is based on a CTFusionNet. The expression is: ； In the formula, H represents the height of the remote sensing image; W represents the width of the remote sensing image; represents the true label of the i-th row and j-th column pixel of the remote sensing image; represents the confidence of the i-th row and j-th column pixel of the remote sensing image output by the first classifier, that is, the prediction probability of the i-th row and j-th column pixel of the remote sensing image output by the first classifier being a sandstorm.

9. The method of claim 7, wherein the method is based on a CTFusionNet-based dust storm recognition method for remote sensing images. The expression is: ； ； In the formula, represents a mask of the i-th row and the j-th column pixel of the remote sensing image; represents a preset confidence threshold value; represents the confidence of the i-th row and the j-th column pixel of the remote sensing image output by the second classifier, that is, the prediction probability of the i-th row and the j-th column pixel of the remote sensing image output by the second classifier being a sandstorm.

Images