Remote sensing image shadow detection method and system based on double-branch encoder

Abstract

The invention relates to the technical field of shadow detection, and provides a remote sensing image shadow detection method and system based on a double-branch encoder. The method comprises the following steps: acquiring a remote sensing image shadow detection data set, performing image enhancement processing on each remote sensing image in the remote sensing image shadow detection data set, and dividing the enhanced remote sensing image shadow detection data set into a training set and a test set; constructing a neural network structure based on a double-branch encoder, and training the neural network structure based on the double-branch encoder by using the training set and the test set to obtain a remote sensing image shadow detection model; and performing image enhancement processing on a to-be-detected remote sensing image, inputting the remote sensing image shadow detection model for shadow detection processing, and outputting a detection result. According to the remote sensing image shadow detection method, the double-branch encoder neural network is constructed, the shallow layer features are effectively multiplexed through multi-scale feature extraction, and the accuracy and robustness of remote sensing image shadow detection are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of shadow detection, and in particular to a remote sensing image shadow detection method and system based on a dual-branch encoder. Background Art

[0002] With the continuous development of remote sensing technology, remote sensing images have become one of the important means to obtain information about the earth's surface. However, in the process of image acquisition, due to the influence of factors such as lighting conditions and terrain, there are often a large number of shadows in the image, which brings certain troubles to the subsequent image analysis and application. Therefore, how to accurately detect and remove shadows in remote sensing images has become one of the important issues in remote sensing image processing. Shadows are usually manifested as areas with low pixel values ​​in remote sensing images. Their formation is due to insufficient light caused by ground objects blocking the sun's rays. Therefore, the basic principle of shadow detection is to identify areas with insufficient light by analyzing the spectral and spatial characteristics of pixels in the image, thereby realizing shadow detection and segmentation. However, although the existing methods have achieved remarkable results, there are still some deficiencies in the shadow extraction process, and the accuracy of shadow extraction still needs to be improved.

[0003] (1) Shadows can be superimposed on different planes to generate more complex features. For example, shadows present different features when superimposed on brighter or darker features. In addition, water bodies, vegetation, and dark objects are spectrally similar to shadows in remote sensing images, which leads to greater spectral heterogeneity. In the process of shadow extraction based on convolutional neural networks, the encoding stage only uses convolution operations, which often only focuses on detail information and easily ignores global information, resulting in the network's lack of shadow feature extraction capabilities.

[0004] (2) In the real world, various objects with height will produce shadows due to the occlusion of sunlight. In an image, different objects have different scale and morphological characteristics, and shadows combine the characteristics of all scales and shapes, showing greater scale and morphological differences. However, the current network structure cannot fully integrate the shadow features of different scales, and the network cannot understand the multi-scale characteristics of shadows, resulting in reduced shadow prediction accuracy at different scales. Summary of the invention

[0005] In order to solve the problem of low accuracy and robustness of remote sensing image shadow detection, the present invention provides a remote sensing image shadow detection method based on a dual-branch encoder, constructs a dual-branch encoder neural network, effectively reuses shallow features through multi-scale feature extraction, and makes full use of feature information at each scale, thereby improving the detection ability of the model and effectively improving the accuracy and robustness of remote sensing image shadow detection.

[0006] In a first aspect, the present invention provides a remote sensing image shadow detection method based on a dual-branch encoder, comprising:

[0007] Acquire a remote sensing image shadow detection dataset, perform image enhancement processing on each remote sensing image in the remote sensing image shadow detection dataset, and divide the enhanced remote sensing image shadow detection dataset into a training set and a test set;

[0008] Constructing a neural network structure based on a dual-branch encoder, and using the training set and the test set to train the neural network structure based on the dual-branch encoder to obtain a remote sensing image shadow detection model; wherein the neural network structure based on the dual-branch encoder includes a dual-branch encoder module, a bottleneck module, a jump connection module and a decoder module, the dual-branch encoder module is used to gradually extract the features of the input remote sensing image, the bottleneck module uses the output of the dual-branch encoder as input, further extracts and fuses features of different scales, the jump connection module is used to adapt and refine the features of the feature maps of each stage of the dual-branch encoder module, and the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module;

[0009] After image enhancement processing, the remote sensing image to be detected is input into the remote sensing image shadow detection model for shadow detection processing, and the detection result is output.

[0010] Furthermore, performing image enhancement processing on each remote sensing image in the remote sensing image shadow detection dataset includes:

[0011] The remote sensing images in the remote sensing image shadow detection data set that are larger than a preset size are cropped, and the remote sensing images that are smaller than the preset size are upsampled.

[0012] Furthermore, the dual-branch encoder includes a CNN branch, a Transformer branch and a first splicing layer; wherein the CNN branch uses ResNet-50 as a backbone network, and the Transformer branch uses a SwinTransformer network;

[0013] Accordingly, the dual-branch encoder module is used to gradually extract features of the input remote sensing image, including:

[0014] Extracting features of an input remote sensing image using the CNN branch to obtain a first feature map;

[0015] Extracting features of the input remote sensing image using the Transformer branch to obtain a second feature map;

[0016] The first splicing layer is used to perform a splicing operation on the first feature map and the second feature map to obtain a third feature map.

[0017] Further, the bottleneck module is a multi-scale feature extraction module, including a first feature extraction branch, a second feature extraction branch, a third feature extraction branch, a fourth feature extraction branch and a second splicing layer;

[0018] Any feature extraction branch includes two convolution blocks, the convolution kernel size of the first convolution block of each feature extraction branch is 1×1, and the convolution kernel sizes of the second convolution block are 1×1, 3×3, 5×5 and 7×7 respectively; the second splicing layer is used to splice the outputs of the first feature extraction branch, the second feature extraction branch, the third feature extraction branch and the fourth feature extraction branch to generate a fourth feature map.

[0019] Furthermore, the jump connection module includes four boundary optimization modules, and the boundary optimization module includes a 3×3 convolution layer, a BN layer, a RELU layer, and a 1×1 convolution layer, and the input of the 3×3 convolution layer is connected to the output residual of the 1×1 convolution layer to output the output of the boundary optimization module;

[0020] Accordingly, the jump connection module is used to perform feature adaptation and refinement on the feature maps of each stage of the dual-branch encoder module, including:

[0021] Determine that the first convolutional layer in the CNN branch outputs a first intermediate feature map, and use a first boundary optimization module to process the first intermediate feature map to obtain a first feature refinement map;

[0022] Determine that the second convolutional layer in the CNN branch outputs a second intermediate feature map, and use a second boundary optimization module to process the second intermediate feature map to obtain a second feature refinement map;

[0023] Determine that the third convolutional layer in the CNN branch outputs a third intermediate feature map, and use a third boundary optimization module to process the third intermediate feature map to obtain a third feature refinement map;

[0024] Determine that the fourth convolutional layer in the CNN branch outputs a fourth intermediate feature map, and use a fourth boundary optimization module to process the fourth intermediate feature map to obtain a fourth feature refinement map.

[0025] Furthermore, the decoder includes three attention fusion modules and an upsampling module, wherein the attention fusion module includes an upsampling layer, a third concatenation layer, a 3×3 convolution layer, a BN layer, a RELU layer, a CA layer, a SA layer and a 1×1 convolution layer;

[0026] Accordingly, the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module, including:

[0027] The first attention fusion module is used to upsample the fourth feature refinement map, concatenate it with the fourth feature map, and extract channel and spatial information to obtain a fifth feature map;

[0028] The second attention fusion module is used to upsample the third feature refinement map, concatenate it with the fifth feature map, and extract channel and spatial information to obtain a sixth feature map;

[0029] The third attention fusion module is used to upsample the second feature refinement map, concatenate it with the sixth feature map, and extract channel and spatial information to obtain a seventh feature map;

[0030] The upsampling module is used to upsample the first feature refinement map, then splice it with the seventh feature map and output it.

[0031] In a second aspect, the present invention provides a remote sensing image shadow detection system based on a dual-branch encoder, comprising:

[0032] A data set construction module is used to obtain a remote sensing image shadow detection data set, perform image enhancement processing on each remote sensing image in the remote sensing image shadow detection data set, and divide the enhanced remote sensing image shadow detection data set into a training set and a test set;

[0033] A model training module, used to construct a neural network structure based on a dual-branch encoder, and use the training set and the test set to train the neural network structure based on the dual-branch encoder to obtain a remote sensing image shadow detection model; wherein the neural network structure based on the dual-branch encoder includes a dual-branch encoder module, a bottleneck module, a jump connection module and a decoder module, the dual-branch encoder module is used to gradually extract the features of the input remote sensing image, the bottleneck module uses the output of the dual-branch encoder as input, further extracts and fuses features of different scales, the jump connection module is used to adapt and refine the features of the feature maps of each stage of the dual-branch encoder module, and the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module;

[0034] The shadow detection execution module is used to input the remote sensing image to be detected into the remote sensing image shadow detection model after image enhancement processing to perform shadow detection processing and output the detection result.

[0035] In a third aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the method described above when executed by a processor.

[0037] Beneficial effects of the present invention:

[0038] The neural network model based on the dual-branch encoder constructed by the present invention effectively reuses shallow features through multi-scale feature extraction, fully utilizes the feature information of each scale, thereby improving the detection ability of the model and solving the problem of difficult feature extraction caused by small inter-class differences. In order to obtain richer spatial and channel information, the spatial attention module and the channel attention module are introduced into the decoder of the network to obtain features with spatial and channel attention information, which significantly improves the accuracy and robustness of shadow detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a flow chart of a remote sensing image shadow detection method based on a dual-branch encoder provided in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the structure of a neural network model based on a dual-branch encoder provided in an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the structure of a multi-scale feature extraction module provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the structure of a boundary optimization module provided in an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of the structure of an attention fusion module provided in an embodiment of the present invention;

[0044] Figure 6 A schematic diagram of the structure of a channel attention module provided in an embodiment of the present invention;

[0045] Figure 7 A schematic diagram of the structure of a spatial attention module provided by an embodiment of the present invention;

[0046] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] like Figure 1 As shown, an embodiment of the present invention provides a remote sensing image shadow detection method based on a dual-branch encoder, and the specific steps include:

[0049] S1: Obtain a remote sensing image shadow detection dataset, perform image enhancement processing on each remote sensing image in the remote sensing image shadow detection dataset, and divide the enhanced remote sensing image shadow detection dataset into a training set and a test set;

[0050] In this embodiment, the AISD remote sensing image dataset is selected as the data source for training and testing. The AISD dataset provides high-resolution aerial shadow images from different regions, which cover a rich variety of ground shadows, such as buildings, bridges, and trees. Such a dataset helps train the neural network model to process multi-scale shadow images, thereby improving the robustness and generalization of the model. By using high-resolution aerial images, the model can better learn and recognize shadows of different types and scales, thereby enhancing its performance in practical applications.

[0051] In this embodiment, the enhanced data set is divided into a training set and a test set, wherein the ratio of the training set to the test set is 7: 3. It should be noted that those skilled in the art can adjust the ratio of the training set to the test set according to specific actual conditions to meet different needs and application scenarios.

[0052] S2: Construct a neural network structure based on a dual-branch encoder, and use the training set and the test set to train the neural network structure based on the dual-branch encoder to obtain a remote sensing image shadow detection model;

[0053] like Figure 2 As shown, a schematic diagram of a neural network structure based on a dual-branch encoder provided in this embodiment, wherein the neural network structure based on the dual-branch encoder includes a dual-branch encoder module, a bottleneck module, a jump connection module and a decoder module. Specifically, the dual-branch encoder module is used to gradually extract the features of the input remote sensing image, the bottleneck module takes the output of the dual-branch encoder as input, further extracts and fuses features of different scales, the jump connection module is used to adapt and refine the features of the feature maps of each stage of the dual-branch encoder module, and the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module.

[0054] In this embodiment, the training set and the test set can be used to iteratively train and optimize the neural network structure based on the dual-branch encoder, and the optimized neural network structure based on the dual-branch encoder is a remote sensing image shadow detection model. The model can detect the position of the shadow in the remote sensing image.

[0055] S3: After image enhancement processing, the remote sensing image to be detected is input into the remote sensing image shadow detection model for shadow detection processing, and the detection result is output.

[0056] Specifically, the remote sensing image to be detected is input into the remote sensing image shadow detection model obtained in step S2 after image enhancement processing, so that the shadow area in the remote sensing image to be detected can be accurately identified and marked, so as to provide valuable information for subsequent image analysis and processing.

[0057] The solution provided in this embodiment builds a neural network structure based on a dual-branch encoder, wherein the dual-branch encoder can extract features from the input image, effectively reuse shallow features through multi-scale feature extraction, and make full use of feature information at each scale, thereby improving the detection capability of the model and solving the problem of difficult feature extraction caused by small differences between classes.

[0058] As a further introduction to the above-mentioned embodiment, image enhancement can be performed in the following manner: taking any remote sensing image in the remote sensing image shadow detection dataset as an example, image enhancement processing is performed, and the specific implementation steps include: determining the image size of the remote sensing image, cropping or upsampling the image to ensure that all remote sensing images are adjusted to a uniform size. In this embodiment, the AISD remote sensing image dataset is selected as the remote sensing image shadow detection dataset. The image size range in this dataset is 256×256 to 1688×1688. Therefore, images larger than 512×512 are first cropped and resized to a fixed size of 512×512; for images smaller than 512×512, upsampling is performed to ensure that all images are adjusted to a uniform size of 512×512.

[0059] As an implementable method, based on the above embodiment, the neural network structure based on the dual-branch encoder includes: a dual-branch encoder module, a bottleneck module, a skip connection module and a decoder module. The steps of constructing the neural network structure based on the dual-branch encoder are introduced below.

[0060] Step one, dual branch encoder module.

[0061] The dual-branch encoder consists of a CNN branch, a Transformer branch, and the first concatenation layer.

[0062] Specifically, based on the previous single CNN branch, the Transformer branch is introduced to form a dual-branch encoder module. The CNN branch uses ResNet-50 as the backbone network to extract context features; the Transformer branch uses Swin Transformer for feature extraction.

[0063] It can be understood that this dual-branch structure leverages the powerful ability of Transformer in mining global context features to effectively identify the subtle differences between dark black objects and shadows, thereby accurately identifying non-shadow pixels.

[0064] Accordingly, the dual-branch encoder module is used to gradually extract the features of the input remote sensing image, including:

[0065] The CNN branch is used to extract the features of the input remote sensing image to obtain the first feature map;

[0066] The Transformer branch is used to extract the features of the input remote sensing image to obtain the second feature map;

[0067] The first feature map and the second feature map are concatenated using the first concatenation layer to obtain a third feature map.

[0068] The second step is the bottleneck module.

[0069] The bottleneck module includes a multi-scale feature extraction module, such as Figure 3 As shown, the module includes a first feature extraction branch, a second feature extraction branch, a third feature extraction branch, a fourth feature extraction branch and a second splicing layer. Any feature extraction branch includes two convolution blocks, the convolution kernel size of the first convolution block of each feature extraction branch is 1×1, and the convolution kernel sizes of the second convolution block are 1×1, 3×3, 5×5 and 7×7 respectively; the second splicing layer is used to splice the outputs of the first feature extraction branch, the second feature extraction branch, the third feature extraction branch and the fourth feature extraction branch to generate a fourth feature map.

[0070] It is understandable that large convolution kernels can capture long-range context information, but the computational cost is high. Therefore, before applying 3×3, 5×5, and 7×7 convolutions, 1×1 convolutions are first used to reduce the dimension of the feature to reduce the computational complexity. Subsequently, the output feature maps of these parallel convolution blocks are concatenated to obtain feature maps that fuse multi-scale information.

[0071] The third step is the skip connection module.

[0072] The skip connection module includes four boundary optimization modules, such as Figure 4As shown in FIG. 1 , the boundary optimization module includes a 3×3 convolutional layer, a BN layer, a RELU layer, and a 1×1 convolutional layer. The input of the 3×3 convolutional layer and the output of the 1×1 convolutional layer are connected to the residual output as the output of the boundary optimization module.

[0073] Specifically, let T1 represent the input, and the refined feature map T2 after the boundary optimization module can be expressed as T2=T1+R(T1), where R(T1) is the residual branch. The expression of R(T1) is:

[0074]

[0075] Among them, F out represents the output of the residual optimization module, F in Represents the input of the residual optimization module, c1 represents 3×3 convolution, BN represents batch normalization, ReLU is the activation function, and c2 represents 1×1 convolution.

[0076] Accordingly, the skip connection module is used to adapt and refine the feature maps of each stage of the dual-branch encoder module, including:

[0077] Determine that the first convolutional layer in the CNN branch outputs a first intermediate feature map, and use a first boundary optimization module to process the first intermediate feature map to obtain a first feature refinement map;

[0078] Determine that the second convolutional layer in the CNN branch outputs a second intermediate feature map, and use a second boundary optimization module to process the second intermediate feature map to obtain a second feature refinement map;

[0079] Determine that the third convolutional layer in the CNN branch outputs a third intermediate feature map, and use a third boundary optimization module to process the third intermediate feature map to obtain a third feature refinement map;

[0080] Determine that the fourth convolutional layer in the CNN branch outputs a fourth intermediate feature map, and use a fourth boundary optimization module to process the fourth intermediate feature map to obtain a fourth feature refinement map.

[0081] Step 4: Decoder.

[0082] The decoder consists of three attention fusion modules and upsampling modules, such as Figure 5 As shown, the attention fusion module includes an upsampling layer, a third concatenation layer, a 3×3 convolution layer, a BN layer, a RELU layer, a CA (spatial attention) layer, a SA (channel attention) layer, and a 1×1 convolution layer.

[0083] Specifically, the low-resolution feature map is upsampled to increase its size to match the corresponding high-resolution feature map. Then, the upsampled low-resolution feature map is concatenated with the high-resolution feature map of the jump connection in the corresponding part. After 3×3 convolution and batch normalization (BN) operations, the concatenated feature map is input into the channel attention module and the spatial attention module in turn to extract channel and spatial information. Finally, the number of channels of the feature map is adjusted through the convolution operation to ensure the accurate representation of the feature map and adapt to subsequent processing.

[0084] Accordingly, the decoder module is used to match and concatenate the outputs of the skip connection module and the bottleneck module, including:

[0085] The first attention fusion module is used to upsample the fourth feature refinement map, concatenate it with the fourth feature map, and extract channel and spatial information to obtain the fifth feature map;

[0086] The second attention fusion module is used to upsample the third feature refinement map, concatenate it with the fifth feature map, and extract channel and spatial information to obtain the sixth feature map;

[0087] The third attention fusion module is used to upsample the second feature refinement map and then concatenate it with the sixth feature map and extract channel and spatial information to obtain the seventh feature map;

[0088] The upsampling module is used to upsample the first feature refinement map, concatenate it with the seventh feature map, and output it.

[0089] Specifically, the structure of the CA (channel attention module) layer is as follows Figure 6 As shown, the overall channel attention process can be expressed as:

[0090]

[0091] Among them, F represents the input feature map, F∈R C×H×W , R represents a real number, C represents the number of channels, H represents the height of the feature map, W represents the width of the feature map, and M c (F) represents the channel attention weight, and F′ represents the output feature map;

[0092]

[0093] Among them, sigmoid represents the activation function, MLP represents the multi-layer perceptron, AvgPool(F) represents the result of average pooling, and MaxPool(F) represents the result of maximum pooling. It also represents the result of average pooling, C represents the channel attention operation, It also represents the result of the maximum pooling, W0 and W1 represent the weights of the multi-layer perceptron, W0∈RC / r×C , W1∈R C×C / r , where r is the reduction ratio. It should be noted that the channel attention operation C specifically refers to Channel Attention, which has no practical meaning and can be removed from the formula.

[0094] Specifically, the structure of the SA (spatial attention module) layer is as follows Figure 7 As shown, the overall spatial attention process can be expressed as:

[0095]

[0096] Among them, F represents the input feature map, F∈R C×H×W , R represents a real number, C represents the number of channels, H represents the height of the feature map, W represents the width of the feature map, and M s (F) represents the spatial attention weight, and F′ represents the output feature map.

[0097]

[0098] Among them, sigmoid represents the activation function, f 7×7 represents a convolution operation with a filter size of 7×7, AvgPool(F) represents the result of average pooling, and MaxPool(F) represents the result of maximum pooling. It also represents the result of average pooling, S represents the spatial attention operation, It also represents the result of maximum pooling. It means the average pooling result and the maximum pooling result are superimposed. It should be noted that the spatial attention operation S specifically refers to Spatial Attention, which has no practical meaning and can be removed from the formula.

[0099] As an implementable method, a remote sensing image shadow detection model is obtained by training a neural network structure based on a dual-branch encoder using a training set and a test set, including:

[0100] Determine the hyperparameters of the neural network architecture based on the dual-branch encoder.

[0101] Before starting training, hyperparameters such as the number of training times, learning rate, batch size, and loss function are set. In this embodiment, the number of training times is set to 150 times and the batch size is 6. The stochastic gradient descent (SGD) method with momentum is used in the training process, where the momentum is set to 0.9 and the weight decay is set to 0.0001 to accelerate model convergence and prevent overfitting. The initial learning rate is set to 0.001, and the learning rate adjustment strategy is adopted at equal intervals, reducing the learning rate by 0.1 times every 50 trainings to ensure that the model can finely adjust the weights during the training process and gradually stabilize. The loss function uses cross entropy loss, which is a loss function widely used in deep learning, especially suitable for tasks such as shadow detection.

[0102] Specifically, the cross entropy loss provides a clear direction for the model to continuously optimize the parameters by comparing the difference between the model's predicted probability distribution and the actual label, so that the prediction results are closer to the actual situation. Its calculation formula is:

[0103] L = -(y log(p) + (1-y) log(1-p))

[0104] Among them, Loss represents the loss function value, y represents the label of the sample, the positive class is assigned a value of 1, the negative class is assigned a value of 0, and p is the probability of predicting that the sample is positive.

[0105] It should be noted that after each training, the model parameters of this training are saved, which helps to track the progress of the model during the training process and go back to the previous state when necessary. Based on the above-mentioned neural network structure based on the dual-branch encoder, its specific training process will not be elaborated here.

[0106] The entire training process will continue until the number of training times of the neural network model reaches the set training cycle, and then the training will be stopped. The model training process in this embodiment ensures that the model can reach the best performance level after sufficient training, providing a reliable foundation for the application of subsequent tasks. This training strategy helps the model to gradually optimize its parameters during training to achieve higher accuracy and generalization ability, thereby showing excellent performance in practical applications.

[0107] Use the images in the test set to test the performance of the trained model.

[0108] In this embodiment, in order to comprehensively evaluate the performance of the model, the F1 score, overall pixel accuracy and intersection-over-union ratio are comprehensively used as evaluation indicators in the performance test. These comprehensive evaluation indicators help to fully understand the performance of the trained neural network model on the actual test data. The F1 score provides a comprehensive evaluation of the model's classification ability by balancing precision and recall; the overall pixel accuracy measures the overall accuracy of the model in the image segmentation task; and the intersection-over-union ratio is used to evaluate the degree of overlap between the predicted area and the actual area. Through the combination of these three indicators, we can have a deeper understanding of the advantages and limitations of the model, thereby providing more targeted guidance for further improving the model. This multi-angle evaluation method ensures a comprehensive grasp of the model's performance and provides sufficient data support for model optimization.

[0109] As an indicator for comprehensively evaluating the accuracy and recall of the model, the F1 score quantifies the performance of the model by calculating the harmonic mean of the accuracy and recall. This indicator not only takes into account the accuracy of the model, but also comprehensively evaluates the model's ability to capture positive samples, thus providing a comprehensive measure of the overall performance of the model.

[0110]

[0111] Among them, F1 represents the F1 score, Precision represents accuracy, Recall represents recall, TP represents the number of true positives, FP represents the number of false positives, and FN represents the number of false negatives.

[0112] Overall pixel accuracy is a simple and intuitive evaluation metric that measures the proportion of correctly classified models on the entire test dataset. This metric provides a quick overview of overall classification accuracy by calculating the proportion of correctly classified pixels predicted by the model. Overall pixel accuracy reflects the performance of the model when processing the entire dataset and is an important metric for evaluating model performance. It not only helps us understand the overall accuracy of the model, but also serves as a benchmark for comparing the performance of different models, allowing us to more effectively judge and optimize the performance of the model.

[0113]

[0114] Among them, OA represents the overall pixel accuracy and TN represents the number of true negatives.

[0115] Intersection over Union (IoU) is an evaluation metric that comprehensively considers the model's accuracy in predicting target boundaries. It measures the model's fit to the target area by calculating the intersection area of ​​the predicted area and the actual target area, and then calculating the ratio of this intersection area to their union area. Specifically, IoU is the area of ​​the intersection of the predicted area and the actual target area divided by the area of ​​their union, providing detailed insights into the model's performance in identifying and locating targets. This metric is particularly suitable for image segmentation tasks because it can accurately reflect the model's prediction accuracy for the target area boundary, helping us evaluate the model's performance in detail processing and further optimizing the model's recognition and positioning capabilities.

[0116]

[0117] Among them, IoU represents the intersection over union ratio, P m represents the predicted pixel set, P gt Represents the true value pixel set, N represents the number of pixels in the intersection or union, ∩ represents the intersection, and ∪ represents the union.

[0118] In this embodiment, by analyzing the three evaluation indicators of F1, overall pixel accuracy (OA) and intersection over union (IoU), the hyperparameters and architecture of the model can be iteratively adjusted to optimize the model performance. At the same time, attention should be paid to the overfitting problem to ensure the generalization ability of the model on the test set. Finally, through repeated experiments and tuning, a set of hyperparameter combinations that achieve the best balance among F1, OA and IoU indicators is found, and the neural network model composed of the model obtained at this time is used as the final model after optimization.

[0119] The above process can be expressed as:

[0120] Preset the number of training times and loss function, and select the target remote sensing image from the training set;

[0121] Input the target remote sensing image into the neural network structure based on the dual-branch encoder and output the detection result;

[0122] The loss function is used to calculate the loss value of the detection result and the actual label of the target remote sensing image, and the parameters of the neural network structure based on the dual-branch encoder are adjusted according to the loss value;

[0123] Determine whether the number of training times reaches the preset number of training times; if so, test the trained neural network structure based on the dual-branch encoder; if not, enter the step of selecting the target remote sensing image from the training set;

[0124] The trained neural network structure based on the dual-branch encoder is tested using the test set. The FI score, overall pixel accuracy and intersection-over-union ratio are calculated as evaluation indicators to optimize the neural network structure of the dual-branch encoder. The optimized neural network structure of the dual-branch encoder is used as a shadow detection model for remote sensing images.

[0125] The present embodiment provides a remote sensing image shadow detection method based on a dual-branch encoder. First, a remote sensing image data set is obtained and image enhancement processing is performed on it, and the data set is divided into a training set and a test set. Next, a remote sensing image shadow detection neural network model based on a dual-branch encoder is constructed and trained to obtain a trained model. Subsequently, a performance test is performed on the trained model, and various parameters are optimized according to the test results to obtain an optimized neural network model. Finally, the optimized model is used to perform shadow detection on the remote sensing image to be detected, thereby obtaining a shadow detection result. The present invention improves the accuracy and robustness of the detection results by comprehensively analyzing the shadows of objects in remote sensing images, and is suitable for various complex scenes.

[0126] The embodiment of the present invention further provides a remote sensing image shadow detection system based on a dual-branch encoder, characterized in that it includes:

[0127] A data set construction module is used to obtain a remote sensing image shadow detection data set, perform image enhancement processing on each remote sensing image in the remote sensing image shadow detection data set, and divide the enhanced remote sensing image shadow detection data set into a training set and a test set;

[0128] A model training module is used to construct a neural network structure based on a dual-branch encoder, and use a training set and a test set to train the neural network structure based on the dual-branch encoder to obtain a remote sensing image shadow detection model; wherein the neural network structure based on the dual-branch encoder includes a dual-branch encoder module, a bottleneck module, a jump connection module and a decoder module, the dual-branch encoder module is used to gradually extract the features of the input remote sensing image, the bottleneck module uses the output of the dual-branch encoder as input, and further extracts and fuses features of different scales, the jump connection module is used to adapt and refine the features of the feature maps of each stage of the dual-branch encoder module, and the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module;

[0129] The shadow detection execution module is used to input the remote sensing image to be detected into the remote sensing image shadow detection model after image enhancement processing to perform shadow detection processing and output the detection result.

[0130] like Figure 8As shown, an embodiment of the present invention further provides an electronic device, which may include: a processor (processor) 801, a communication interface (Communications Interface) 802, a memory (memory) 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804. The processor 801 can call the logic instructions in the memory 803 to execute the method provided in the above embodiment, for example, including:

[0131] Obtain a remote sensing image shadow detection dataset, perform image enhancement processing on each remote sensing image in the remote sensing image shadow detection dataset, and divide the enhanced remote sensing image shadow detection dataset into a training set and a test set;

[0132] A neural network structure based on a dual-branch encoder is constructed, and the training set and the test set are used to train the neural network structure based on the dual-branch encoder to obtain a remote sensing image shadow detection model; wherein the neural network structure based on the dual-branch encoder includes a dual-branch encoder module, a bottleneck module, a jump connection module and a decoder module, the dual-branch encoder module is used to gradually extract the features of the input remote sensing image, the bottleneck module takes the output of the dual-branch encoder as input, further extracts and fuses features of different scales, the jump connection module is used to adapt and refine the features of the feature maps of each stage of the dual-branch encoder module, and the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module;

[0133] After image enhancement, the remote sensing image to be detected is input into the remote sensing image shadow detection model for shadow detection processing, and the detection result is output.

[0134] In addition, when the logic instructions in the above-mentioned memory 803 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0135] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the above embodiment is implemented, for example, including:

[0136] Obtain a remote sensing image shadow detection dataset, perform image enhancement processing on each remote sensing image in the remote sensing image shadow detection dataset, and divide the enhanced remote sensing image shadow detection dataset into a training set and a test set;

[0137] A neural network structure based on a dual-branch encoder is constructed, and the training set and the test set are used to train the neural network structure based on the dual-branch encoder to obtain a remote sensing image shadow detection model; wherein the neural network structure based on the dual-branch encoder includes a dual-branch encoder module, a bottleneck module, a jump connection module and a decoder module, the dual-branch encoder module is used to gradually extract the features of the input remote sensing image, the bottleneck module takes the output of the dual-branch encoder as input, further extracts and fuses features of different scales, the jump connection module is used to adapt and refine the features of the feature maps of each stage of the dual-branch encoder module, and the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module;

[0138] After image enhancement, the remote sensing image to be detected is input into the remote sensing image shadow detection model for shadow detection processing, and the detection result is output.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image shadow detection method based on a dual-branch encoder, characterized in that: include: Acquire a remote sensing image shadow detection dataset, perform image enhancement processing on each remote sensing image in the remote sensing image shadow detection dataset, and divide the enhanced remote sensing image shadow detection dataset into a training set and a test set; Constructing a neural network structure based on a dual-branch encoder, and using the training set and the test set to train the neural network structure based on the dual-branch encoder to obtain a remote sensing image shadow detection model; wherein the neural network structure based on the dual-branch encoder includes a dual-branch encoder module, a bottleneck module, a jump connection module and a decoder module, the dual-branch encoder module is used to gradually extract the features of the input remote sensing image, the bottleneck module uses the output of the dual-branch encoder as input, further extracts and fuses features of different scales, the jump connection module is used to adapt and refine the features of the feature maps of each stage of the dual-branch encoder module, and the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module; After image enhancement processing, the remote sensing image to be detected is input into the remote sensing image shadow detection model for shadow detection processing, and the detection result is output.

2. A remote sensing image shadow detection method based on a dual-branch encoder according to claim 1, characterized in that: Performing image enhancement processing on each remote sensing image in the remote sensing image shadow detection dataset includes: The remote sensing images in the remote sensing image shadow detection data set that are larger than a preset size are cropped, and the remote sensing images that are smaller than the preset size are upsampled.

3. The remote sensing image shadow detection method based on a dual-branch encoder according to claim 1, characterized in that: The dual-branch encoder includes a CNN branch, a Transformer branch and a first splicing layer; wherein the CNN branch uses ResNet-50 as a backbone network, and the Transformer branch uses a Swin Transformer network; Accordingly, the dual-branch encoder module is used to gradually extract features of the input remote sensing image, including: Extracting features of an input remote sensing image using the CNN branch to obtain a first feature map; Extracting features of the input remote sensing image using the Transformer branch to obtain a second feature map; The first splicing layer is used to perform a splicing operation on the first feature map and the second feature map to obtain a third feature map.

4. The remote sensing image shadow detection method based on a dual-branch encoder according to claim 1 is characterized in that: The bottleneck module is a multi-scale feature extraction module, including a first feature extraction branch, a second feature extraction branch, a third feature extraction branch, a fourth feature extraction branch, and a second splicing layer; Any feature extraction branch includes two convolution blocks, the convolution kernel size of the first convolution block of each feature extraction branch is 1×1, and the convolution kernel sizes of the second convolution block are 1×1, 3×3, 5×5 and 7×7 respectively; the second splicing layer is used to splice the outputs of the first feature extraction branch, the second feature extraction branch, the third feature extraction branch and the fourth feature extraction branch to generate a fourth feature map.

5. The remote sensing image shadow detection method based on a dual-branch encoder according to claim 1, characterized in that: The jump connection module includes four boundary optimization modules, each of which includes a 3×3 convolutional layer, a BN layer, a RELU layer, and a 1×1 convolutional layer, and the input of the 3×3 convolutional layer is connected to the output residual of the 1×1 convolutional layer to output the output of the boundary optimization module; Accordingly, the jump connection module is used to perform feature adaptation and refinement on the feature maps of each stage of the dual-branch encoder module, including: Determine that the first convolutional layer in the CNN branch outputs a first intermediate feature map, and use a first boundary optimization module to process the first intermediate feature map to obtain a first feature refinement map; Determine that the second convolutional layer in the CNN branch outputs a second intermediate feature map, and use a second boundary optimization module to process the second intermediate feature map to obtain a second feature refinement map; Determine that the third convolutional layer in the CNN branch outputs a third intermediate feature map, and use a third boundary optimization module to process the third intermediate feature map to obtain a third feature refinement map; Determine that the fourth convolutional layer in the CNN branch outputs a fourth intermediate feature map, and use a fourth boundary optimization module to process the fourth intermediate feature map to obtain a fourth feature refinement map.

6. The remote sensing image shadow detection method based on a dual-branch encoder according to claim 1, characterized in that: The decoder includes three attention fusion modules and an upsampling module, wherein the attention fusion module includes an upsampling layer, a third concatenation layer, a 3×3 convolution layer, a BN layer, a RELU layer, a CA layer, a SA layer, and a 1×1 convolution layer; Accordingly, the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module, including: The first attention fusion module is used to upsample the fourth feature refinement map, concatenate it with the fourth feature map, and extract channel and spatial information to obtain a fifth feature map; The second attention fusion module is used to upsample the third feature refinement map, concatenate it with the fifth feature map, and extract channel and spatial information to obtain a sixth feature map; The third attention fusion module is used to upsample the second feature refinement map, concatenate it with the sixth feature map, and extract channel and spatial information to obtain a seventh feature map; The upsampling module is used to upsample the first feature refinement map, then splice it with the seventh feature map and output it.

7. A remote sensing image shadow detection system based on a dual-branch encoder, characterized in that: include: A data set construction module is used to obtain a remote sensing image shadow detection data set, perform image enhancement processing on each remote sensing image in the remote sensing image shadow detection data set, and divide the enhanced remote sensing image shadow detection data set into a training set and a test set; A model training module, used to construct a neural network structure based on a dual-branch encoder, and use the training set and the test set to train the neural network structure based on the dual-branch encoder to obtain a remote sensing image shadow detection model; wherein the neural network structure based on the dual-branch encoder includes a dual-branch encoder module, a bottleneck module, a jump connection module and a decoder module, the dual-branch encoder module is used to gradually extract the features of the input remote sensing image, the bottleneck module uses the output of the dual-branch encoder as input, further extracts and fuses features of different scales, the jump connection module is used to adapt and refine the features of the feature maps of each stage of the dual-branch encoder module, and the decoder module is used to match and splice the outputs of the jump connection module and the bottleneck module; The shadow detection execution module is used to input the remote sensing image to be detected into the remote sensing image shadow detection model after image enhancement processing to perform shadow detection processing and output the detection result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is performed.

Images