A floating object detection method and device based on remote sensing images, a storage medium and an apparatus

Abstract

The application discloses a floating object detection method and device based on a remote sensing image, a storage medium and equipment, and belongs to the technical field of image processing, and comprises the following steps: acquiring a remote sensing image; preprocessing the remote sensing image; inputting the preprocessed remote sensing image into a trained floating object detection model to obtain a label image containing a detection result; the feature shunting module is used for classifying strong and weak feature layers, and the strong feature layer is reserved, so that the network load is reduced, and the model segmentation precision is improved; the multiscale feature extraction module is used for obtaining a variable scale feature extraction window, so that the demand of feature extraction of floating objects with different sizes is met, and the acquisition capability of the model for context feature information is improved; the cascaded feedback attention module is used for fusing an input image after the output of each attention mechanism to prevent excessive attention or attention deviation; and the floating object detection precision is improved.

Description

Technical Field

[0001] This invention relates to a method, apparatus, storage medium, and device for detecting floating objects based on remote sensing images, belonging to the field of image processing technology. Background Technology

[0002] By the end of 2022, my country's total railway operating mileage reached 155,000 kilometers, ranking second in the world. However, with the large-scale construction of railways, railway safety has become a major concern for railway departments, and foreign object intrusion is one of the factors threatening railway operational safety. On April 14, 2019, due to strong winds, a piece of plastic sheeting got caught on the overhead contact line of the Beijing-Shanghai High-Speed ​​Railway, causing six trains to be delayed by nearly two hours. In 2021, on the Dingzhou East to Baoding East section of the Beijing-Guangzhou High-Speed ​​Railway, agricultural plastic film was blown up by strong winds and got caught on the overhead contact line, causing all 16 trains departing from Beijing West Station on May 1 to be cancelled. Accidents can range from minor train stoppages and slowdowns to serious threats to passenger safety. In the face of frequent intrusion problems, timely detection of floating foreign objects and prevention of their intrusion into the tracks are necessary prerequisites for ensuring the safe operation of trains.

[0003] Existing railway safety operation plans mainly revolve around "prevention-avoidance-rescue." "Rescue" refers to the timely handling of encroachment incidents by maintenance personnel to restore operation in the shortest possible time. Existing devices such as fences, radar, and cameras can address "rescue," but the time required from detecting and resolving an encroachment incident already disrupts normal public and societal operations due to train delays and cancellations. "Avoidance" refers to avoiding areas prone to strong winds and areas with dense floating debris; however, rerouting railway lines incurs significant costs, a problem the railway department urgently needs to address. The "prevention" aspect lacks a complete and mature system within the plan; how to improve a safety operation plan centered on "prevention" is a crucial issue that the railway department needs to resolve. A review of relevant materials revealed no such information.

[0004] Starting from the perspective of "prevention," this approach aims to detect and prevent accidents by proactively sensing floating objects (with greenhouses and dust nets as typical examples) around railway lines and establishing an intrusion risk level mechanism that considers wind speed and direction. Therefore, accurately and precisely segmenting greenhouses and dust nets along railway lines is a necessary prerequisite for establishing a risk level mechanism. In recent years, the rapid development of deep learning in computer vision applications has made semantic segmentation a mainstream method in image processing. Semantic segmentation is a technique that segments different objects in an image from a pixel perspective and labels each pixel in the original image. By using semantic segmentation technology to process remote sensing images along railway lines, it is possible to achieve precise segmentation of greenhouses and dust nets.

[0005] Existing semantic segmentation techniques still have the following drawbacks:

[0006] 1. When using a backbone network to perform deep feature extraction on remote sensing images with complex backgrounds, multiple feature layers with mixed strengths and weaknesses are obtained. The number of feature layers increases the burden on the network operation. More importantly, weak feature layers can lead to blurred boundaries, incorrect or missed segmentation of targets, and inconsistencies may occur when weak feature layers are fused with higher-level features, affecting the consistency and accuracy of the segmentation results.

[0007] 2. The greenhouses and dust nets along the railway line are spatially dispersed and have varying coverage areas, making it difficult to achieve good segmentation results using only feature extraction windows of the same scale.

[0008] 3. The presence of residential areas, bushes, highways, rivers and lakes along the railway line, which are irrelevant to the segmentation task, can cause segmentation bias in the model. Using only a single attention mechanism may also overemphasize some unimportant targets, ultimately resulting in low segmentation accuracy.

[0009] In conclusion, the accuracy of floating object detection in existing technologies still needs to be improved. Summary of the Invention

[0010] The purpose of this invention is to provide a method, apparatus, storage medium, and device for detecting floating objects based on remote sensing images, thereby solving the problem of low accuracy in the prior art.

[0011] To achieve the above objectives, the present invention employs the following technical solution:

[0012] In a first aspect, the present invention provides a method for detecting floating objects based on remote sensing images, comprising:

[0013] Acquire remote sensing images;

[0014] The remote sensing image is preprocessed;

[0015] The preprocessed remote sensing image is input into the trained floating object detection model to obtain a labeled image containing the detection results.

[0016] In conjunction with the first aspect, the preprocessing further includes cropping the remote sensing image to a size of 256 x 256 pixels.

[0017] In conjunction with the first aspect, the floating object detection model further includes an improved ResNet-50, a feature splitting module, a multi-scale feature extraction module, and a cascaded feedback attention module;

[0018] The improved ResNet-50 is the backbone network of the floating object detection model, used to initially extract feature information from remote sensing images and output the first feature map;

[0019] The feature splitting module performs strong and weak feature filtering on the first feature map, retains the strong feature layer in the first feature map, and outputs the second feature map.

[0020] The multi-scale feature extraction module extracts a multi-scale feature map from the second feature map;

[0021] The cascaded feedback attention module calculates the weights of the multi-scale feature map in the spatial and channel dimensions, respectively, and then fuses the obtained weights with the multi-scale feature map to obtain the label image.

[0022] In conjunction with the first aspect, the improvements of the improved ResNet-50 compared to ResNet-50 include: changing the downsampling of the fourth feature layer to 8 times downsampling, while increasing the number of channels to 1024 layers.

[0023] In conjunction with the first aspect, the feature splitting module further performs strong and weak feature filtering on the first feature map, including:

[0024] The first and second feature layers of the first feature map are stacked, and the third and fourth feature layers of the first feature map are stacked to obtain two first intermediate feature maps.

[0025] The weights of the two first intermediate feature maps are calculated layer by layer using global average pooling;

[0026] Feature layers with weights greater than 0 are strong feature layers, and feature layers with weights less than or equal to 0 are weak feature layers. By removing the weak feature layers from the two first intermediate feature maps, two first intermediate strong feature maps are obtained.

[0027] The two first intermediate strong feature maps are convolved separately, and then the features are stacked to obtain the second feature map.

[0028] In conjunction with the first aspect, the multi-scale feature extraction module further includes a multi-scale dilated convolution module, an average pooling layer, a max pooling layer, and a sigmoid function connected in sequence.

[0029] The multi-scale dilated convolution module includes three dilated convolutions with different dilation coefficients connected in a cascaded manner. The expression for the multi-scale dilated convolution module is as follows:

[0030] k′=k+(k-1)×(r-1)

[0031] RF = (2 r+1 -1)×(2 r+1 -1)

[0032] f sum1 =up[Conv r＝2 [Conv r＝2 (f in1 )]]+Conv r＝3 (f in1 )

[0033] f sum2 =up[Conv r＝2 [Conv r＝2 (f sum1 )]]+up[Conv r＝3 (f sum1 )]+Conv r＝4 (f in1 )

[0034] Where k′ is the size of the new nucleus after expansion, k is the original nucleus size, r is the expansion coefficient, RF is the receptive field, and f sum1 This is the first branch feature map, up[] is the upsampled feature map size adjustment, Conv r＝i This represents the dilated convolution with dilation coefficients r = i, where i = 2, 3, 4, f in1 It is the input to the multi-scale dilated convolution block, f sum2 It is the output of the multi-scale dilated convolution module;

[0035] The expression for the average pooling layer is:

[0036] f a =Conv 1×1 {Avgpool 1×1 (input3)+Avgpool 3×3 (input3)+Avgpool 9×9 (input3)}

[0037] f m =Conv 1×1 {MSDConv(input1)+MSDConv(input2)}

[0038] Among them, f a This indicates that input3 is processed through three average pooling layers of different sizes, followed by feature stacking, and then the output f. m This represents the output of input1 and input2 after multi-scale dilated convolution and feature stacking. Input1 is the input from the first layer of the backbone network, input2 is the input from the second layer of the backbone network, and input3 is the input from the fourth layer of the backbone network. Conv 1×1 Avgpool represents a 1×1 convolution.1×1 Avgpool represents 1×1 average pooling. 3×3 Avgpool represents 3×3 average pooling. 9×9 represents 9×9 average pooling, and MSDConv represents multi-scale dilated convolution;

[0039] The expressions for the max pooling layer and the Sigmoid function are as follows:

[0040] f w =Sigmoid{Conv 1×1 (Maxpool(input3))}

[0041] Among them, f w It is the probability output obtained after processing by the max pooling layer and the sigmoid function. Maxpool means max pooling.

[0042] f out1 =f w ×(f m +f a )

[0043] Among them, f out1 This is the output of the multi-scale feature extraction module.

[0044] In conjunction with the first aspect, the expression for the cascaded feedback attention module is further as follows:

[0045] W c =Sigmoid(MLP(AvgPool(f in2 ))+MLP(MaxPool(f in2 )))

[0046] f c =f in2 ×W c

[0047] W s =Sigmoid(Conv 7×7 ([AvgPool(f c ), MaxPool(f c )]))

[0048] f s =f in2 ×W s

[0049] f out2 =Conv 1×1 (f s +f in2 )

[0050] Among them, W c The feature vector output by the spatial attention mechanism is represented by 'Sigmoid', 'MLP' represents the multilayer perceptron, 'AvgPool' represents average pooling, and 'MaxPool' represents max pooling. in2 f represents the input to the cascaded feedback attention module. c f in2 W is a weighted output of the feature vectors output by the spatial attention mechanism. s Conv represents the feature vector output by the spatial attention mechanism. 7×7 This represents a convolution with a kernel of 7, f s f in2 The weighted output of f, which is the feature vector output by the channel attention mechanism. in2 f represents the input to the cascaded feedback attention module. out2 This represents the output of the cascaded feedback attention module.

[0051] Secondly, the present invention also provides a floating object detection device based on remote sensing images, comprising:

[0052] The image acquisition module is configured to acquire remote sensing images;

[0053] The image preprocessing module is configured to preprocess the remote sensing image;

[0054] The floating object detection module is configured to input preprocessed remote sensing images into a trained floating object detection model to obtain labeled images containing the detection results.

[0055] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the floating object detection method based on remote sensing images as described in any of the first aspects.

[0056] Fourthly, the present invention also provides an apparatus comprising:

[0057] Memory, used to store instructions;

[0058] A processor is configured to execute the instructions, causing the device to perform operations implementing the floating object detection method based on remote sensing images as described in any of the first aspects.

[0059] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0060] This invention provides a method, apparatus, storage medium, and device for detecting floating objects based on remote sensing images. It improves the detection accuracy by using an improved floating object detection model to detect objects in remote sensing images.

[0061] An innovative feature splitting module was designed to visualize the feature layers, calculate the feature information weights contained in each feature layer, classify strong and weak feature layers, retain strong feature layers and perform further feature extraction, thereby reducing network load while improving model segmentation accuracy.

[0062] An innovative multi-scale feature extraction module was designed, which sets different dilation coefficients for dilated convolution and then performs multi-branch combination to obtain a variable-scale feature extraction window, which meets the feature extraction needs of greenhouses and dust nets of different sizes, thereby improving the model's ability to acquire contextual feature information;

[0063] By combining existing attention mechanisms, a cascaded feedback attention module is designed. Based on using a spatial attention mechanism to obtain the spatially interested target and a channel attention mechanism to obtain the channels related to the target of interest, the input image is fused after the output of each attention mechanism to prevent over-attention or attention bias.

[0064] By combining the above three points, the accuracy of detecting floating objects can be improved. Attached Figure Description

[0065] Figure 1 This is a flowchart of a floating object detection method based on remote sensing images provided in an embodiment of the present invention;

[0066] Figure 2 This is a schematic diagram of the framework of the floating object detection model provided in an embodiment of the present invention;

[0067] Figure 3 This is a visualization diagram of two target channels of interest provided in an embodiment of the present invention;

[0068] Figure 4 This is a schematic diagram of the feature splitting module provided in an embodiment of the present invention;

[0069] Figure 5 This is a schematic diagram of the structure of the multi-scale feature extraction module provided in an embodiment of the present invention;

[0070] Figure 6 yes Figure 5 A schematic diagram of the structure of a multi-scale dilated convolutional block;

[0071] Figure 7 This is a schematic diagram of the cascaded feedback attention module provided in an embodiment of the present invention;

[0072] Figure 8 yes Figure 7 Schematic diagram of the mid-space attention mechanism;

[0073] Figure 9 yes Figure 7Schematic diagram of the mid-channel attention mechanism;

[0074] Figure 10 These are the input remote sensing images and the output labeled images provided in the embodiments of the present invention. Detailed Implementation

[0075] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solution of the present invention more clearly, and should not be used to limit the scope of protection of the present invention.

[0076] Example 1

[0077] like Figure 1 As shown, this invention provides a method for detecting floating objects based on remote sensing images, comprising the following steps:

[0078] S1. Acquire remote sensing images.

[0079] S2. Preprocess the remote sensing image.

[0080] The original remote sensing image was 8192×4732 pixels (width×height) and contained targets such as greenhouses and dust nets. We used image processing software to physically crop the original image to a size of 256 pixels × 256 pixels (width×height).

[0081] S3. Input the preprocessed remote sensing image into the trained floating object detection model to obtain a label image containing the detection results.

[0082] The floating object detection model was prepared in advance, and the preparation process is as follows:

[0083] Step 1: Creating a dataset of floating objects (in this embodiment, greenhouses and dust nets are used as examples).

[0084] During our research, there were no precedents for studies on greenhouses and dust control nets along railway lines, nor were there any suitable datasets for semantic segmentation tasks. Therefore, we used railway remote sensing images from Google Earth to create our dataset. The original images were 8192×4732 pixels (width×height) and contained targets such as greenhouses and dust control nets. We used image processing software to uniformly crop the original images into 256×256 pixels (width×height) sub-images and created labels. All sub-images used RGB channels, with the label images having a black background and different colors assigned to greenhouses and dust control nets. Sub-images without greenhouses and dust control nets were filtered out, resulting in 3500 suitable sub-images, each with a corresponding label image. The selected images were then divided into training and validation sets in a 7:3 ratio.

[0085] Step 2, construct as follows Figure 2 The floating object detection model shown is a semantic segmentation network.

[0086] Step 201, Selection of the backbone network. The backbone network is the main part of the network model, mainly used to extract various feature information from remote sensing images. The selection and optimization of the backbone network directly affects the segmentation effect of semantic segmentation. Typical backbone networks include VGGNet, MobileNet, ResNet, and DenseNet. In the semantic segmentation task of complex environments such as railway lines, in order to obtain high-precision semantic information, multiple layers of convolutional layers need to be stacked. The deeper the layer, the more complex and abstract the extracted information becomes. However, increasing the depth can lead to problems such as gradient explosion. The ResNet network increases the network depth and avoids gradient explosion by introducing residual blocks, which skip this layer and directly connect to the results of subsequent network layers. Therefore, this invention uses an improved ResNet-50 as the backbone network of the model. Compared with the original ResNet-50, considering that greenhouses and dust nets may have extremely large or small sizes, 16x downsampling will result in extremely large targets filling the entire feature layer, thus affecting feature extraction. Therefore, we only downsampled the fourth layer by 8 times, while increasing the number of channels to 1024 layers. We improved ResNet-50 to initially extract feature information from the remote sensing image and output the first feature map.

[0087] Step 202, Design of the feature-based traffic splitting module, the module framework is as follows: Figure 4 As shown. Although using an improved ResNet-50 backbone network can extract richer semantic information, the large amount of feature information will increase the burden on the network model during training and learning, and the output feature map will also contain strong and weak feature layers. The weak feature layers will affect the final result in subsequent processing. If we filter each feature layer of the feature map, retaining the strong feature layers and removing the weak feature layers, we can not only enhance feature extraction but also reduce the burden on network training and learning. Therefore, we innovatively designed a feature splitting module, which can filter the strong and weak features of the feature layers. The specific implementation steps are shown in steps 20201 to 20204.

[0088] Step 20201: The module performs feature stacking on the first and second layers of the first feature map output by the backbone network (improved ResNet-50), and also on the third and fourth layers. The second layer undergoes transformations such as size reduction and channel increase based on the first layer. Since the two layers in the feature stack need to be the same size, the first layer is subjected to a convolution transformation with a kernel of 3 and a stride of 2 before being stacked with the second layer. The third and fourth layers of the backbone network have the same size, so they can be directly stacked. The final result is a first intermediate feature map with sizes of 32×32×1024 and 16×16×2048 (width×height×channels), respectively.

[0089] Step 20202: The feature information contained in the feature map influences the network's subsequent focus on the target object. Therefore, we perform layer-by-layer weight calculation on the features in the two first intermediate feature maps obtained above. Global average pooling can compress the overall feature map, retaining important feature information. By averaging all elements in each feature layer of the feature map, a scalar value is obtained, which represents the amount of feature information contained in that feature layer. Therefore, we perform a global average pooling (GAP) calculation on the first intermediate feature map obtained above to obtain the weights of each feature layer.

[0090] In step S20203, we visualize the experimental samples after calculating the weights using the feature layer. The visualization results are as follows: Figure 3 As shown in the figure, we can clearly see that when the weights are negative, the smaller the weights, the lower the feature information contained in the feature layer, which may affect the performance of subsequent neural network layers; when the weights are positive, the larger the weights, the richer the feature information contained in the feature layer, and the closer it is to the original image. Therefore, we set the judgment criterion for weights as follows: feature layers with weights greater than 0 are strong feature layers, and feature layers with weights less than or equal to 0 are weak feature layers; the results obtained in step 20202 are filtered into strong feature layers and weak feature layers, and then the weak feature layers are removed to obtain two first intermediate strong feature maps.

[0091] Step 20204: Since the exact number of strong and weak feature layers is unknown, we perform a convolution operation to force a specific number of layers in the strong feature map composed of strong feature layers. The module performs a convolution operation with a kernel of 3 and a stride of 2 on the first intermediate strong feature map of the first and second layer feature stacks, resulting in a 16×16×512 strong feature map. The module then performs a convolution operation with a kernel of 3 and a stride of 1 on the first intermediate strong feature map of the third and fourth layer feature stacks, resulting in a 16×16×512 strong feature map. Finally, the strong feature maps obtained from the two convolutions are stacked to obtain the second feature map with the richest features, measuring 16×16×1024.

[0092] Step 203, design of the multi-scale feature extraction module, the module framework is as follows: Figure 5 As shown, the edge contours, colors, shapes, and sizes of greenhouses and dust nets in remote sensing images pose significant challenges to segmentation. After the encoder continuously downsamples the image, when the resulting feature map is smaller than the size of the greenhouse in the original image, features at this scale are difficult to capture, leading to the loss of semantic information about the edge contours of the greenhouse and dust net. Furthermore, the coverage area of ​​the greenhouse and dust net varies under different conditions, and using only a single sliding window cannot accurately capture the feature information. Therefore, we innovatively designed a multi-scale feature extraction module. Applying sliding windows of different sizes can enhance the network's ability to acquire global information, thereby achieving the goal of capturing special zone features at different scales. The multi-scale feature extraction module includes a multi-scale dilated convolution module, an average pooling layer, and a max pooling layer. The specific implementation process is shown in steps 20301 to 20303.

[0093] Step 20301: Dilated convolution is a special type of convolution that increases the receptive field by introducing a dilation coefficient into the convolution kernel. This type of convolution can be used to form sliding windows of various sizes to extract feature information from images. This module uses three dilated convolutions with different dilation coefficients in a cascaded manner to form a multi-scale dilated convolution module to extract features of different sizes. The module framework is as follows: Figure 6 As shown. The input is the output of the first and second layers of the backbone network, and the dilation coefficients of the modules are set to 2, 3, and 4. The 64×64×256 features of the first layer are transformed into a 56×56×512 feature map through multiple dilated convolutional blocks, and the 32×32×512 features of the second layer are transformed into a 24×24×1024 feature map through multiple dilated convolutional blocks. The implementation process of expanding the receptive field through multi-scale dilated convolutional blocks can be mathematically represented as follows:

[0094]

[0095] f sum1 =up[Conv r＝2 [Conv r＝2 (f in1 )]]+Conv r＝3 (f in1 )

[0096] f sum2 =up[Conv r＝2 [Conv r＝2 (f sum1 )]]+up[Conv r＝3 (f sum1 )]+Conv r＝4 (f in1 )

[0097] Where k′ is the size of the new nucleus after expansion, k is the original nucleus size, r is the expansion coefficient, RF is the receptive field, and f sum1 This is the first branch feature map, up[] is the upsampled feature map size adjustment, Conv r＝i This represents the dilated convolution with dilation coefficients r = i, where i = 2, 3, 4, f in1 It is the input to the multi-scale dilated convolution block, f sum2 It is the output of the multi-scale dilated convolution module.

[0098] Step 20302: For greenhouses and dust control nets with different coverage areas, this module designs three merging methods. The feature maps are averaged at sizes of 1×1, 3×3, and 9×9. The fourth layer 16×16×1024 feature map, after the three pooling methods, yields feature maps of sizes 8×8×1024, 7×7×1024, and 4×4×1024, respectively. This method improves the ability to acquire global information by aggregating contextual features of different regions and sizes, which is beneficial for improving the accuracy of segmenting extremely large or small greenhouses and dust control nets. The mathematical expression for the specific implementation process is as follows:

[0099] f a =Conv 1×1 {Avgpool 1×1 (input3)+Avgpool 3×3 (input3)+Avgpool 9×9 (input3)}

[0100] f m =Conv 1×1 {MSDConv(input1)+MSDConv(input2)}

[0101] Among them, f a This indicates that input3 is processed through three average pooling layers of different sizes, followed by feature stacking, and then the output f. m This represents the output of input1 and input2 after multi-scale dilated convolution and feature stacking. Input1 is the input from the first layer of the backbone network, input2 is the input from the second layer of the backbone network, and input3 is the input from the fourth layer of the backbone network. Conv 1×1 Avgpool represents a 1×1 convolution. 1×1 Avgpool represents 1×1 average pooling. 3×3 Avgpool represents 3×3 average pooling. 9×9 represents 9×9 average pooling, and MSDConv represents multi-scale dilated convolution.

[0102] Step 20303: In the final module, max pooling is performed, followed by batch normalization (BN) after the 1×1 convolution to prevent gradient explosion and accelerate training. Then, the Sigmoid function is used to generate channel attention tensors as weights to optimize the feature map, thereby facilitating the identification and extraction of boundary information. The mathematical expression for the specific implementation process is as follows:

[0103] f w =Sigmoid{Conv 1×1 (Maxpool(input3))}

[0104] Among them, f w It is the probability output obtained after processing by the max pooling layer and the sigmoid function. Maxpool means max pooling.

[0105] f out1 =f w ×(f m +f a )

[0106] Among them, f out1 This is the output of the multi-scale feature extraction module.

[0107] Step 204, Design of the cascaded feedback attention module, the module framework is as follows: Figure 7 As shown, deep convolutional neural networks can transform the RGB color space in railway remote sensing images into a high-level feature space. However, due to the complex and diverse environment along railway lines, traditional models extract feature information from various irrelevant targets, such as green belts and highways. When the model cannot accurately distinguish targets, pixels cannot be accurately assigned to the correct labels. Enhancing the network's attention to regions of interest can improve segmentation accuracy. Therefore, we designed a cascaded feedback attention module. This module calculates weights for both the spatial and channel dimensions of the input feature map, and then fuses these weights with the original input feature map, thereby improving the model's accuracy in identifying greenhouses and dust nets in railway remote sensing images with complex backgrounds.

[0108] Step 20401: To help the model accurately locate the target areas of the greenhouse and dust net in the remote sensing image, we introduce a spatial attention mechanism. The framework of the spatial attention mechanism is as follows: Figure 8 As shown. Input image f in2After performing max pooling and average pooling operations respectively, the image size is compressed while maintaining the same number of channels, resulting in two (1, 1, C) feature maps. The module then feeds these two feature maps into a multilayer perceptron (MLP), which has two layers of neurons, with ReLU as the activation function between them. The output feature maps of the MLP are stacked channel by channel and then subjected to a sigmoid activation operation to obtain a weighted feature vector W. c , will f in2 and W c f is obtained by fusion c The mathematical expression for the above transformation process is as follows:

[0109] W c =Sigmoid(MLP(AvgPool(f in2 ))+MLP(MaxPool(f in2 )))

[0110] f c =f in2 ×W c

[0111] In the formula W c This represents the feature vector obtained after the input undergoes a spatial attention mechanism, where Sigmoid represents the activation function, MLP represents a multilayer perceptron, AvgPool represents average pooling, and MaxPool represents max pooling. in2 f represents the input to the cascaded feedback attention module. c Indicates input f in2 The weighted output of the feature vectors obtained by the spatial attention mechanism.

[0112] Step 20402: To help the model select the most suitable channels for the greenhouse and dust control net in the feature map, we introduce a channel attention mechanism. The framework of the channel attention mechanism is as follows: Figure 9 As shown. By applying different attention weights to each channel, the model can automatically learn the importance of each channel, improving its expressive power and performance. Input image f c After performing max pooling and average pooling operations respectively, the number of channels in the image is compressed while maintaining the same image size, resulting in two (W, H, 1) feature vectors. The model stacks these two feature vectors and then feeds the result into a 7×7 convolutional layer. This convolutional layer calculates the correlation between each channel and other channels, outputting a (W, H, 1) feature vector. Next, the model uses the sigmoid function to increase the non-linearity of the features, obtaining the channel attention weight vector W. s , will f in2 and W s f is obtained by fusion sFinally, the model will f in2 and f s Channel features are superimposed, and then processed through a 1×1 convolutional layer to adjust the channels, resulting in an output that is consistent with f. in2 Output image f with consistent size and channels out The mathematical expression for the transformation process is as follows:

[0113] W s =Sigmoid(Conv 7×7 ([AvgPool(f c ), MaxPool(f c )]))

[0114] f s =f in2 ×W s

[0115] f out2 =Conv 1×1 (f s +f in2 )

[0116] In the formula W s This represents the feature vector obtained after the input undergoes a spatial attention mechanism, where Sigmoid represents the activation function, and Conv... 7×7 This indicates a convolution with a kernel of 7, AvgPool represents average pooling, MaxPool represents max pooling, and f c f represents the weighted output of the input and the feature vector obtained by the spatial attention mechanism. s Indicates input f in2 The weighted output of the feature vector obtained by the channel attention mechanism, f in2 f represents the input to the cascaded feedback attention module. out2 This represents the output of the cascaded feedback attention module.

[0117] Step 3, Model Learning and Training Process. Using the dataset obtained in S1, the network model is trained in the segmentation network established in S2. This invention adopts a supervised training mode, and the specific steps are as follows:

[0118] Step 301: Manually label the datasets of greenhouses and dust nets, and then perform operations such as flipping and rotating the datasets to expand the datasets.

[0119] Step 302: The images and labels are matched one-to-one as input to the model. The experimental platform uses NVIDIA GeForce RTX 3080 and PyTorch environment. Cross-entropy is used as the maximum loss function. The Adam optimizer is used to update the model's learning rate. The initial learning rate is set to 0.0015. The exponential decay coefficient of the model is 0.95. The image training batch is set to 10. The number of iterations is set to 300.

[0120] Model prediction process:

[0121] The experimental results selected Mean Intersection over Union (MIOU), Pixel Precision (PA), Recall (Rc), and Precision (P) as quantitative indicators, with MIOU being the primary evaluation metric. The formulas for each indicator are as follows.

[0122]

[0123]

[0124]

[0125]

[0126] In the formula, TP represents the target pixel correctly predicted in the sample image, TN represents the non-target pixel correctly predicted in the sample image, FP represents the non-target pixel predicted as the target pixel in the sample image, and FN represents the target pixel predicted as the non-target pixel in the sample image.

[0127] Input a sample image, and after processing the trained weights to extract features, perform encoding and decoding operations, output labeled images of greenhouses and dust control nets, such as... Figure 10 As shown.

[0128] Example 2

[0129] This invention also provides a floating object detection device based on remote sensing images, comprising:

[0130] The image acquisition module is configured to acquire remote sensing images;

[0131] The image preprocessing module is configured to preprocess the remote sensing image;

[0132] The floating object detection module is configured to input preprocessed remote sensing images into a trained floating object detection model to obtain labeled images containing the detection results.

[0133] Example 3

[0134] This invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the floating object detection method based on remote sensing images as provided in Embodiment 1;

[0135] Acquire remote sensing images;

[0136] The remote sensing image is preprocessed;

[0137] The preprocessed remote sensing image is input into the trained floating object detection model to obtain a labeled image containing the detection results.

[0138] Example 4

[0139] This invention provides a device comprising:

[0140] Memory, used to store instructions;

[0141] The processor is configured to execute the instructions, causing the device to perform operations implementing the floating object detection method based on remote sensing images as provided in Embodiment 1:

[0142] Acquire remote sensing images;

[0143] The remote sensing image is preprocessed;

[0144] The preprocessed remote sensing image is input into the trained floating object detection model to obtain a labeled image containing the detection results.

[0145] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting floating objects based on remote sensing images, characterized in that, include: Acquire remote sensing images; The remote sensing image is preprocessed; The preprocessed remote sensing image is input into the trained floating object detection model to obtain a labeled image containing the detection results; The floating object detection model includes an improved ResNet-50, a feature splitting module, a multi-scale feature extraction module, and a cascaded feedback attention module; The improved ResNet-50 is the backbone network of the floating object detection model, used to initially extract feature information from remote sensing images and output the first feature map; The feature splitting module performs strong and weak feature filtering on the first feature map, retains the strong feature layer in the first feature map, and outputs the second feature map. The multi-scale feature extraction module extracts a multi-scale feature map from the second feature map; The cascaded feedback attention module calculates the weights of the multi-scale feature map in the spatial and channel dimensions, respectively, and fuses the obtained weights with the multi-scale feature map to obtain the label image. The feature splitting module performs strong and weak feature filtering on the first feature map, including: The first and second feature layers of the first feature map are stacked, and the third and fourth feature layers of the first feature map are stacked to obtain two first intermediate feature maps. The weights of the two first intermediate feature maps are calculated layer by layer using global average pooling; Feature layers with weights greater than 0 are strong feature layers, and feature layers with weights less than or equal to 0 are weak feature layers. By removing the weak feature layers from the two first intermediate feature maps, two first intermediate strong feature maps are obtained. The two first intermediate strong feature maps are convolved separately, and then the features are stacked to obtain the second feature map.

2. The floating object detection method based on remote sensing images according to claim 1, characterized in that, The preprocessing includes cropping the remote sensing image to a size of 256 x 256 pixels.

3. The floating object detection method based on remote sensing images according to claim 1, characterized in that, The improvements of the improved ResNet-50 compared to ResNet-50 include: changing the downsampling of the fourth feature layer to 8 times downsampling, while increasing the number of channels to 1024 layers.

4. The floating object detection method based on remote sensing images according to claim 1, characterized in that, The multi-scale feature extraction module includes a multi-scale dilated convolution module, an average pooling layer, a max pooling layer, and a multi-scale dilated convolution module connected in sequence. function; The multi-scale dilated convolution module includes three dilated convolutions with different dilation coefficients connected in a cascaded manner. The expression for the multi-scale dilated convolution module is as follows: ； ； ； ； in, It is the size of the new nucleus after expansion. It is the original kernel size. It is the coefficient of thermal expansion. It is about experiencing the wild. This is the first branch feature map. It involves upsampling to adjust the feature map size. Indicates the coefficient of thermal expansion dilated convolution, , It is the input to a multi-scale dilated convolution block. It is the output of the multi-scale dilated convolution module; The expression for the average pooling layer is: ； ； in, express The output is obtained after three average pooling layers of different sizes and then through feature stacking. express , The output is processed through multi-scale dilated convolution and then through feature stacking. It is the input from the first layer of the backbone network. It is the input from the second layer of the backbone network. It is the input from the fourth layer of the backbone network. This represents a 1×1 convolution. This represents 1×1 average pooling. This represents 3×3 average pooling. This represents 9×9 average pooling. This represents multi-scale dilated convolution; The maximum pooling layer and The function expression is: ； in, It is after max pooling layer and The probability output obtained from the function processing, This indicates max pooling; ； in, This is the output of the multi-scale feature extraction module.

5. The floating object detection method based on remote sensing images according to claim 1, characterized in that, The expression for the cascaded feedback attention module is: ； ； ； ； ； in, This represents the feature vector output by the spatial attention mechanism. express Activation function This represents a multilayer perceptron. Indicates average pooling. This indicates max pooling. This represents the input to the cascaded feedback attention module. express The weighted output of the feature vectors from the spatial attention mechanism. This represents the feature vector output by the spatial attention mechanism. This represents a convolution with a kernel of 7. express The weighted output of the feature vectors from the channel attention mechanism. This represents the input to the cascaded feedback attention module. This represents the output of the cascaded feedback attention module.

6. A floating object detection device based on remote sensing images, characterized in that, include: The image acquisition module is configured to acquire remote sensing images; The image preprocessing module is configured to preprocess the remote sensing image; The floating object detection module is configured to input the preprocessed remote sensing image into the trained floating object detection model to obtain a labeled image containing the detection results. The floating object detection model includes an improved ResNet-50, a feature splitting module, a multi-scale feature extraction module, and a cascaded feedback attention module. The improved ResNet-50 is the backbone network of the floating object detection model, used to initially extract feature information from remote sensing images and output the first feature map; The feature splitting module performs strong and weak feature filtering on the first feature map, retains the strong feature layer in the first feature map, and outputs the second feature map. The multi-scale feature extraction module extracts a multi-scale feature map from the second feature map; The cascaded feedback attention module calculates the weights of the multi-scale feature map in the spatial and channel dimensions, respectively, and fuses the obtained weights with the multi-scale feature map to obtain the label image. The feature splitting module performs strong and weak feature filtering on the first feature map, including: The first and second feature layers of the first feature map are stacked, and the third and fourth feature layers of the first feature map are stacked to obtain two first intermediate feature maps. The weights of the two first intermediate feature maps are calculated layer by layer using global average pooling; Feature layers with weights greater than 0 are strong feature layers, and feature layers with weights less than or equal to 0 are weak feature layers. By removing the weak feature layers from the two first intermediate feature maps, two first intermediate strong feature maps are obtained. The two first intermediate strong feature maps are convolved separately, and then the features are stacked to obtain the second feature map.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the floating object detection method based on remote sensing images as described in any one of claims 1-5.

8. A device, characterized in that, include: Memory, used to store instructions; A processor is configured to execute the instructions, causing the device to perform operations that implement the floating object detection method based on remote sensing images as described in any one of claims 1-5.

Images