A method for crop region extraction based on feature interaction and attention decoding

Abstract

The application discloses a crop area extraction method based on feature interaction and attention decoding, which comprises the following steps: step 1, remote sensing images are acquired by using a UAV, and a series of data preprocessing is performed on the remote sensing images to make a data set; step 2, a deep feature interaction module is constructed in a network model; step 3, a spatial attention decoding module is constructed in the network model, and the overall network model is trained by using the data set to obtain trained model weights; step 4, a remote sensing area image to be identified is input into the model to obtain a segmentation result; and step 5, the area of a crop area is calculated according to the resolution of the remote sensing image and the segmentation result. The application can save a large amount of manpower and material resources and effectively extract the area of a rural crop area.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, specifically to a method for extracting rural crop areas using unmanned aerial vehicle (UAV) technology. Background Technology

[0002] Crop planting management and yield estimation have always been key concerns for the government. Crops are essential for human survival, a vital guarantee for national industrial development, and a crucial foundation for maintaining social stability. With the rapid pace of urbanization in my country, many rural areas are seeing a reduction in crop planting areas. Therefore, quickly assessing the area of ​​crop-growing regions is of significant practical importance for crop yield estimation and planting management.

[0003] Due to the rapid development of drone technology, a large number of remote sensing images of the same area can be acquired in a short time. These images are of very high quality, easily capturing important information such as crop color, texture, and shape, providing a crucial foundation for crop region extraction. Initially, many researchers used machine learning methods for crop region extraction. While this achieved some success, the complex background of crop areas and the diverse characteristics of crops themselves meant that machine learning methods couldn't effectively filter background noise and extract crop features. With the widespread application of semantic segmentation technology across various fields, agriculture has also adopted it, but problems persist, including insufficient extraction of target features and poor edge segmentation. The paper "Remote Sensing Image Farmland Segmentation and Extraction" by Zhang Yuanfang et al. used an improved PSPnet to extract farmland regions. Although this replaced the original backbone network and reduced model complexity, it resulted in insufficient feature extraction. Furthermore, after feature fusion, a crude upsampling operation was used to obtain the predicted image, without considering the benefits of the interaction between shallow and deep features in fitting the segmentation boundary.

[0004] This invention constructs a deep feature interaction module and a spatial attention decoding module to extract features from crop areas more fully, reduce the impact of noise, and better fit the segmentation boundary, thereby achieving rapid extraction of the area of ​​crop regions. Summary of the Invention

[0005] The purpose of this invention is to solve the technical problems of insufficient feature extraction and poor edge segmentation of crop regions in the prior art, and to provide a method for crop region extraction based on feature interaction and attention decoding.

[0006] The technical solution of this invention is as follows:

[0007] A method for crop region extraction based on feature interaction and attention decoding includes the following steps:

[0008] Step 1: Acquire remote sensing images and perform a series of data preprocessing steps on the remote sensing images to create a dataset;

[0009] Step 2: Construct a deep feature interaction module in the network model;

[0010] Step 3: Construct a spatial attention decoding module in the network model, and train the overall network model with the dataset to obtain the trained model weights;

[0011] Step 4: Input the image of the remote sensing region to be identified into the network model to obtain the segmentation result;

[0012] Step 5: Calculate the area of ​​the crop region based on the resolution and segmentation results of the remote sensing image.

[0013] In step 2, a deep feature interaction module is constructed to extract and fuse multi-scale features of crops, including the following steps:

[0014] 2-1: The backbone network ResNet50 consists of 5 sets of convolutions. When an image is input into ResNet50, feature maps F0, F1, F2, F3, and F4 are obtained sequentially. The obtained feature map F4 is input into the deep feature interaction module, which consists of parallel convolution branches (one convolution operation Conv1, three dilated convolution operations Conv2, Conv3, and Conv4, and one global average pooling operation Pool) and two channel attentions.

[0015] 2-2: Feature map F4 is convolved by Conv1 to obtain feature map C1, and then convolved by Conv2 to obtain feature map C2. Feature map C2 and feature map F4 are concatenated on the channel and processed by the channel attention module, and then used as input to convolution Conv3 to obtain feature map C3. Feature map C3 and feature map F4 are concatenated on the channel and processed by the channel attention module, and then used as input to convolution Conv4 to obtain feature map C4. Feature map F4 is then processed by global average pooling to obtain feature map C5. After parallel operation, the five feature maps are fused to obtain high-level feature map F5, and the number of channels of feature map F5 is reduced.

[0016] In step 3, a spatial attention decoding module is constructed, and then the overall network model containing the deep feature interaction module and the spatial attention decoding module is trained using the dataset, including the following steps:

[0017] 3-1: Obtain two low-level feature maps F1 and F2 from ResNet50. Upsample feature map F2 to the same size as feature map F1. Perform a 1×1 convolution on feature map F2 and adjust the number of channels. Input feature maps F1 and F2 into the spatial attention module and concatenate the channels to obtain a fused feature map. Use a 3×3 convolution to reduce the number of channels to obtain feature map F6.

[0018] 3-2: Upsample the high-level feature map F5, input feature maps F5 and F6 into the spatial attention module and perform channel concatenation to obtain the fused feature map, and then perform two 3×3 convolution and upsampling operations.

[0019] 3-3: After the network is built, input the completed dataset into the network for training and obtain the network model weights.

[0020] In steps 3-1 and 3-2, constructing the spatial attention module includes the following steps: performing average pooling and max pooling operations on the input feature map FX (in this invention, F2 after upsampling and convolution is the input feature map FX; F5 after upsampling is the input feature map FX) to obtain feature maps Favg and Fmax, which are used for high-level feature acquisition of spatial attention points. Favg and Fmax are fused in channels to obtain a feature map with 2 channels, and then a 7×7 convolution operation is performed to reduce the number of channels to 1. Next, a sigmoid operation is performed, and finally multiplied with the feature map FY (in this invention, F1 is the feature map FY; F6 is the feature map FY) to weight FY.

[0021] In step 3-3, when training the overall network model, a hybrid loss function is used, defined as follows:

[0022]

[0023] Where N is the total number of pixels, M is the number of categories, ε is the smoothing coefficient, c is a certain category, and t ic For the target label, y ic Let be the probability that the i-th pixel belongs to class c.

[0024] Network models (e.g.) Figure 2 As shown below:

[0025] Encoding layer 1 → Encoding layer 2 → Encoding layer 3 → Encoding layer 4 → Encoding layer 5 → Deep feature interaction module → Output feature map of deep feature interaction module;

[0026] Second coding layer → Output feature map of the second coding layer;

[0027] The third layer of the coding layer → the output feature map of the third layer of the coding layer;

[0028] The output feature map of the second layer of the encoding layer, the output feature map of the third layer of the encoding layer, and the output feature map of the deep feature interaction module → spatial attention decoding module;

[0029] The output feature map of the deep feature interaction module is upsampled, and then the spatial attention decoding module performs a 3×3 convolution operation on the feature fusion map, followed by upsampling and a prediction map.

[0030] Deep feature interaction module (such as) Figure 3 As shown below:

[0031] Output feature map of the backbone network → convolution operation;

[0032] Output feature map of the backbone network → first dilated convolution operation;

[0033] The output feature map of the backbone network consists of the first dilated convolution operation → the first concat operation → the first channel attention operation → the second dilated convolution operation.

[0034] The output feature map of the backbone network, the second dilated convolution operation → the second concat operation → the second channel attention operation → the third dilated convolution operation;

[0035] Output feature map of the backbone network → global average pooling operation;

[0036] Convolution operation, first dilated convolution operation, second dilated convolution operation, third dilated convolution operation, global average pooling operation → third concat operation → feature fusion map.

[0037] Spatial attention decoding module (e.g.) Figure 4 As shown below:

[0038] Input feature map F2 → Convolution operation → 2x upsampling operation → Upsampled feature map;

[0039] Input feature map F1, upsampled feature map → first spatial attention operation;

[0040] Upsampled feature map, first spatial attention operation → first concat operation → feature fusion map → convolution operation → feature map F6 after convolution;

[0041] Input feature map F5 → 4x upsampled feature map → upsampled feature map;

[0042] Feature map F6 after convolution, upsampled feature map → second spatial attention operation;

[0043] The upsampled feature map undergoes a second spatial attention operation followed by a second concat operation, resulting in a fused feature map.

[0044] Both the first and second spatial attention operations used the spatial attention module.

[0045] Spatial attention modules (e.g.) Figure 5 As shown below:

[0046] Input feature map FX → Average pooling feature map;

[0047] Input feature map FX → Max pooled feature map;

[0048] Average pooling feature map, max pooling feature map → Concat operation → Feature fusion map → Convolution operation → Sigmoid operation → Normalized feature map;

[0049] Input feature map FY, normalized feature map → feature map multiplication operation → output feature map.

[0050] Compared with the prior art, the present invention has the following technical effects:

[0051] 1) This method provides a way to extract rural crop areas using UAV remote sensing imagery. Compared with traditional extraction methods, this method trains its model with a large amount of data, resulting in a model with strong generalization ability, which is more suitable for extracting large-scale crop areas. Traditional segmentation methods, on the other hand, require selection based on different crop areas, have weak generalization ability, and require the creation of multiple datasets to meet the requirements for effective area extraction. In comparison, the method of this invention is more cost-effective.

[0052] 2) This invention creates a deep feature interaction module that uses convolutions with different dilation rates to extract features, forming parallel branches to obtain feature information at different scales. The outputs of adjacent branches are concatenated, allowing feature information from different branches to interact. Simultaneously, channel attention is used to enhance the weights of effective features, enabling effective extraction of target information even in complex natural backgrounds, while reducing the impact of noise on target extraction.

[0053] 3) This invention creates a spatial attention decoding module that uses spatial attention to guide the fusion of high-level and low-level features. Low-level features contain more edge information. Spatial attention is used to enhance the positional information of high-level features, thereby weighting the low-level features with enhanced positional information and strengthening the edge information. By fusing high-level and low-level features, the edge information is better fitted, and the boundary segmentation effect is optimized. Attached Figure Description

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0055] Figure 1 This is a flowchart of the present invention;

[0056] Figure 2 This is a network structure diagram of the present invention;

[0057] Figure 3 This is a diagram of the deep feature interaction module of the present invention;

[0058] Figure 4 This is a diagram of the spatial attention decoding module of the present invention;

[0059] Figure 5 This is a diagram of the spatial attention module of the present invention. Detailed Implementation

[0060] like Figure 1 As shown, a method for crop region extraction based on feature interaction and attention decoding includes the following steps.

[0061] Step 1: Use drones to acquire remote sensing images and perform a series of data preprocessing steps on the remote sensing images to create a dataset;

[0062] Step 2: Construct a deep feature interaction module and a spatial attention decoding module in the network model;

[0063] Step 3: Train the overall network model using the dataset to obtain the trained model weights;

[0064] Step 4: Input the remote sensing image of the area to be identified into the model to obtain the segmentation result, and calculate the area of ​​the crop area based on the resolution of the remote sensing image and the segmentation result;

[0065] Step 1 specifically includes the following steps:

[0066] 1) Use drones to photograph crop areas and obtain drone remote sensing images;

[0067] 2): Select suitable images, annotate them, and create a dataset;

[0068] 3) Use data augmentation methods to augment the dataset. This includes methods such as random scaling, arbitrary rotation, and adding noise to expand the dataset's size.

[0069] Step 2 specifically includes the following steps:

[0070] 1) Construct a deep feature interaction module to extract and fuse multi-scale features of crops;

[0071] 2): Construct a spatial attention decoding module to optimize the segmentation results;

[0072] In step 1), the deep feature interaction module consists of parallel convolutional branches and channel attention modules. The output of one parallel convolution is concatenated with the output of the backbone network ResNet50, serving as the input to the adjacent parallel convolution. Channel concatenation is performed on the outputs of the parallel branches, and finally, channel compression is applied. Specifically, the following steps are included:

[0073] 1-1): The backbone network ResNet50 consists of 5 sets of convolutions. When an image is input into ResNet50, feature maps F0, F1, F2, F3, and F4 are obtained in sequence. The obtained feature map F4 is input into the deep feature interaction model. The parallel convolutions include one convolution operation Conv1 (with a kernel size of 1), three dilated convolution operations Conv2, Conv3, and Conv4, one global average pooling operation Pool, and two channel attention operations.

[0074] 1-2): F4 undergoes convolution operation Conv1 to obtain feature map C1, and then undergoes convolution operation Conv2 to obtain feature map C2. C2 and F4 are concatenated along the channel and processed by the channel attention module, serving as input to convolution Conv3 to obtain feature map C3. C3 and F4 are concatenated along the channel and processed by the channel attention module, serving as input to convolution Conv4 to obtain feature map C4. F4 undergoes global average pooling to obtain feature map C5. After parallel operations, the five feature maps are fused to obtain high-level feature map F5, and the number of channels in F5 is reduced.

[0075] In step 2), channel fusion is performed on feature maps F1 and F2 to reduce the number of channels. Then, they are concatenated with the high-level feature map along channels, again reducing the number of channels. Spatial attention is used to guide the fusion of high-level and low-level features in this process. Specifically, this includes the following steps:

[0076] 2-1) Obtain two low-level feature maps F1 and F2 from ResNet50. Upsample F2 to the same size as F1. Perform a 1×1 convolution operation on F2, adjust the number of channels, and then combine F1 and F2 (F1 corresponds to F1). Figure 5 FY, F2 correspond to Figure 5 The FX input spatial attention module is used to concatenate channels to obtain a fused feature map. A 3×3 convolution is then used to reduce the number of channels to obtain feature map F6.

[0077] 2-2) Upsample the high-level feature map F5, and then compare F5 and F6 (F6 corresponds to...). Figure 5 FY, F5 corresponds to Figure 5The FX input spatial attention module is used to perform channel concatenation to obtain a fused feature map. Then, two 3×3 convolutions and upsampling operations are performed to obtain the prediction result.

[0078] In step 2, the loss function used for network training is defined as follows:

[0079]

[0080] Where N is the total number of pixels, M is the number of categories, ε is the smoothing coefficient, c is a certain category, and t ic For the target label, y ic Let be the probability that the i-th pixel belongs to class c.

[0081] In step 3, the dataset is used to train the overall network model to obtain the trained model weights.

[0082] In step 4, the drone remote sensing image to be identified is segmented and input into the model to obtain the segmentation result. The number of pixels of the mask identified as agricultural area is calculated, and then the area of ​​the overall agricultural area is calculated based on the area represented by a single pixel.

[0083] Example

[0084] The present invention provides a method for extracting rural crop areas from UAV remote sensing images, which is carried out in the following manner:

[0085] Step 1: The drone acquires remote sensing imagery. The imagery is then filtered, and suitable images are selected. ENVI is used to label crop areas, creating label maps. After labeling, the remote sensing images and label maps are cut into 512×512 images to create a dataset, which is then randomly divided into training and validation sets. OpenCV is used to augment the dataset using methods including random scaling, random angle rotation, and adding Gaussian noise.

[0086] Step 2: Input the image into the backbone network. The image passes through convolution sets Layer1, Layer2, Layer3, and Layer4, resulting in feature maps of sizes 256×128×128 (F1), 512×64×64 (F2), 1024×32×32 (F3), and 2048×32×32 (F4), respectively. F4 undergoes a 1×1 convolution operation to obtain feature map C1. F4 then undergoes a 3×3 dilated convolution with r=6 to obtain feature map C2. C2 and F4 are concatenated along the channels and processed by the channel attention module, serving as input to a 3×3 dilated convolution with r=12 to obtain feature map C3. C3 and F4 are concatenated along the channels and processed by the channel attention module, serving as input to a 3×3 dilated convolution with r=18 to obtain feature map C4. F4 undergoes global average pooling to obtain feature map C5. After parallel operations, five feature maps are obtained and fused to obtain a high-level feature map F5. The number of feature channels of F5 is reduced by 1×1 convolution.

[0087] Step 3: Obtain two low-level feature maps F1 and F2 from ResNet50. Upsample F2 to the same size as F1. Perform a 1×1 convolution on F2 to adjust the number of channels. Input F1 and F2 into the spatial attention module and concatenate the channels to obtain a fused feature map. Use a 3×3 convolution to reduce the number of channels to obtain feature map F6. Upsample the high-level feature map F5. Input F5 and F6 into the spatial attention module and concatenate the channels to obtain a fused feature map. Then perform two 3×3 convolutions and upsampling operations to obtain the prediction result.

[0088] Step 4: Define the loss function:

[0089]

[0090] Where N is the total number of pixels, M is the number of categories, ε is the smoothing coefficient, c is a certain category, and t ic For the target label, y ic Let be the probability that the i-th pixel belongs to class c.

[0091] Step 5: Segment the extracted UAV remote sensing image and input the segmentation result into the pre-trained model; after stitching the predicted images together, calculate the overall area of ​​the crop region based on the number of pixels of the mask identified as crop areas in the image and the area represented by a single pixel.

[0092] Table 1 shows the comparison results between the proposed model and different network models. It can be seen that the segmentation effect of our model is better than that of the comparison models.

[0093]

[0094] Table 1 Comparison of results for different network models

[0095] In this invention, the model's Iou is 4.15 higher than that of Apcnet, 2.46 higher than that of Dmnet, 3.31 higher than that of Dnlnet, 3.72 higher than that of Isanet, and 1.20 higher than that of Knet.

[0096] This method provides a way to extract rural crop areas using UAV remote sensing imagery. Compared with traditional extraction methods, this method trains its model with a large amount of data, resulting in a model with strong generalization ability, which is more suitable for extraction tasks covering large areas of crop areas.

Claims

1. A method for crop region extraction based on feature interaction and attention decoding, characterized in that, Includes the following steps: Step 1: Acquire remote sensing images and perform a series of data preprocessing steps on the remote sensing images to create a dataset; Step 2: Construct a deep feature interaction module in the network model; Step 3: Construct a spatial attention decoding module in the network model, and train the overall network model with the dataset to obtain the trained model weights; Step 4: Input the image of the remote sensing region to be identified into the network model to obtain the segmentation result; Step 5: Calculate the area of ​​the crop region based on the resolution and segmentation results of the remote sensing image; In step 2, a deep feature interaction module is constructed to extract and fuse multi-scale features of crops, including the following steps: 2-1: The backbone network ResNet50 consists of 5 sets of convolutions. When an image is input into ResNet50, feature maps F0, F1, F2, F3, and F4 are obtained in sequence. The obtained feature map F4 is input into the deep feature interaction module, which consists of parallel convolution branches and two channel attention. 2-2: Feature map F4 is convolved by Conv1 to obtain feature map C1, and then convolved by Conv2 to obtain feature map C2. Feature map C2 and feature map F4 are concatenated on the channel and processed by the channel attention module, and then used as input to convolution Conv3 to obtain feature map C3. Feature map C3 and feature map F4 are concatenated on the channel and processed by the channel attention module, and then used as input to convolution Conv4 to obtain feature map C4. Feature map F4 is then processed by global average pooling to obtain feature map C5. After parallel operation, the five feature maps are fused to obtain high-level feature map F5, and the number of channels of feature map F5 is reduced.

2. The method according to claim 1, characterized in that, In step 3, a spatial attention decoding module is constructed, and then the overall network model containing the deep feature interaction module and the spatial attention decoding module is trained using the dataset, including the following steps: 3-1: Obtain two low-level feature maps F1 and F2 from ResNet50. Upsample feature map F2 to the same size as feature map F1. Perform a 1×1 convolution on feature map F2 and adjust the number of channels. Input feature maps F1 and F2 into the spatial attention module and concatenate the channels to obtain a fused feature map. Use a 3×3 convolution to reduce the number of channels to obtain feature map F6. 3-2: Upsample the high-level feature map F5, input feature maps F5 and F6 into the spatial attention module and perform channel concatenation to obtain the fused feature map, and then perform two 3×3 convolution and upsampling operations. 3-3: After the network is built, input the completed dataset into the network for training and obtain the network model weights.

3. The method according to claim 1, characterized in that, In step 3-3, when training the overall network model, a hybrid loss function is used, defined as follows: ； in, Total number of pixels For the number of categories, For smoothing coefficients, For a certain category, For target tags, For the first The pixel is the first The probability of a class.

4. The method according to any one of claims 1 to 3, characterized in that, The network model is as follows: Encoding layer 1 (1) → Encoding layer 2 (2) → Encoding layer 3 (3) → Encoding layer 4 (4) → Encoding layer 5 (5) → Deep feature interaction module (6) → Output feature map of deep feature interaction module (9); The second layer of the coding layer (2) → the output feature map of the second layer of the coding layer (7); The third layer of the coding layer (3) → the output feature map of the third layer of the coding layer (8); Output feature map of the second layer of the coding layer (7), output feature map of the third layer of the coding layer (8), output feature map of the deep feature interaction module (9) → spatial attention decoding module (10); Upsample the output feature map (9) of the deep feature interaction module, then the spatial attention decoding module (10) → feature fusion map (11) → 3×3 convolution operation (12), upsample → prediction map (13).

5. The method according to claim 4, characterized in that, The deep feature interaction module is as follows: Output feature map of the backbone network (14) → Convolution operation (15); Output feature map of the backbone network (14) → First dilated convolution operation (16); The output feature map of the backbone network (14), the first dilated convolution operation (16) → the first concat operation (17) → the first channel attention operation (18) → the second dilated convolution operation (19); Output feature map of the backbone network (14), second dilated convolution operation (19) → second concat operation (20) → second channel attention operation (21) → third dilated convolution operation (22); Output feature map of the backbone network (14) → Global average pooling operation (23); Convolution operation (15), first dilated convolution operation (16), second dilated convolution operation (19), third dilated convolution operation (22), global average pooling operation (23) → third concat operation (24) → feature fusion map (25).

6. The method according to claim 4, characterized in that, The spatial attention decoding module is as follows: Input feature map F2 (29) → Convolution operation (30) → 2x upsampling operation (31) → Upsampled feature map (32); Input feature map F1 (26), upsampled feature map (32) → first spatial attention operation (27); The upsampled feature map (32), the first spatial attention operation (27) → the first concat operation (28) → the feature fusion map (33) → the convolution operation (34) → the convolutional feature map F6 (35); Input feature map F5 (36) → 4 times upsampled feature map (37) → upsampled feature map (38); Feature map F6 (35) after convolution, feature map (38) after upsampling → second spatial attention operation (39); Upsampled feature map (38), second spatial attention operation (39) → second concat operation (40) → feature fusion map (41); The first spatial attention operation (27) and the second spatial attention operation (39) both use the spatial attention module.

7. The method according to claim 6, characterized in that, The spatial attention module is as follows: Input feature map FX (42) → Average pooling feature map (43); Input feature map FX (42) → Max pooled feature map (44); Average pooling feature map (43), max pooling feature map (44) → Concat operation (45) → Feature fusion map (46) → Convolution operation (47) → Sigmod operation (48) → Normalized feature map (49); Input feature map FY (50), normalized feature map (49) → feature map multiplication operation (51) → output feature map (52).

Images