A method for extracting lodging area of multi-bearing period wheat

Abstract

This invention relates to a method for extracting the lodging area of ​​wheat at multiple growth stages, comprising: data acquisition; image preprocessing; data augmentation; screening the augmented images, removing images without lodging, and randomly dividing the images into training and validation sets at a 4:1 ratio; improving the Mask2Former network model, inputting the training set into the improved Mask2Former network model for training, and selecting the optimal segmentation model; inputting the wheat image to be detected into the optimal segmentation model to calculate the actual lodging area of ​​wheat. This invention collects wheat lodging data at different growth stages and applies data augmentation to simulate complex natural environments in the field to improve the robustness and generalization of the model. The introduction of the Hierarchical Interactive Feature Pyramid Network (HI-FPN) improves the Mask2Former network model, realizing the extraction and fusion of multi-scale features, improving the accuracy of lodging area segmentation, and accurately calculating the lodging area of ​​wheat.

Description

Technical Field

[0001] This invention relates to the fields of unmanned aerial vehicle (UAV) remote sensing and deep learning technologies, and in particular to a method for extracting the lodging area of ​​wheat with multiple growth stages. Background Technology

[0002] As one of the world's three major cereal crops, wheat has the largest planting area, highest yield, and widest distribution, serving as a primary source of starch and energy for humankind. However, lodging is a common phenomenon during wheat growth, caused by external environmental factors (such as windstorms or hail) and the complexity of its internal structure, leading to imbalance, tilting, bending, or breaking. This damages wheat yield and quality, significantly impacting the wheat harvest. Therefore, timely and accurate monitoring of wheat lodging and the extraction of lodged area are crucial for obtaining disaster information, guiding post-disaster emergency response, estimating yield and losses, and facilitating insurance claims.

[0003] Currently, the main methods for obtaining information on wheat lodging area include manual assessment, satellite imagery analysis, and traditional machine learning. Manual assessment requires significant human and material resources, necessitating on-site surveys, measurements, and recording by professionals. This is time-consuming and labor-intensive, increasing assessment time and costs. Furthermore, it can only assess a limited sample, failing to provide a comprehensive assessment of the entire farmland or a large area, potentially leading to inaccurate and unrepresentative results. Due to the subjectivity and human factors inherent in manual assessment, discrepancies may exist between different assessors, often resulting in disputes during disaster assessment. Satellite imagery analysis is limited by weather and cloud cover, affecting data quality and availability. It is also constrained by temporal and spatial resolution. Obtaining high-quality satellite imagery may require payment or waiting for specific time windows, limiting the real-time nature and flexibility of satellite imagery for crop lodging information. Traditional machine learning methods typically require manual feature engineering, which involves manually selecting and designing suitable features to train the model, such as the shape, color, and texture of wheat plants. This process requires domain knowledge and experience, can be very time-consuming and error-prone. Furthermore, when dealing with complex and large-scale datasets, they are prone to overfitting or underfitting. These models often fail to generalize well to unseen data, limiting their effectiveness in practical applications. Summary of the Invention

[0004] To address the challenge of extracting lodging contours from wheat due to the complex field background and variations in shape and appearance across different growth stages, this invention aims to provide a method for extracting lodging area of ​​wheat at multiple growth stages. This method enables precise localization and segmentation of lodging regions and calculates the lodging area, making it suitable for efficient, real-time, and multi-growth-stage wheat lodging detection scenarios.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for extracting the lodging area of ​​wheat with multiple growth stages, the method comprising the following sequential steps:

[0006] (1) Data collection: Use drones to collect information on lodging of wheat at different growth stages at different time points, including the grain-filling stage, early maturity stage and late maturity stage;

[0007] (2) Image preprocessing: The images captured by the UAV are sequentially stitched together, labeled and cropped into blocks to obtain the preprocessed images;

[0008] (3) Data augmentation: Perform geometric transformation, noise addition, image filtering and brightness / contrast adjustment on the preprocessed image to obtain the data-augmented image;

[0009] (4) Dataset construction: The images after data augmentation are screened, and images without overturning are removed. After removal, the images are randomly divided into training set and validation set in a ratio of 4:1.

[0010] (5) Improve the Mask2Former network model to obtain the improved Mask2Former network model. Input the training set into the improved Mask2Former network model for training. During the training process, the average intersection-union ratio and the average precision are used as evaluation indicators to select the optimal segmentation model.

[0011] (6) Calculate the lodging area of ​​wheat to obtain the actual lodging area: Input the wheat image to be detected into the optimal segmentation model, the optimal segmentation model extracts the lodging area, counts the number of pixels in the lodging area, and calculates the actual lodging area of ​​wheat based on the mapping relationship between pixels and actual area.

[0012] In step (2), the image stitching refers to stitching together UAV images, performing geometric correction by injecting ground control points, and finally stitching together high-resolution images of the entire farmland in three periods; the image annotation refers to using polygons to depict the lodging boundary lines of wheat at different growth stages, suppressing noise, labeling the lodged areas as foreground and the rest as background, and the final label information is used to describe the lodged wheat areas, providing mask labels for subsequent lodging segmentation tasks; the block cropping refers to using a 512*512 window to crop the stitched high-resolution image into blocks.

[0013] In step (3), the geometric transformation refers to rotating, translating, distorting, and mirroring the image to change its shape and position; the noise addition refers to adding random noise to the image to simulate noise and interference in a real scene; the image filtering refers to applying a filter to blur the image to simulate different image qualities and sharpness; the brightness and contrast adjustment refers to changing the appearance of the image by adjusting its brightness, contrast, and color saturation to simulate interference from field lighting factors and increase the diversity of the data.

[0014] In step (5), the improvement of the Mask2Former network model to obtain the improved Mask2Former network model specifically refers to: improving the Mask2Former network model through the Hierarchical Interaction Feature Pyramid Network (HI-FPN), and fusing the HI-FPN with the Mask2Former network model; the HI-FPN consists of four fusion and refinement modules (FRMs), where the 0th layer of the FRM corresponds to the features extracted by the backbone network, corresponding to high, medium, and low resolutions from bottom to top; the 1st layer of the FRM performs initial processing on the features, using convolutional layers, batch normalization layers, and ReLU layers to ensure that the feature maps of the three resolutions in the 0th layer have the same number of channels, providing a foundation for subsequent interactions; fusion The second layer of the FRM (Fusion Refinement Module) is an interaction layer that allows feature maps of different scales to interact. Low-resolution features are upsampled using nearest-neighbor interpolation, while high-resolution features are downsampled using average pooling. The two feature maps obtained from the low-resolution upsampling and high-resolution downsampling are then element-wise added to the mid-resolution features to achieve fusion. Simultaneously, the mid-resolution features are also upsampled and downsampled, and then element-wise added to the high- and low-resolution features respectively to obtain three scaled feature maps that have been fused and refined. The third layer of the FRM module interacts and fuses the three scaled feature maps to obtain a further refined mid-scale feature map. Finally, the fourth layer of the FRM module element-wise adds the mid-scale feature map to the label mapping of the mid-scale feature map, using a residual learning strategy to ensure that features of other scales are only used as supplements.

[0015] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, the present invention utilizes drones to collect wheat lodging data at multiple growth stages, enabling real-time monitoring of wheat lodging at different growth stages. This has extremely high practical application value, enabling real-time and rapid post-disaster detection and providing accurate quantitative data analysis support for emergency response and insurance claims. Second, the present invention improves upon the Mask2Former network model, addressing practical difficulties such as the complex field background during wheat growth and the differences in wheat shape and appearance characteristics at multiple growth stages. It introduces a hierarchical interactive feature pyramid network (HI-FPN), which can effectively capture the multi-scale features of convolutional networks in continuous space, improving the accuracy of lodging area localization and segmentation. Third, the present invention has been compared with other advanced segmentation models, and the results show that it achieves the best results in terms of mean accuracy (mAP) and mean intersection-union ratio (mIoU). This indicates that the present invention can meet the high accuracy requirements for lodging extraction in complex environments, providing reliable technical support for calculating lodging area. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention;

[0017] Figure 2 This is a schematic diagram of wheat field splicing data and mask labels for three growth stages in this invention;

[0018] Figure 3 This is a schematic diagram of data amplification in this invention;

[0019] Figure 4 Here is a structural diagram of the Mask2Former network model;

[0020] Figure 5 A framework diagram of the improved Mask2Former network model;

[0021] Figure 6 A structural diagram of the fusion refining module;

[0022] Figure 7 This is a schematic diagram showing the wheat lodging detection results for different segmentation models. Detailed Implementation

[0023] like Figure 1 As shown, a method for extracting the lodging area of ​​wheat with multiple growth stages includes the following steps in sequence:

[0024] (1) Data collection: Using drones to collect lodging information of wheat at different growth stages at different time points, including the grain-filling stage, early maturity stage and late maturity stage; The wheat lodging segmentation dataset used in this invention was collected from Lujiang County, Hefei City, Anhui Province, and DJI 4pro drones were used to collect lodging information of wheat at different growth stages at different time points.

[0025] (2) Image preprocessing: The images captured by the UAV are sequentially stitched together, labeled and cropped into blocks to obtain the preprocessed images;

[0026] (3) Data augmentation: Perform geometric transformation, noise addition, image filtering and brightness / contrast adjustment on the preprocessed image to obtain the data-augmented image;

[0027] (4) Dataset construction: The images after data augmentation are screened, and images without overturning are removed. After removal, the images are randomly divided into training set and validation set in a ratio of 4:1.

[0028] (5) Improve the Mask2Former network model to obtain the improved Mask2Former network model. Input the training set into the improved Mask2Former network model for training. During the training process, the average intersection-union ratio and the average precision are used as evaluation indicators to select the optimal segmentation model.

[0029] (6) Calculate the lodging area of ​​wheat to obtain the actual lodging area: Input the wheat image to be detected into the optimal segmentation model, the optimal segmentation model extracts the lodging area, counts the number of pixels in the lodging area, and calculates the actual lodging area of ​​wheat based on the mapping relationship between pixels and actual area.

[0030] In step (2), as Figure 2 As shown in the original image (a), the image stitching refers to stitching together UAV images using PhotoScan software, performing geometric correction by injecting ground control points, and finally stitching together high-resolution images of the entire farmland at three different stages; the image annotation refers to using Labelme software to depict the lodging boundary lines of wheat at different growth stages using polygons, suppressing noise, labeling the lodged areas as foreground and the rest as background, and finally obtaining label information to describe the lodged wheat areas, providing mask labels for subsequent lodging segmentation tasks, such as... Figure 2As shown in mask (b) in the figure; the block cropping refers to using a 512*512 window to crop the stitched high-resolution image into blocks. To ensure data availability, processing large-scale image data requires preserving the detailed features of lodged wheat without loss while reducing GPU resource consumption. Therefore, a 512*512 window is used to crop the stitched high-resolution image into blocks, and each window image can be processed independently, reducing computational load and memory requirements.

[0031] like Figure 3 As shown, since many of the cropped window images do not exhibit the folding phenomenon, in order to improve the robustness and generalization of the model and avoid overfitting, this invention performs a series of transformations and expansion operations on the folded window images and mask labels to generate more diverse training data. In step (3), the geometric transformation refers to rotating, translating, distorting, and mirroring the image to change its shape and position and increase the diversity of the data; the noise addition refers to adding random noise to the image to simulate noise and interference in real scenes and improve the robustness of the model; the image filtering refers to applying a filter to blur the image to simulate different image qualities and sharpness; the brightness and contrast adjustment refers to changing the appearance of the image by adjusting its brightness, contrast, and color saturation to simulate interference from field lighting factors and increase the diversity of the data.

[0032] After filtering the window data after segmentation and cropping, data without lodging were removed, resulting in 207 images of wheat lodging at three different growth stages. After amplification, the amount of wheat lodging data reached 2070 images. The training set and validation set were randomly divided at a ratio of 4:1 and then fed into the model for training.

[0033] like Figure 4 As shown, the Mask2Former network model is a unified segmentation architecture based on Transformer, demonstrating excellent performance in semantic, instance, and panoptic segmentation tasks. Similar to Mask R-CNN, this model is based on object detection methods and further performs semantic segmentation. It mainly consists of three key steps. First, Mask2Former uses a backbone network to extract image features, typically employing a convolutional neural network (CNN) to capture the feature representation of the input image. Second, the feature maps are fed into a pixel decoder, which gradually restores the low-resolution feature maps to the original image size through upsampling operations. Finally, these features are decoded and modeled by a Transformer decoder. During the segmentation process, the Transformer decoder can integrate global contextual information into pixel-level predictions to improve segmentation accuracy. This method can model global relationships between pixels, making it very effective for handling long-distance contextual information and complex scenes.

[0034] like Figure 5 As shown, in step (5), the improvement of the Mask2Former network model to obtain the improved Mask2Former network model specifically refers to: improving the Mask2Former network model through the Hierarchical Interaction Feature Pyramid Network (HI-FPN), and fusing the HI-FPN with the Mask2Former network model; the HI-FPN consists of four fusion and refinement modules (FRM), such as... Figure 6 As shown, layer 0 of the Fusion Refinement Module (FRM) corresponds to the features extracted by the backbone network, representing high, medium, and low resolutions from bottom to top, respectively. Layer 1 of the FRM performs initial feature processing, using convolutional layers, batch normalization, and ReLU layers to ensure that the feature maps of the three resolutions in layer 0 have the same number of channels, providing a foundation for subsequent interactions. Layer 2 of the FRM is the interaction layer, allowing feature maps of different scales to interact. Low-resolution features are upsampled using nearest neighbor interpolation, and high-resolution features are downsampled using average pooling. The results obtained from the low-resolution upsampling and high-resolution downsampling are then combined. The two feature maps are element-wise added to the medium-resolution feature to achieve fusion. At the same time, the medium-resolution feature is also upsampled and downsampled, and then element-wise added to the high-resolution and low-resolution features respectively to obtain three scale feature maps that have been fused and refined. In the third layer of the fusion and refinement module FRM, the three scale feature maps are interactively fused to obtain a further fused and refined medium-scale feature map. Finally, in the fourth layer of the fusion and refinement module FRM, the medium-scale feature map is element-wise added to the label mapping of the medium-scale feature map. A residual learning strategy is used to ensure that the features of other scales are only used as supplements and that more critical feature information is captured and output.

[0035] To address the complex field conditions of wheat growth and the practical challenges posed by the differences in shape and appearance characteristics among wheat at various growth stages, a hierarchical interactive feature pyramid network (HI-FPN) is adaptively proposed to improve the Mask2Former network model. A feature fusion mechanism between adjacent layers is employed, allowing multi-scale feature interactions. This overcomes the problems of high-level semantic information being diluted and low-level spatial information failing to influence high-level information, thus improving the model's accuracy and robustness. Furthermore, the use of a residual learning strategy enables the model to better utilize features at other scales as supplementary elements, thereby more accurately capturing key feature information and enhancing model stability.

[0036] The training, validation, and test sets were divided into groups of 8 images each (batch size), and batch training was performed using this method. The validation set was used to optimize hyperparameters, with an initial learning rate (lr) of 0.001 and a weight decay rate of 1e.-4 The momentum factor is 0.9 and the maximum number of iterations is 100.

[0037] The model's performance was evaluated using the mean accuracy (mAP) and mean intersection-union ratio (mIoU) metrics, and the optimal model weights were selected. mIoU represents the ratio of the area of ​​intersection to the area of ​​union between the predicted and actual values ​​for all classes, as shown in Equation (1), ranging from 0 to 1. A higher score indicates that the model's segmentation performance is closer to the actual values. mAP measures the overall accuracy performance of the model across all classes. Compared to mIoU, it places more emphasis on the localization and edge accuracy of the segmentation results, as shown in Equation (2).

[0038]

[0039]

[0040] In this context, TP represents the area correctly predicted as lodging, FP represents the area predicted as lodging but not actually lodging, and FN represents the area predicted as background but actually lodging.

[0041] This invention uses a DJI Phantom 4 Pro drone to conduct aerial photography of wheat fields at a flight altitude of 20m and a flight speed of 3m / s. At this altitude, the ground resolution is 0.5cm / pixel. Based on the lodged areas extracted by the optimal segmentation model, the number of pixels is counted, and the actual lodged area of ​​the wheat can be calculated according to the mapping relationship between pixels and actual area.

[0042] Quantitative analysis was conducted on the lodging extraction results of wheat with multiple growth stages under various thresholds using different classical models and the improved model proposed in this invention. The mean IoU (mean crossover ratio) and mean AP (mean precision) were used as evaluation metrics. As shown in Table 1, the method proposed in this invention achieved better results at all thresholds, with higher mIoU and mAP values ​​compared to the classical model. Furthermore, in... Figure 7 The comparison of model prediction results is shown in the figure. It can be observed that the segmentation prediction results of the method proposed in this invention are closest to the true values. It can not only accurately locate the lodging area of ​​wheat, but also form clear boundaries. This advantage is due to the interaction and fusion of domain features of HI-FPN, which makes it show high application potential in lodging detection of wheat at different growth stages.

[0043] Table 1 Comparison of mIoU and mAP for different segmentation models at different thresholds.

[0044]

[0045]

[0046] In summary, this invention utilizes drones to collect wheat lodging data at different growth stages and processes the data using techniques such as stitching and cropping. Simultaneously, data augmentation techniques are applied to simulate complex natural environments in the field, improving the model's robustness and generalization. Furthermore, the introduction of the Hierarchical Interactive Feature Pyramid Network (HI-FPN) improves the Mask2Former network model, enabling multi-scale feature extraction and fusion, thereby enhancing the accuracy of lodging area segmentation and allowing for precise calculation of wheat lodging area. This invention has practical application significance, suitable for efficient, real-time, and multi-growth-stage wheat lodging detection scenarios, and can provide quantitative data analysis and technical support for guiding post-disaster emergency response and insurance claims.

Claims

1. A method for extracting the lodging area of ​​wheat with multiple growth stages, characterized in that: The method includes the following steps in sequence: (1) Data collection: Use drones to collect information on lodging of wheat at different growth stages at different time points, including the grain-filling stage, early maturity stage and late maturity stage; (2) Image preprocessing: The images collected by the UAV are sequentially stitched together, labeled and cropped into blocks to obtain the preprocessed images; (3) Data augmentation: Perform geometric transformation, noise addition, image filtering and brightness / contrast adjustment on the preprocessed image to obtain the data augmented image; (4) Dataset construction: The images after data augmentation are screened, and images without overturning are removed. After removal, the images are randomly divided into training set and validation set in a ratio of 4:

1. (5) Improve the Mask2Former network model to obtain the improved Mask2Former network model. Input the training set into the improved Mask2Former network model for training. During the training process, the average intersection-union ratio and the average precision are used as evaluation indicators to select the optimal segmentation model. (6) Calculate the lodging area of ​​wheat to obtain the actual lodging area: Input the wheat image to be detected into the optimal segmentation model, the optimal segmentation model extracts the lodging area, counts the number of pixels in the lodging area, and calculates the actual lodging area of ​​wheat based on the mapping relationship between pixels and actual area; In step (5), the improvement of the Mask2Former network model to obtain the improved Mask2Former network model specifically refers to: improving the Mask2Former network model through the Hierarchical Interaction Feature Pyramid Network (HI-FPN), and fusing the HI-FPN with the Mask2Former network model; the HI-FPN consists of four fusion and refinement modules (FRMs), where the 0th layer of the fusion and refinement module corresponds to the features extracted by the backbone network, and from bottom to top, they correspond to high, medium, and low resolutions respectively; the 1st layer of the fusion and refinement module performs initial processing on the features, using convolutional layers, batch normalization layers, and ReLU layers to ensure that the feature maps of the three resolutions in the 0th layer have the same number of channels, providing a foundation for subsequent interactions; fusion The second layer of the FRM (Fusion Refinement Module) is an interaction layer that allows feature maps of different scales to interact. Low-resolution features are upsampled using nearest-neighbor interpolation, while high-resolution features are downsampled using average pooling. The two feature maps obtained from the low-resolution upsampling and high-resolution downsampling are then element-wise added to the mid-resolution features to achieve fusion. Simultaneously, the mid-resolution features are also upsampled and downsampled, and then element-wise added to the high- and low-resolution features respectively to obtain three scaled feature maps that have been fused and refined. The third layer of the FRM module interacts and fuses the three scaled feature maps to obtain a further refined mid-scale feature map. Finally, the fourth layer of the FRM module element-wise adds the mid-scale feature map to the label mapping of the mid-scale feature map, using a residual learning strategy to ensure that features of other scales are only used as supplements.

2. The method for extracting lodging area of ​​multi-stage wheat according to claim 1, characterized in that: In step (2), the image stitching refers to stitching together UAV images, performing geometric correction by injecting ground control points, and finally stitching together high-resolution images of the entire farmland in three periods; the image annotation refers to using polygons to depict the lodging boundary lines of wheat at different growth stages, suppressing noise, labeling the lodged areas as foreground and the rest as background, and finally obtaining label information to describe the lodged wheat areas, providing mask labels for subsequent lodging segmentation tasks; the block cropping refers to using a 512*512 window to crop the stitched high-resolution image into blocks.

3. The method for extracting lodging area of ​​multi-stage wheat according to claim 1, characterized in that: In step (3), the geometric transformation refers to rotating, translating, distorting and mirroring the image to change its shape and position; the noise addition refers to adding random noise to the image to simulate noise and interference in a real scene; the image filtering refers to applying a filter to blur the image to simulate different image qualities and sharpness; the brightness and contrast adjustment refers to changing the appearance of the image by adjusting its brightness, contrast and color saturation to simulate interference from field lighting factors and increase the diversity of the data.

Images