A method and system for farmland segmentation and crop classification based on deep learning of multi-source remote sensing data
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-18
- Publication Date
- 2026-08-14
AI Technical Summary
[0003]·单一数据源易受时空分辨率不可兼得的技术限制，难以兼顾农田分割的高空间分辨率和作物分类的高时间分辨率需求
[0043]1、多源融合提升精度：融合了不同时空分辨率多源遥感数据，有效克服了单一数据源在地块破碎、种植复杂场景下的局限性，同步提升了地块分割与作物分类的精度。
Smart Images
Figure CN120808168B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image interpretation and intelligent agricultural monitoring technology, and more specifically to a method and system for farmland segmentation and crop classification based on deep learning of multi-source remote sensing data. Background Technology
[0002] Currently, remote sensing technology has been widely applied to crop classification and plot segmentation, especially the fusion of multi-source and multi-temporal remote sensing data, which has gradually become a research hotspot. Traditional methods struggle to achieve crop classification at the farmland scale, and in complex plots and mixed crop environments, the accuracy of plot segmentation and crop classification is limited. In recent years, the introduction of deep learning methods has significantly improved the accuracy of farmland plot segmentation and crop planting classification, but existing methods still have shortcomings in the following aspects:
[0003] • A single data source is subject to the technical limitation that spatiotemporal resolution cannot be simultaneously achieved, making it difficult to meet the requirements of high spatial resolution for farmland segmentation and high temporal resolution for crop classification.
[0004] • Insufficient fusion of features from data with different spatiotemporal resolutions makes it difficult to fully leverage the complementary advantages of different remote sensing data.
[0005] • The precision of land parcel segmentation and instance recognition needs to be improved, as issues such as blurred boundaries and overlapping land parcels are prone to occur.
[0006] Therefore, there is an urgent need to develop methods for farmland segmentation and crop classification based on the fusion of multi-source remote sensing data to meet the practical needs of precision agriculture and large-scale agricultural monitoring. Summary of the Invention
[0007] To overcome the aforementioned shortcomings of existing technologies, this invention provides a method and system for farmland segmentation and crop classification based on deep learning of multi-source remote sensing data.
[0008] The technical solution of this invention is as follows:
[0009] This invention provides a method for farmland segmentation and crop classification based on deep learning from multi-source remote sensing data, comprising the following steps:
[0010] Step 1: Remote sensing data acquisition and preprocessing;
[0011] Step 2: Construct the U-TMMNet spatiotemporal feature encoding network;
[0012] Step 3: Semantic segmentation;
[0013] Step 4: Panoramic Segmentation;
[0014] Step 5: Accuracy evaluation and model optimization.
[0015] Furthermore, step one, remote sensing data acquisition and preprocessing, specifically includes:
[0016] 1.1 Acquire multi-source remote sensing data of the target area, including:
[0017] Low-resolution multispectral temporal images; high-resolution optical images of a single scene.
[0018] 1.2 The following preprocessing steps were performed on each source image:
[0019] Image radiometric normalization and atmospheric correction; band resampling and uniformity; spatial registration (ensuring pixel alignment); cloud removal and noise reduction and spatial mask generation.
[0020] Furthermore, step two specifically includes:
[0021] 2.1 We propose to construct the U-TMMNet spatiotemporal feature encoding network based on the U-Net architecture and the lightweight temporal attention mechanism (L-TAE) to realize a spatiotemporal feature fusion mechanism driven by spatial details.
[0022] 2.2 Input the preprocessed images from each source into the U-TMMNet model according to their data types: 1) For low-resolution temporal images, use the U-TMMNet spatiotemporal encoder to perform multi-scale convolution, sequence modeling, and temporal attention feature extraction to automatically obtain rich temporal, spectral, and spatial information. 2) For single-scene high-resolution images, extract fine-grained spatial texture information through the spatial feature branch of U-TMMNet.
[0023] 2.4 Multi-source features are spliced and fused in the model encoder through feature channel dimensions to obtain a unified spatiotemporal feature representation.
[0024] Furthermore, step three, the semantic segmentation module, specifically includes:
[0025] 3.1 Input the fused multimodal spatiotemporal features into the semantic segmentation decoder of U-TMMNet.
[0026] 3.2 A multi-scale decoding and skip connection mechanism is adopted to achieve the integration of spatial details and high-level semantic information.
[0027] 3.3 Output a pixel-level crop category probability map of the same size as the input image. Obtain pixel-level semantic segmentation results through softmax, that is, assign a unique crop type label to each pixel to achieve unified classification of plots and their internal pixels.
[0028] 3.4 The class discrimination ability and spatial consistency of the model are evaluated using metrics such as mIoU and OA, and the semantic segmentation performance of different data combinations is compared.
[0029] Furthermore, step four, the panoramic segmentation module, specifically includes:
[0030] 4.1 The deep fusion features of U-TMMNet are input into the panoramic segmentation module and processed as follows: 1) Plot center point detection: The spatial center point of each crop plot is detected by heatmap regression or center point prediction network; 2) Bounding box and instance mask regression: The boundary branch of panoramic segmentation is used to predict the plot bounding box parameters and the corresponding pixel-level instance mask; 3) Plot semantic classification: The main crop type of the plot is determined by combining the center point position and the plot instance features, thus achieving instance-level semantic segmentation.
[0031] 4.2 Finally, the pixel-level mask and unique crop type label of each plot are output to realize the instance segmentation and classification of crop plots.
[0032] Furthermore, step five, accuracy evaluation and model optimization, specifically includes:
[0033] 5.1 The segmentation results are systematically evaluated using metrics such as confusion matrix, overall accuracy (OA), Kappa coefficient, F1-score, mIoU (mean Intersection over Union), panoptic quality (PQ), segmentation quality (SQ), and recognition quality (RQ).
[0034] 5.2 Based on evaluation metrics and ground truth data, we adjust the U-TMMNet parameters, feature fusion method, and panoramic segmentation module structure to improve the model's robustness, generalization ability, and segmentation and classification accuracy.
[0035] 5.3 Support the promotion and application in different regions, different crop types and complex plot structures.
[0036] Another aspect of the present invention provides a farmland segmentation and crop classification system based on deep learning of multi-source remote sensing data, the system comprising:
[0037] The remote sensing data acquisition and preprocessing module is used to acquire multi-source remote sensing data of the target area and perform the following preprocessing on the multi-source images;
[0038] We propose and construct a U-TMMNet spatiotemporal feature coding network module, which takes multi-source preprocessed images as input and feeds them into different coding branches of the U-TMMNet model. In the fusion layer of the model, we perform high-order late-stage fusion of deep features of each modality by channel splicing to generate a unified high-dimensional feature representation.
[0039] The semantic segmentation module is used to input the fused multimodal spatiotemporal features into the U-TMMNet decoder. It adopts a U-shaped structure and a skip connection mechanism to achieve joint decoding of spatial details and global features. Finally, it outputs a crop pixel probability map of the same size as the original image. The crop type label of each pixel is obtained through softmax, realizing pixel-level crop semantic segmentation.
[0040] The panorama segmentation module is used for panorama segmentation. It is based on the deep fusion features output by U-TMMNet and input into the panorama segmentation module.
[0041] The accuracy evaluation and model optimization module is used to train the model end-to-end using remote sensing sample data with real crop category and plot instance labels, and loss functions such as cross-entropy and Focal Loss. Through multiple ablation and comparison experiments, the performance of different data combinations (Sentinel-2, Sentinel-2+Planet) and different segmentation strategies (semantic segmentation / panoramic segmentation) is compared to optimize the model parameters. The segmentation and classification results are quantitatively evaluated using multiple indicators such as mIoU, OA (Overall Accuracy), PQ (Panoptic Quality), SQ (Segmentation Quality), and RQ (Recognition Quality) to comprehensively verify the accuracy and applicability of the method.
[0042] The technical effects and advantages of this invention are as follows:
[0043] 1. Multi-source fusion improves accuracy: It integrates remote sensing data from multiple sources with different spatiotemporal resolutions, effectively overcoming the limitations of a single data source in fragmented plots and complex planting scenarios, and simultaneously improving the accuracy of plot segmentation and crop classification.
[0044] 2. Spatiotemporal feature encoding: The U-TMMNet model is proposed to achieve deep fusion of multimodal spatiotemporal features, capture crop growth dynamics and multi-source complementary information, and adapt to complex agricultural landscapes.
[0045] 3. Integrated semantic and panoramic segmentation: It not only supports high-precision pixel-level semantic segmentation, but also enables plot-level instance segmentation, ensuring segmentation integrity and spatial coherence.
[0046] 4. Strong generalization and practicality: The model has good generalization ability for different regions, different crops and complex climatic environments, and is suitable for large-scale promotion and application.
[0047] 5. Superior system performance: Significantly improved segmentation and classification indicators, resulting in higher engineering application value and practical effects. Attached Figure Description
[0048] Figure 1 This is an overall flowchart of an example of the present invention;
[0049] Figure 2 This is a flowchart of the method.
[0050] Figure 3 This is a schematic diagram of a semantic segmentation task;
[0051] Figure 4 This is a comparison chart of semantic segmentation results;
[0052] Figure 5 This is a comparison image of the panoramic segmentation results. Detailed Implementation
[0053] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0054] Example 1
[0055] like Figure 1 and Figure 2 The method shown is a farmland segmentation and crop classification method based on deep learning of multi-source remote sensing data. The method includes the following steps:
[0056] Step 1: Acquisition and Preprocessing of Multi-Source Remote Sensing Image Data
[0057] 1.1 Acquire multi-source remote sensing data of the target area, including:
[0058] Sentinel-2 multispectral image time-series data;
[0059] PlanetScope high-resolution single- or multi-temporal optical images.
[0060] 1.2 The above multi-source images are preprocessed as follows:
[0061] Perform radiation normalization, atmospheric correction, and band consistency;
[0062] Pixel alignment of images from different data sources and at different times is achieved through geometric correction and spatial registration;
[0063] Cloud removal processing is performed on abnormal bands such as those with cloud cover to eliminate contaminated data phases, and finally a unified multi-source remote sensing dataset is constructed.
[0064] 1.3 All images were cropped to the study area according to their spatial extent, and the spatial resolution was standardized (e.g., 10 meters). They were then archived as a four-dimensional stacked structure (time × band × space) to prepare for subsequent model input.
[0065] Step 2: Spatiotemporal Feature Encoding
[0066] 2.1 Using multi-source preprocessed images as input, they are fed into different coding branches of the U-TMMNet model:
[0067] 1) For Sentinel-2 time series data, spatiotemporal convolution and lightweight multi-head temporal attention mechanisms (such as L-TAE) are used to encode crop growth dynamics, temporal variations and spatial structure features;
[0068] 2) For PlanetScope high-resolution images, only spatial convolution is used to extract rich spatial texture and plot details.
[0069] 2.2 In the fusion layer of the model, the deep features of each modality are fused in a high-order late stage by channel splicing to generate a unified high-dimensional feature representation.
[0070] Step 3: Semantic Segmentation and Panoptic Segmentation
[0071] 3.1 Semantic segmentation task:
[0072] The fused multimodal spatiotemporal features are input into the U-TMMNet decoder, which uses a U-shaped structure and a skip connection mechanism to achieve joint decoding of spatial details and global features.
[0073] The final output is a crop pixel probability map of the same size as the original image. The crop type label for each pixel is obtained through softmax, achieving pixel-level crop semantic segmentation (e.g., ...). Figure 3 (As shown).
[0074] 3.2 Panoramic Segmentation Task:
[0075] Based on the deep fusion features output by U-TMMNet, input them into the panorama segmentation module:
[0076] First, predict the location of the crop plot's center point by detecting the center point branch;
[0077] Then, the bounding box regression branch predicts the parcel extent and instance mask;
[0078] Simultaneously, the main crop type is determined by combining the multimodal features within the plot, and a unique label is output for each plot.
[0079] Step 4: Model Training and Evaluation
[0080] 4.1 Using remote sensing sample data with real crop categories and plot instance labels, the model is trained end-to-end using loss functions such as cross-entropy and FocalLoss.
[0081] 4.2 Through multiple ablation and comparison experiments, the performance of different data combinations (Sentinel-2, Sentinel-2+Planet) and different segmentation strategies (semantic segmentation / panoptical segmentation) was compared, and the model parameters were optimized.
[0082] 4.3 The segmentation and classification results are quantitatively evaluated using multiple indicators such as mIoU, OA (Overall Accuracy), PQ (Panoptic Quality), SQ (Segmentation Quality), and RQ (Recognition Quality) to comprehensively verify the accuracy and applicability of this method.
[0083] Example 2
[0084] A farmland segmentation and crop classification system based on deep learning of multi-source remote sensing data is used to run the above-mentioned method. The system includes:
[0085] The remote sensing data acquisition and preprocessing module is used to acquire multi-source remote sensing data of the target area and perform the following preprocessing on the multi-source images;
[0086] A U-TMMNet spatiotemporal feature coding network module is constructed to take multi-source preprocessed images as input and feed them into different coding branches of the U-TMMNet model. In the fusion layer of the model, the deep features of each modality are fused in a high-order late stage by channel splicing to generate a unified high-dimensional feature representation.
[0087] The semantic segmentation module is used to input the fused multimodal spatiotemporal features into the U-TMMNet decoder. It adopts a U-shaped structure and a skip connection mechanism to achieve joint decoding of spatial details and global features. Finally, it outputs a crop pixel probability map of the same size as the original image. The crop type label of each pixel is obtained through softmax, realizing pixel-level crop semantic segmentation.
[0088] The panorama segmentation module is used to input the deep fusion features output by U-TMMNet into the panorama segmentation module;
[0089] The accuracy evaluation and model optimization module is used to train the model end-to-end using remote sensing sample data with real crop category and plot instance labels, and loss functions such as cross-entropy and Focal Loss. Through multiple ablation and comparison experiments, the performance of different data combinations (Sentinel-2, Sentinel-2+Planet) and different segmentation strategies (semantic segmentation / panoramic segmentation) is compared to optimize the model parameters. The segmentation and classification results are quantitatively evaluated using multiple indicators such as mIoU, OA (Overall Accuracy), PQ (Panoptic Quality), SQ (Segmentation Quality), and RQ (Recognition Quality) to comprehensively verify the accuracy and applicability of the method.
[0090] Example 3
[0091] This invention first acquires and preprocesses multi-source remote sensing data, including Sentinel-2 optical temporal imagery and PlanetScope high-resolution imagery. Through spatial registration, band consistency, and normalization, a unified spatiotemporal remote sensing dataset is constructed. Subsequently, the proposed U-TMMNet (U-Net with Temporal Attention Encoder for Multi-Model data Fusion) spatiotemporal coding network is used to extract and fuse deep features from data sources with different spatiotemporal resolutions. Multi-source data features are concatenated in a high-order dimension, fully leveraging the advantages of each data source in terms of spatiotemporal resolution and information complementarity. Furthermore, by integrating a semantic segmentation decoding network and a panoramic segmentation module, crop classification is achieved based on pixel-level farmland semantic segmentation and land parcel segmentation, automatically outputting a spatial mask and a unique crop type label for each parcel. This method innovatively utilizes the complementarity of multi-source remote sensing data with different spatiotemporal resolutions, organically combining efficient spatiotemporal feature encoding with crop classification, significantly improving the ability to identify parcel boundaries and the accuracy of classification in complex agricultural landscapes. The effectiveness and applicability of this method were verified using major agricultural production areas in France and the middle and lower reaches of the Yangtze River in China as examples, and it has broad application prospects for promotion to other regions and multiple crop scenarios.
[0092] Specific applications and promotion are as follows:
[0093] 1. This invention is applicable to the high-precision segmentation and classification of farmland plots in various crop planting areas, and can be used as input data for applications such as growth monitoring, yield estimation, and land resource management.
[0094] 2. Through real-world examples (such as multi-source remote sensing data from major agricultural production areas in France in 2020), the superior performance of this method is fully demonstrated in scenarios such as complex landscapes, multiple crop distributions, and fragmented land parcels.
[0095] 3. This invention can also be flexibly expanded according to different regions, crop types and remote sensing data conditions to provide standardized solutions for different agricultural remote sensing application scenarios.
[0096] Example 4
[0097] Result description:
[0098] This invention constructs U-TMMNet semantic segmentation and U-TMMNet+panoramic segmentation models, fusing Sentinel-2 + Planet data. The overall semantic segmentation accuracy is higher than that of the model using only Sentinel-2 data; the geometric accuracy (RQ) of panoramic segmentation reaches 56.7%, an increase of 3.6% compared to the model using only Sentinel-2 data, and the overall accuracy (PQ) reaches 46.5%, also a significant improvement. The results show that after fusing the spatial information of the high-resolution Planet data, the model provides detailed information on crop boundaries, which helps improve the geometric accuracy of plot instance segmentation, thereby simultaneously improving the crop classification accuracy.
[0099] Table 1: Comparison of Semantic Segmentation Results Figure 4 Comparison of semantic segmentation results
[0100] data Macro average crossover ratio Micro-average crossover ratio accuracy Sentinel-2 (S2) 61.5% 70.7% 82.9% Sentinel-2+Plannet (S2P) 62.3% 71.3% 83.3%
[0101] Table 2: Comparison of Panoramic Segmentation Results Figure 5 Comparison of panoramic segmentation results
[0102] data SQ RQ PQ Sentinel-2 (S2) 81.5 53.1% 43.7% Sentinel-2+Plannet (S2P) 81.2 56.7% 46.5%
[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for farmland segmentation and crop classification based on deep learning of multi-source remote sensing data, characterized in that: Includes the following steps: Step 1: Remote sensing data acquisition and preprocessing; Step 2: Construct the U-TMMNet spatiotemporal feature encoding network; Step 3: Semantic segmentation; Step 4: Panoramic Segmentation; Step 5: Accuracy Evaluation and Model Optimization; Step two, constructing the U-TMMNet spatiotemporal feature encoding network, specifically includes: 2.1 Based on the U-Net architecture, a lightweight temporal attention mechanism (L-TAE) and a spatial detail-driven spatiotemporal feature fusion mechanism, we propose and construct the U-TMMNet (U-Net with Temporal Attention Encoder for Multi-Model data Fusion) spatiotemporal feature encoding network to achieve feature fusion of data with different spatiotemporal resolutions. 2.2 Input the preprocessed images from each source into the U-TMMNet model according to their data types: 1) For low-resolution temporal images, use the U-TMMNet spatiotemporal encoder to perform multi-scale convolution, sequence modeling and temporal attention feature extraction to automatically obtain rich temporal, spectral and spatial information; 2) For high-resolution images of a single scene, extract fine-grained spatial texture information through the spatial feature branch of U-TMMNet. 2.4 Multi-source features are spliced and fused at the deep layer of the model encoder through feature channel dimension to obtain a unified spatiotemporal feature representation; Step three, the semantic segmentation module, specifically includes: 3.1 Input the fused multimodal spatiotemporal features into the semantic segmentation decoder of U-TMMNet; 3.2 A multi-scale decoding and skip connection mechanism is adopted to achieve the integration of spatial details and high-level semantic information; 3.3 Output and input image size pixel-level crop category probability map, obtain pixel-level semantic segmentation results through softmax, that is, assign a unique crop type label to each pixel to achieve unified classification of plots and their internal pixels; 3.4 The crop classification ability and spatial consistency of the model are evaluated using mIoU and OA indices, and the semantic segmentation performance of different data combinations is compared. Step four, the panoramic segmentation module, specifically includes: 4.1 The deep fusion features of U-TMMNet are input into the panoramic segmentation module and processed as follows: 1) Plot center point detection: The spatial center point of each crop plot is detected by heatmap regression or center point prediction network; 2) Bounding box and instance mask regression: The boundary branch of panoramic segmentation is used to predict the plot bounding box parameters and the corresponding pixel-level instance mask; 3) Plot semantic classification: The main crop type of the plot is determined by combining the center point position and the plot instance features, thus achieving instance-level semantic segmentation. 4.2 Finally, the pixel-level mask and unique crop type label of each plot are output to realize panoramic segmentation and classification of crop plots.
2. The method for farmland segmentation and crop classification based on deep learning of multi-source remote sensing data according to claim 1, characterized in that: Step one, remote sensing data acquisition and preprocessing, specifically includes: 1.1 Acquire multi-source remote sensing data of the target area, including: Low-resolution multispectral temporal images; single-scene high-resolution optical images; 1.2 The following preprocessing steps were performed on each source image: Image radiometric normalization and atmospheric correction; band resampling and homogenization; spatial registration (ensuring pixel alignment); cloud and noise removal and spatial mask generation; time archiving of all temporal images according to a unified time reference.
3. The method for farmland segmentation and crop classification based on deep learning of multi-source remote sensing data according to claim 1, characterized in that: Step five, accuracy evaluation and model optimization, specifically includes: 5.1 The segmentation results were systematically evaluated using confusion matrix, overall accuracy (OA), Kappa coefficient, F1-score, mIoU (mean Intersection over Union), panoptic segmentation quality (PQ), segmentation quality (SQ), and recognition quality (RQ). 5.2 Based on evaluation metrics and ground truth data, we adjust the U-TMMNet parameters, feature fusion method, and panoramic segmentation module structure to improve the model's robustness, generalization ability, and segmentation and classification accuracy. 5.3 Support the promotion and application in different regions, different crop types and complex plot structures.
4. A farmland segmentation and crop classification system based on deep learning of multi-source remote sensing data, characterized in that: The system for performing the method according to any one of claims 1-3 comprises: The remote sensing data acquisition and preprocessing module is used to acquire multi-source remote sensing data of the target area and perform the following preprocessing on the multi-source images; A U-TMMNet spatiotemporal feature coding network module is constructed to take multi-source preprocessed images as input and feed them into different coding branches of the U-TMMNet model. In the fusion layer of the model, the deep features of each modality are fused in a high-order late stage by channel splicing to generate a unified high-dimensional feature representation. The semantic segmentation module is used to input the fused multimodal spatiotemporal features into the U-TMMNet decoder. It adopts a U-shaped structure and a skip connection mechanism to achieve joint decoding of spatial details and global features. Finally, it outputs a crop pixel probability map of the same size as the original image. The crop type label of each pixel is obtained through softmax, realizing pixel-level crop semantic segmentation. The panorama segmentation module is used to input the deep fusion features output by U-TMMNet into the panorama segmentation module; The accuracy evaluation and model optimization module uses remote sensing sample data with real crop categories and plot instance labels to perform end-to-end training of the model using cross-entropy and Focal Loss loss functions. Through multiple ablation and comparison experiments, the performance of different data combinations and different segmentation strategies (semantic segmentation / panoptical segmentation) is compared to optimize the model parameters. The segmentation and classification results are quantitatively evaluated using mIoU, OA (Overall Accuracy), PQ (Panoptic Quality), SQ (Segmentation Quality), and RQ (Recognition Quality) indicators to comprehensively verify the accuracy and applicability of the proposed method.