A method and system for breeding field instance segmentation by fusing uav orthoimage and digital surface model

Abstract

The application discloses a breeding field block instance segmentation method fusing unmanned aerial vehicle orthographic image and digital surface model. The method acquires unmanned aerial vehicle RGB orthographic image and digital surface model (DSM) of a breeding test field block; a double-flow instance segmentation network fusing an RGB branch and a DSM branch is constructed, and multi-level features of two modes are extracted respectively; cross-modal hierarchical fusion of two modal features is carried out through hierarchical bimodal fusion, multi-scale space-channel attention and boundary attention based on modal gradient, so that fusion features complementary to spectral texture information and crown height structure information are obtained; and finally, an instance segmentation mask of each field block is output by a query-based instance decoder. The application converts image segmentation into three-dimensional-based field block boundary structure feature recognition by using crown height structure information, and can effectively separate adjacent and adhered breeding field blocks under complex conditions such as crown layer sealing, so that the field block segmentation precision and boundary recognition quality are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of agricultural remote sensing and image processing technology, and in particular to a method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models. Background Technology

[0002] Crop breeding trials are typically conducted in the field in the form of numerous side-by-side breeding plots (microplots), with each plot containing one genotype or treatment. High-throughput field phenotypic analysis requires repeated and objective observations of the breeding population at the plot level. Accurately extracting the spatial boundaries of each breeding plot is a prerequisite for reliably extracting plot-level phenotypic parameters (such as canopy cover, plant height, and canopy volume). Due to its high spatial resolution, flexible observation time, and relatively low cost, UAV remote sensing has become an important means of acquiring field phenotypic data. Therefore, automatic segmentation of breeding plots is not a dispensable preprocessing step, but a crucial link in the UAV high-throughput phenotypic identification process.

[0003] Existing automatic segmentation methods for breeding fields or field plots mainly rely on UAV RGB orthophotos. For example: one type of method uses a fully convolutional neural network to perform semantic segmentation on RGB images, and then extracts field boundaries through geometric post-processing such as image thinning and Hough line detection (e.g., publication number CN116469019A); another type of method uses a rotating target detection instance segmentation network with an attention mechanism to adapt to the rotational arrangement of field plots relative to the image coordinate axes (e.g., publication number CN120107599A); yet another type of method introduces time-series information and prior knowledge of planting design, and segments individual fields based on canopy semantic segmentation results through traction line aggregation (e.g., publication number CN120107587A). In addition, there are also multi-source remote sensing segmentation methods for farmland plots, but these typically fuse optical images with different spatiotemporal resolutions (e.g., low-resolution multispectral time-series images and single-scene high-resolution optical images) for farmland plot segmentation and crop classification (e.g., publication number CN120808168A).

[0004] The objective problem with the aforementioned existing technology is that its determination of field boundaries mainly relies on the spectral and texture information of RGB images. In the early and middle stages of crop growth, bare soil and gaps between rows are clearly visible within the field, and RGB images provide sufficient spectral and texture contrast, making field boundaries relatively easy to identify. However, in the later stages of growth, as the canopy gradually closes (i.e., row closure), canopy adhesion occurs between adjacent fields, and the gaps between fields gradually disappear in the two-dimensional image. The spectrum tends to saturate, and the texture becomes more uniform. Combined with the effects of weed invasion, canopy shading, and changes in light intensity, the visual boundaries between adjacent fields become severely blurred. At this point, models relying solely on RGB information are prone to mis-segmenting and merging multiple adjacent fields into a single entity, leading to boundary localization failure and further contaminating the extraction of downstream field-level phenotypic parameters, thus limiting its practicality in multi-environment breeding experiments.

[0005] Digital Surface Models (DSMs), as a byproduct of UAV photogrammetry in generating orthophotos, carry information about the vertical structure of the canopy: even if adjacent fields are highly similar in color and texture, differences in canopy height and discontinuities in local surface structure still exist at their boundaries. However, DSMs reconstructed from UAV photogrammetry are not ideal active depth data, as they are affected by various uncertainties such as structural blurring caused by wind-induced canopy swaying, reconstruction errors in areas with insufficient texture, and local voids. Meanwhile, RGB images are also affected by shadows, changes in illumination, and weeds. Therefore, while there is a complementary relationship between RGB and DSM, there are also differences and inconsistencies in local data quality. Simply using DSM as an additional channel for early stitching or later fusion with RGB, while introducing height information, easily introduces noise specific to each modality into the feature learning process, thus weakening the stability of boundary discrimination. How to stably and effectively fuse RGB images and DSMs under complex field conditions to achieve accurate instance segmentation of breeding fields remains an unsolved problem in current technology. Summary of the Invention

[0006] To address the problems of existing breeding field segmentation methods that rely solely on RGB spectral texture information, are prone to merging adjacent fields due to adhesion under complex field conditions such as canopy closure, and suffer from boundary positioning failure, this invention provides a breeding field instance segmentation method that integrates UAV orthophotos and digital surface models. By stably introducing and utilizing canopy height structure information through dual-stream cross-modal fusion, the segmentation task, which originally relied on two-dimensional color differences, is transformed into a boundary recognition task based on three-dimensional canopy structure, thereby improving the accuracy and boundary quality of breeding field instance segmentation under complex conditions.

[0007] Technical solution:

[0008] This invention first discloses a method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models, comprising the following steps:

[0009] S1. Obtain UAV RGB orthophotos and digital surface models of the same breeding test area. The RGB orthophotos and digital surface models have the same spatial reference system and are aligned at the pixel level. The aligned RGB orthophotos and digital surface models are divided into corresponding RGB image blocks and digital surface model image blocks.

[0010] S2. Construct a dual-stream instance segmentation network, which includes an RGB encoding branch and a digital surface model encoding branch with mutually independent parameters. Use the RGB encoding branch and the digital surface model encoding branch to perform multi-level feature extraction on the RGB image block and the digital surface model image block respectively, and obtain RGB multi-level features and high-level multi-level features accordingly.

[0011] S3. Perform cross-modal fusion on the RGB multi-level features and the height multi-level features to obtain multi-level fused features that fuse RGB spectral texture information and digital surface model canopy height structure information at each level;

[0012] S4. Input the multi-level fusion features into the instance decoder to predict and output the instance segmentation mask for each breeding field in the breeding test area.

[0013] Preferably, step S1, the pixel-level alignment and segmentation, includes: unifying the RGB orthophoto and the digital surface model to the same spatial reference system and the same ground sampling distance; truncating the digital surface model numerically according to a set percentile, and normalizing each segmented digital surface model image block separately to suppress elevation reference differences between different image blocks; segmenting the aligned data into image blocks with overlapping areas, and simultaneously applying geometric enhancements, including flipping, rotation, and scaling, to the RGB image blocks, digital surface model image blocks, and instance annotation masks, while applying photometric enhancement only to the RGB image blocks.

[0014] Preferably, in step S2, the RGB encoding branch and the digital surface model encoding branch each use a parameter-independent residual network as the backbone. The two encoding branches have the same network structure but do not share parameters. They output multiple layers of feature maps from shallow to deep layers respectively, thus obtaining the RGB multi-level features and the high-level multi-level features.

[0015] Preferably, the cross-modal fusion in step S3 includes hierarchical bimodal fusion: at multiple feature levels, the highly multi-level features are decomposed into low-frequency components representing the overall canopy structure and high-frequency components representing the field boundaries, and then fused with the local texture components of the RGB multi-level features to obtain hierarchical fusion features; the hierarchical fusion features are then fed back into the RGB encoding branch and the digital surface model encoding branch respectively, as inputs to the two encoding branches of the next level, so as to realize bidirectional information exchange at each level.

[0016] Preferably, the cross-modal fusion in step S3 further includes multi-scale spatial-channel attention recalibration of the features at each level after the hierarchical fusion: multi-scale features are extracted by convolution of multiple receptive fields, and attention weights are generated in the channel dimension and spatial dimension respectively to weight the features, so as to adapt to the differences in field morphology under different crop types and different growth stages.

[0017] Preferably, the cross-modal fusion in step S3 further includes boundary enhancement: calculating the modal gradients of the RGB and digital surface models respectively, estimating the field boundary response based on the modal gradients, and using the field boundary response to enhance the fusion features in the field boundary region, so that the division of the field boundary is based on the relative structural changes between adjacent regions rather than absolute elevation values.

[0018] Preferably, the instance decoder in step S4 is a query-based decoder that uses a preset number of learnable query vectors to predict the category and mask of a breeding field instance for each query under the supervision of real instances, and outputs instance segmentation masks that separate each breeding field from the others.

[0019] Preferably, the training loss of the dual-stream instance segmentation network includes instance segmentation loss and feature alignment loss, wherein the feature alignment loss is used to constrain the spatial consistency of the features extracted by the RGB encoding branch and the digital surface model encoding branch.

[0020] Preferably, the method further includes: S5, converting the instance segmentation mask into a vector polygon, removing isolated false targets through area filtering and compactness filtering, and merging broken masks with high spatial overlap; then using the geographic transformation relationship of the RGB orthophoto, transforming the vector polygon from image coordinates to map coordinates to obtain the geographic registration polygon of each breeding field.

[0021] Preferably, it also includes: S6, assigning a unique identifier to each breeding plot: rotating the centroid of the georeferenced polygon to the main axis of the breeding test area, clustering along the row direction, sorting along the column direction, and assigning a unique identifier to each breeding plot in the format of site-year-crop-number.

[0022] Preferably, it further includes: S7, extracting field-level phenotypic parameters based on the instance segmentation mask, and calculating at least one of the following parameters from the RGB pixels and digital surface model pixels within each breeding field under the mask constraint:

[0023] 1) The area of ​​the field is the product of the number of pixels within the mask and the square of the ground sampling distance;

[0024] 2) Canopy coverage is the ratio of the number of vegetation pixels within the canopy to the total number of pixels within the canopy;

[0025] 3) Over-green index, which is the average value of the normalized red, green and blue components of each pixel in the mask, calculated by subtracting the red component from the blue component by twice the green component.

[0026] 4) Plant height, which is the difference between the high quantile value of the digital surface model inside the mask and the low quantile value of the digital surface model outside the buffer zone of the field boundary;

[0027] 5) The canopy volume is obtained from the canopy coverage, the plant height, and the field area.

[0028] Preferably, the digital surface model is generated by photogrammetric reconstruction of low-altitude images collected by the UAV, and is generated in the same photogrammetric process as the RGB orthophoto and shares a spatial reference frame.

[0029] This invention also discloses a breeding field instance segmentation system that integrates UAV orthophotos and digital surface models, comprising:

[0030] S1, Data Acquisition and Registration Module, is used to acquire UAV RGB orthophotos and digital surface models of the same breeding test area, so that the two have the same spatial reference system and complete pixel-level alignment, and divide the aligned RGB orthophotos and digital surface models into corresponding RGB image blocks and digital surface model image blocks.

[0031] S2, a dual-stream coding module, including a mutually parameter-independent RGB coding branch and a digital surface model coding branch, used to perform multi-level feature extraction on the RGB image block and the digital surface model image block respectively, to obtain RGB multi-level features and high-level multi-level features.

[0032] S3, a cross-modal fusion module, is used to perform cross-modal fusion of the RGB multi-level features and the height multi-level features to obtain multi-level fused features that fuse RGB spectral texture information and digital surface model canopy height structure information at each level; the cross-modal fusion module includes a hierarchical bimodal fusion submodule for bidirectional information exchange between shallow and middle layers, an attention submodule for multi-scale spatial-channel attention recalibration of the fused features, and a boundary attention submodule for enhancing the boundary region based on modal gradient;

[0033] S4, Instance Decoding Module, is used to input the multi-level fusion features into a query-based instance decoder to predict and output instance segmentation masks for each breeding field.

[0034] Beneficial effects of the present invention

[0035] (1) This invention uses a parameter-independent dual-stream encoder and cross-modal fusion to stably introduce the canopy height structure information carried by the DSM into the segmentation of breeding fields, transforming the segmentation that depends on two-dimensional color differences into boundary recognition based on three-dimensional canopy structure; even if adjacent fields are highly similar in color and texture, they can still be separated based on the difference in canopy height and structural discontinuity at the boundary, thus solving the problem of adjacent fields being mistakenly merged under canopy closure conditions from a mechanistic perspective.

[0036] (2) This invention does not simply splice the DSM as an additional channel, but adopts hierarchical dual-modal fusion, multi-scale spatial-channel attention, boundary attention based on modal gradient and feature alignment loss to address the uncertainties such as wind-induced blur, reconstruction error and local voids in UAV photogrammetry DSM, suppress modal noise, improve the stability of fusion and boundary perception, so that height information can be stably used under complex field conditions.

[0037] (3) The method described in this invention was validated on six UAV field test datasets covering 3 crops, 4 locations, a total of 412 breeding plots, and shooting years spanning from 2020 to 2025. The average intersection-over-union ratio (mIoU) reached 93.64%, AP@50 reached 93.84%, panoramic quality (PQ) reached 86.92%, and boundary F1 reached 78.15%. It outperformed the five baseline models U-Net, Mask R-CNN, YOLOv8x-seg, YOLO11x-seg, and YOLO26x-seg in all six evaluated metrics. The ablation experiment showed that the introduction of DSM height information improved mIoU by about 2.00 percentage points and boundary F1 by about 8.39 percentage points, which is the main source of the improvement in segmentation accuracy. The fusion modules further cumulatively improved mIoU by about 3.33 percentage points.

[0038] (4) The field-level phenotypic parameters extracted by the prediction mask of the present invention are highly consistent with the results extracted by the manually labeled mask. The coefficients of determination (R²) of canopy coverage, plant height and canopy volume are 0.853, 0.759 and 0.637, respectively, with a deviation close to zero. This indicates that the automatic segmentation results of the present invention can provide a reliable spatial basis for field-level phenotypic analysis comparable to manual labeling, and support cross-point comparison and time-series phenotypic integration of multi-environment breeding experiments. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the overall process of the breeding field instance segmentation method described in this invention.

[0040] Figure 2 This is a schematic diagram of the structure of the dual-stream instance segmentation network described in this invention.

[0041] Figure 3 This is a schematic diagram of the cross-modal hierarchical dual-modal fusion module, the multi-scale spatial-channel attention module, and the boundary attention module described in this invention.

[0042] Figure 4 This is a schematic diagram illustrating the process of segmentation mask vectorization and breeding field coding in the example described in this invention.

[0043] Figure 5 This is a schematic diagram of the field-level phenotypic parameter extraction described in this invention. Detailed Implementation

[0044] The present invention will be further described below with reference to embodiments, but the scope of protection of the present invention is not limited thereto:

[0045] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings. The embodiments described are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Any equivalent substitutions and improvements made based on the technical solution of the present invention shall fall within the scope of protection of the present invention.

[0046] Example 1

[0047] This embodiment provides a method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models. The overall process is as follows: Figure 1 As shown, steps S1 to S6 are included, as detailed below.

[0048] 1. Data acquisition and preprocessing (corresponding to step S1)

[0049] A drone platform equipped with an RGB camera was used to conduct low-altitude aerial photography of the breeding test fields, acquiring single images with overlapping flight and lateral directions. The acquired images were then processed using photogrammetry to generate RGB orthophotos and digital surface models (DSMs) sharing the same spatial reference frame, with pixel-level alignment. In this embodiment, OpenDroneMap was used to stitch and reconstruct the original drone images. The dataset used is summarized in Table 1, covering three crops (wheat, rice, and corn) and four locations: Rugao, Suining, and Yandu in Jiangsu Province, and Dengzhou in Henan Province, totaling 412 breeding fields, covering various phenological stages from early-growth open canopy to late-growth closed canopy.

[0050] Table 1 Overview of the UAV dataset used in this embodiment

[0051]

[0052] The pixel-aligned RGB orthophoto and DSM were divided into 512×512 pixel image blocks with overlapping areas. Breeding plots were labeled as instance polygons and rasterized to the same grid to maintain the pixel-level correspondence between spectral texture, height, and plot identity. To suppress elevation benchmark differences between different image blocks caused by topographic relief and photogrammetric tilt, the DSM for each image block was first fitted to the effective pixels and a trend surface was subtracted once to remove topographic trends. Then, the values ​​were truncated according to set upper and lower percentiles (e.g., the 1st and 99th percentiles) and normalized to the [0,1] interval in a minimum-maximum manner to obtain the normalized height channel corresponding to each pixel of the RGB image, which served as the input to the digital surface model encoding branch. Geometric enhancements, including random flipping, rotation, and scale jitter, were applied simultaneously to the RGB image blocks, the normalized height channel, and the instance mask. Luminosity enhancements, such as brightness and contrast enhancements, were applied only to the RGB image blocks to avoid disrupting the numerical relationships of the height channel.

[0053] 2. Construction and feature extraction of the dual-stream instance segmentation network (corresponding to step S2)

[0054] like Figure 2As shown, a dual-stream instance segmentation network (denoted as RGBD-PlotFormer in this embodiment) is constructed. The three color channels of the RGB image patch are stacked with a normalized height channel derived from DSM to form a four-channel input. At the network entry point, it is split into an RGB stream (3 channels) and a height stream (1 channel), which are then fed into two parameter-independent encoding branches. Both encoding branches use ResNet-50 as the backbone network, with identical structures but non-shared parameters. The first convolutional layer of the height branch is constructed with a single-channel input, and the remaining structure is consistent with the RGB branch. Each branch outputs four levels of feature maps with downsampling ratios of 4, 8, 16, and 32, and channel numbers of 256, 512, 1024, and 2048, respectively, denoted as C2, C3, C4, and C5. The two branches are independent of each other, avoiding the problem of cross-contamination of different modal noises when a single encoder performs mixed encoding on the concatenated input.

[0055] 3. Cross-modal fusion (corresponding to step S3)

[0056] like Figure 3 As shown, the RGB multi-level features and the high-level multi-level features are subjected to the following cross-modal fusion processing in sequence:

[0057] (3.1) Hierarchical Dual-Modal Fusion Module (HBFM): This module fuses two modal features at three shallower and intermediate levels: C2, C3, and C4. For RGB features, local texture components are extracted via cascaded convolution. For height features, low-frequency components representing the overall canopy structure are obtained through downsampling followed by upsampling. These low-frequency components are then subtracted from the height features and followed by depthwise separable convolution to obtain high-frequency components representing field boundaries. The RGB local texture components, low-frequency height components, and high-frequency height components are concatenated and then fused using an inverse residual bottleneck structure of "channel compression, depthwise separable convolution, and channel restoration." This fusion is then added to the residuals of the two modal inputs to obtain the fused feature for that level. This fused feature is further injected into the RGB and height branches via two independent convolutional projection feedback mechanisms, serving as inputs to the two branches of the next level. This enables a step-by-step bidirectional exchange of RGB and height information during the encoding process. For the deepest level, C5, the two modal features are concatenated and then fused sequentially through pointwise convolution and depthwise separable convolution.

[0058] (3.2) Multi-Scale Spatial-Channel Attention Module (MSCAM): This module recalibrates the fused features at each level. First, multi-scale features are extracted and averaged using three parallel depthwise separable convolutional branches (3×3, 5×5, and dilated 3×3 convolutions, respectively), then mixed via pointwise convolution. Subsequently, channel attention and spatial attention are applied sequentially—channel attention is generated by global average pooling and global max pooling, followed by a shared multilayer perceptron and sigmoid function to produce channel weights; spatial attention is generated by taking the mean and maximum values ​​along the channel dimension, followed by convolution and sigmoid function to produce spatial weights. These two attention weights are applied to the multi-scale features, followed by pointwise convolution, and then added back to the original fused features as residuals to obtain the refined fused features. This module improves the model's adaptability to differences in field morphology for different crop types (such as wheat, rice, and corn) and different growth stages.

[0059] (3.3) Boundary Attention Module (BAM): Located between the backbone network and the instance decoder, it processes the fused features of each layer scale-wise. Learnable gradient convolutions are applied to the RGB branch features and the height branch features respectively (the convolution kernels are initialized by channel using the Sobel operator and further learned during training) to obtain the gradient responses of the two modalities. The RGB gradient responses, height gradient responses and the fused features of this layer are concatenated, and then the boundary evidence features are obtained by pointwise convolution and depthwise separable convolution with residuals. Finally, the boundary attention map a is generated by pointwise convolution and Sigmoid. b ;According to "Enhanced Feature = Fusion Feature × (1 + α·a)", b The fusion features are enhanced in the boundary region using the "×" method, where × represents element-wise multiplication and α is a learnable scale coefficient. BAM makes the field boundary division based on the relative structural changes between adjacent regions rather than absolute elevation values, thus providing robustness to uncertainties such as overall elevation drift and local holes in the DSM. In addition, the boundary evidence features at each scale are upsampled and aggregated, and then output as a boundary prediction map through pointwise convolution. The field boundary labels generated by instance annotations are used for supervision to further enhance the network's perception of field boundaries. After the above processing, the enhanced multimodal fusion features at each level are obtained.

[0060] 4. Instance Decoding and Network Training (corresponding to step S4)

[0061] The multi-level fused features, enhanced by boundary reinforcement, are input into a query-based instance decoder. This embodiment employs a Mask2Former-style decoder, including a pixel decoder and a Transformer decoder based on multi-scale deformable attention. The pixel decoder encodes the multi-level fused features and outputs high-resolution mask features. The Transformer decoder sets 100 learnable queries, which are refined layer by layer through multi-level (9 layers in this embodiment) cross-attention and self-attention. Each query predicts the category and mask of a breeding field instance, with categories including background and field. The mask for each query is obtained by performing a channel-wise dot product between the query's mask embedding vector and the mask features output by the pixel decoder, followed by activation. The final output is an instance segmentation mask that separates each breeding field.

[0062] For end-to-end network training, the total training loss is the sum of instance segmentation loss, feature alignment loss, and boundary loss.

[0063] Equation (1) Total loss: L = L M2F + L align + L bnd .

[0064] Among them, L M2F The instance segmentation loss is calculated at each prediction layer of the decoder. After performing a one-to-one optimal match between the predicted query and the real field instance using the Hungarian algorithm, it is then converted to the classification loss L. cls Mask cross-entropy loss L mask With Dice loss L dice The result is a weighted sum, with the three terms having weights of 2.0, 5.0, and 5.0 respectively. L align The feature alignment loss is used to constrain the consistency of features extracted by the RGB branch and the height branch at intermediate levels (C4 level in this embodiment): global average pooling is performed on the features of each branch at this level, and then the global description vectors z of the two modalities are obtained through their respective linear projections and layer normalization. r With z h Defined by cosine similarity:

[0065] Equation (2) Feature alignment loss: L align = λ align × ( 1 - cos(z r , z h ), where the coefficient λ align Take 0.02.

[0066] Feature alignment loss constrains the two branches to maintain global semantic consistency when local quality differences exist between the two modalities (e.g., a local hole in the DSM corresponds to a normal RGB region), thereby suppressing modal noise propagation and stabilizing the fusion effect. bndThe boundary loss is used to supervise the boundary prediction map output by the boundary attention module. It is composed of a weighted binary cross-entropy loss with class balance and a Dice loss, with a weight of 0.5. Its boundary ground truth is generated by morphological dilation of pixels with different labels in adjacent instances.

[0067] Training was performed using the AdamW optimizer with an initial learning rate of 1×10⁻ 4 The weight decay is 0.05, and the learning rate of the backbone network is 0.1 times the global learning rate. Gradient norm pruning is adopted, and the learning rate is adjusted by a polynomial decay strategy with a power of 0.9. Linear warm-up is performed in the early stage of training. The input size is 512×512 pixels, the batch size is 2, and a total of 80,000 iterations are trained. The model weights with the highest average intersection-union ratio (mIoU) on the validation set are retained.

[0068] 5. Field vectorization and unique coding (corresponding to step S5)

[0069] like Figure 4 As shown, the instance segmentation mask output by the network is converted into vector polygons. Isolated false targets are removed by area filtering and compactness filtering, and fragmented masks with high spatial overlap are merged. Using the geographic transformation relationship of RGB orthophotos, the vector polygons are transformed from image coordinates to map coordinates to obtain the geographic registration polygons of each breeding field. Then, row and column encoding is performed: the centroid of the polygon is rotated to the main axis of the breeding test area, clustered along the row direction, sorted along the column direction, and a unique identifier is assigned to each breeding field according to the format of "site-year-crop-number", forming a digital farmland for digital management to support the integration and time-series analysis of field-level phenotypic data across years and locations.

[0070] 6. Field-level phenotypic extraction (corresponding to step S6)

[0071] like Figure 5 As shown, under the mask constraint of each breeding plot, plot-level phenotypic parameters are calculated from the RGB and DSM pixels within that plot. This ensures that each measurement is confined to a single plot, reducing contamination from adjacent plots, passageways, and soil background. Let n be the number of pixels within the mask of plot p. p The number of vegetation pixels is n. veg,p If the ground sampling distance is GSD, then:

[0072] Formula (3) Field area: A p = n p × GSD²

[0073] Equation (4) Canopy coverage: C p = n veg,p / n p ;

[0074] Equation (5) Greenness Index: The normalized greenness index ExG is calculated pixel by pixel. i = (2g i -r i - b i ) / (r i + g i + b i + ε), where r i g i b i For red, green, and blue components, ε is a very small positive number to prevent the denominator from being zero; within the field, for ExG i Vegetation pixels were obtained through Otsu thresholding, and the field-level overgreen index was obtained by taking the ExG values ​​within the field. i The mean;

[0075] Formula (6) Plant height: H p = Q0.95(z in,p ) - g p , where z in,p The DSM value within the field mask is given, with Q0.95 being its 95th percentile. p This is the ground reference elevation for the location of the field; g p The lower quantile (e.g., the 5th percentile) of the DSM of the buffer zone outside the field boundary can be taken, or the value of the ground trend surface fitted by the bare soil pixels between fields at the centroid of the field can be taken.

[0076] Equation (7) Vegetation area: S veg,p = n veg,p × GSD²

[0077] Equation (8) Canopy volume: V p = Σ {i∈veg} max(z i -g p , 0) × GSD², which is the height integral of each vegetation pixel within the field relative to the ground reference; it can also be simplified to V p = C p × H p × A p .

[0078] The above vegetation-background discrimination was performed separately inside the mask in each field; ground reference g p The ground trend surface was obtained by fitting bare soil pixels between fields. Plant height and canopy volume were both obtained using robust quantile statistics to reduce the impact of local cavities, outliers and field topographic undulations.

[0079] 7. Effect Verification

[0080] Under the same dataset and training settings, the RGBD-PlotFormer described in this embodiment was compared with five baseline models, and some results are shown in Table 2. This embodiment ranked first in all evaluated metrics, with mIoU of 93.64%, boundary F1 of 78.15%, and mean absolute error of field count (CountMAE) of 0.364, which is significantly better than the baseline model that only relies on RGB.

[0081] Table 2. Comparison of segmentation accuracy (%) between the present invention and the baseline model under the same settings.

[0082]

[0083] The ablation experiment results are shown in Table 3. Introducing DSM height information into the RGB-only model, even with simple fusion, improved mIoU from 88.31% to 90.31% (an improvement of 2.00 percentage points), and boundary F1 from 73.44% to 81.83% (an improvement of 8.39 percentage points), indicating that DSM height information is the main source of improved segmentation accuracy. Furthermore, HBFM, MSCAM, BAM, and alignment loss further improved mIoU to 93.64%. The most significant decrease in mIoU was observed when the boundary attention module was removed, indicating that boundary attention contributes the most to boundary quality.

[0084] Table 3 Ablation results (%) of DSM and each fusion module

[0085]

[0086] To verify the reliability of the segmentation results for downstream phenotypic analysis, a maize trial in Dengzhou, Henan Province, on July 31, 2025 (where the closed canopy and dense field arrangement present the most challenging scenario for boundary delineation) was used as an example. For the 66 breeding fields in this trial, canopy coverage, plant height, and canopy volume were extracted using both the prediction mask of this invention and manually labeled masks following the same process, and the results were compared. The results showed a high degree of consistency: the coefficient of determination (R²) for canopy coverage was 0.853 with a root mean square error (RMSE) of 0.0122; the R² for plant height was 0.759 with an RMSE of 0.13 m; and the R² for canopy volume was 0.637 with an RMSE of 1.93 m³. The deviations for each phenotype were close to zero, indicating that the automatic segmentation of this invention did not introduce significant systematic bias, and its mask can support field-level phenotypic extraction with comparable accuracy to manual labeling.

[0087] 8. Method pseudocode

[0088] Input: UAV RGB orthophoto I rgb With digital surface model I dsm(Same reference frame, cell-level alignment) 1: By I dsm Detrending and quantile normalization yields a high-level channel; with I rgb Stacked as 4-channel input and segmented into overlapping image blocks {x}2: for each image block x do3: segmented into x by channel rgb With x h 4: {R s} ← RGB encoding branch (x rgb ); {H s} ← Height-encoded branch (x h / / Parameters are independent, s represents each level 5: for each level sdo 6: F s ← HBFM(R s H s / / High-frequency decomposition + local RGB, complementary fusion 7: F s ← MSCAM(F s / / Multi-scale + Channel / Spatial Attention 8: F s Feedback is injected into the R and H branches of the next level / / Bidirectional switching at each level 9: end for 10: {F s} ←BAM({F s}, {R s}, {H s}) / / Boundary enhancement based on modal gradient 11: M ← Mask2Former-style instance decoder({F s}) / / Query 100, output field instance mask 12: end for 13: Training objective: L = L M2F +L align + L bnd 14: Reasoning: stitch together image patches → whole-field instance mask; vectorization, filtering and merging, image coordinates → map coordinates 15: centroid rotate to principal axis, row and column encoding → unique identifier (digital farmland) 16: under mask constraints, extract field-level phenotypic output from equations (3)-(8): instance mask of each breeding field, unique identifier and field-level phenotypic parameters

[0089] Example 2

[0090] The difference between this embodiment and Embodiment 1 is that: the backbone network of the encoding branch can be replaced with other convolutional neural networks or visual Transformer backbones; the number of queries in the instance decoder can be adjusted according to the number of fields in a single image patch; the bidirectional exchange of the hierarchical dual-modal fusion module can be applied only to the shallow layer or simultaneously to the shallow and middle layers; the modal gradient of the boundary attention module can be implemented using other gradient operators or learnable boundary prediction branches. When only the breeding field instance segmentation result is needed, steps S1 to S4 can be executed; when digital farmland management and field-level phenotypic analysis are required, steps S5 and S6 are further executed. The above substitutions and adjustments do not depart from the core concept of this invention, "to achieve breeding field instance segmentation by stably introducing DSM high-structure information through parameter-independent dual-stream coding and cross-modal fusion," and all fall within the protection scope of this invention.

[0091] The primary technical problem this invention aims to solve is that after crop canopy closure, the color and texture of adjacent breeding plots in orthophotos tend to be continuous, and the two-dimensional spectral features no longer exhibit separable gradients at plot boundaries. Therefore, a segmentation network using RGB single-modality as input will inevitably merge adjacent plots into the same connected region. It can be deduced that to fundamentally restore separability at boundaries, a physical quantity that maintains abrupt changes under canopy closure conditions must be introduced. The canopy top elevation recorded by photogrammetric digital surface models possesses this property—the absence of crop canopy at passageways, furrows, or planting intervals causes a step along plot boundaries, and this step does not disappear as canopy colors converge. Therefore, this invention uses pixel-level aligned RGB orthophotos and DSM to form a dual-modal input, and extracts spectral texture features and height structure features respectively using two coding branches with non-shared weights. Since the discrimination criteria of the two modalities are independent of each other (one comes from the reflectance spectrum of ground objects, and the other comes from three-dimensional morphology), their failure conditions do not overlap. The fused features can still maintain the discriminative power of the other modality when either modality fails. Furthermore, this invention does not perform a one-time fusion at the end of the network, but performs cross-modal fusion at multiple feature levels and feeds the fused features back into the two coding branches. This allows height structure information to participate in the feature formation of the RGB branch in the early stages of encoding. The shallow boundary response is strengthened by the height step, instead of waiting until the deep semantic abstraction is completed before being introduced, at which point the boundary details have become irrecoverable due to step-by-step downsampling. Based on this, it can be deduced that the basis for dividing breeding fields has been transformed from "two-dimensional color difference" to "three-dimensional structural boundary recognition". Adjacent and contiguous fields can be separated into independent instances. Moreover, this separation does not rely on artificial geometric post-processing that is tied to a specific field layout, such as Hough line detection, traction line prior, or row and column regularity assumption. The adaptability of the scheme to different crop types, different growth stages, and different test sites is thus improved.

[0092] Secondly, photogrammetric DSM has reconstruction errors and local voids in areas with sparse canopies, poor texture, or wind-induced disturbances, and its data quality is not uniform in space. If the two modes are directly stitched together without distinction, noise will be transmitted into the fusion feature without attenuation. To this end, this invention employs a triple design in its fusion structure, the effects of which can be deduced sequentially: First, height features are decomposed into low-frequency and high-frequency components—the low-frequency component represents the overall canopy morphology and is insensitive to local reconstruction errors, while the high-frequency component represents height abrupt changes at the boundaries, which is essential for separating fields. These two components are then complementary and fused with the RGB local texture components, allowing the network to utilize the truly effective components of height information as needed, without having to bear all the noise. Second, the boundary attention module extracts relative structural changes from the two modal features using learnable gradient convolutions as boundary evidence. Since it relies on local gradients rather than absolute elevation thresholds, it is unaffected by terrain undulations and absolute crop height. No threshold recalibration is required when migrating across sites or crops. Simultaneously, the explicit supervision of the boundary prediction head guides the network attention to the contour rather than the interior of the region, resulting in improved boundary localization accuracy independent of region overlap accuracy. Third, the cross-modal feature alignment loss constrains the two branch features to fall into a consistent semantic space. When the DSM locally degenerates, the RGB... The reliable characteristics of branching can exert traction on height branches, thereby suppressing the cross-modal propagation of noise. This mechanism was confirmed in experiments: introducing DSM branching increased the average crossover-union ratio by 2.00 percentage points, while the boundary F1 score increased by 8.39 percentage points—the latter increase being significantly greater than the former. This precisely illustrates that the contribution of height information mainly acts on the most difficult-to-segment part—the field boundary—which is completely consistent with the aforementioned reasoning. Ultimately, this method achieved an average crossover-union ratio of 93.64%. Because its output consists of mutually separated and accurately defined field instances, after vectorization and unique row and column encoding, it can be matched one-to-one with the field design table, forming a digital farmland base map that can be reused across years. Under mask constraints, it stably extracts field-level phenotypic parameters (with a consistency R² of 0.637-0.853 with ground-measured values), transforming the original manual field delineation process in breeding phenotypic identification into an automated workflow.

[0093] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for segmenting breeding field instances by integrating UAV orthophotos and digital surface models, characterized in that, Includes the following steps: S1. Obtain UAV RGB orthophotos and digital surface models of the same breeding test area. The RGB orthophotos and digital surface models have the same spatial reference system and are aligned at the pixel level. The aligned RGB orthophotos and digital surface models are divided into corresponding RGB image blocks and digital surface model image blocks. S2. Construct a dual-stream instance segmentation network, which includes an RGB encoding branch and a digital surface model encoding branch with mutually independent parameters. Use the RGB encoding branch and the digital surface model encoding branch to perform multi-level feature extraction on the RGB image block and the digital surface model image block respectively, and obtain RGB multi-level features and high-level multi-level features accordingly. S3. Perform cross-modal fusion on the RGB multi-level features and the height multi-level features to obtain multi-level fused features that fuse RGB spectral texture information and digital surface model canopy height structure information at each level; S4. Input the multi-level fusion features into the instance decoder to predict and output the instance segmentation mask for each breeding field in the breeding test area.

2. The method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models according to claim 1, characterized in that, Step S1, the pixel-level alignment and segmentation, includes: unifying the RGB orthophoto and the digital surface model to the same spatial reference system and the same ground sampling distance; truncating the digital surface model numerically according to a set percentile, and normalizing each segmented digital surface model image block separately to suppress elevation reference differences between different image blocks; segmenting the aligned data into image blocks with overlapping areas, and simultaneously applying geometric enhancements, including flipping, rotation, and scaling, to the RGB image blocks, digital surface model image blocks, and instance annotation masks, while applying photometric enhancement only to the RGB image blocks.

3. The method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models according to claim 1, characterized in that, In step S2, the RGB encoding branch and the digital surface model encoding branch each use a parameter-independent residual network as their backbone. The two encoding branches have the same network structure but do not share parameters. They output multiple layers of feature maps from shallow to deep layers respectively, thus obtaining the RGB multi-level features and the high-level multi-level features.

4. The method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models according to claim 1, characterized in that, Step S3, the cross-modal fusion, includes hierarchical bimodal fusion, multi-scale spatial-channel attention recalibration, and boundary enhancement. Specifically, at multiple feature levels, the highly multi-level features are decomposed into low-frequency components representing the overall canopy structure and high-frequency components representing field boundaries. These components are then complementary and fused with the local texture components of the RGB multi-level features to obtain hierarchical fusion features. These hierarchical fusion features are then fed back into the RGB encoding branch and the digital surface model encoding branch, respectively, as inputs to the two encoding branches at the next level, to achieve hierarchical bidirectional information exchange. Multi-scale features are extracted through convolution with multiple different receptive fields, and attention weights are generated in both the channel and spatial dimensions to weight the features, adapting to differences in field morphology under different crop types and growth stages. Modal gradients of RGB and the digital surface model are calculated respectively. Field boundary responses are estimated based on these modal gradients, and the field boundary responses are used to enhance the fusion features in the field boundary region, ensuring that the division of field boundaries is based on relative structural changes between adjacent regions rather than absolute elevation values.

5. The method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models according to claim 1, characterized in that, The instance decoder described in step S4 is a query-based decoder. It uses a preset number of learnable query vectors and predicts the category and mask of a breeding field instance for each query under the supervision of real instances, outputting instance segmentation masks that separate each breeding field from the others.

6. The method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models according to any one of claims 1 to 5, characterized in that, The training loss of the dual-stream instance segmentation network includes instance segmentation loss and feature alignment loss. The feature alignment loss is used to constrain the spatial consistency of the features extracted by the RGB encoding branch and the digital surface model encoding branch.

7. The method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models according to claim 1, characterized in that, Also includes: S5. Convert the instance segmentation mask into a vector polygon, remove isolated false targets by area filtering and compactness filtering, and merge broken masks with high spatial overlap; then use the geographic transformation relationship of the RGB orthophoto to transform the vector polygon from image coordinates to map coordinates to obtain the geographic registration polygon of each breeding field.

8. The method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models according to claim 7, characterized in that, Also includes: S6. Assign a unique identifier to each breeding plot: Rotate the centroid of the georeferenced polygon to the main axis of the breeding test area, cluster along the row direction, sort along the column direction, and assign a unique identifier to each breeding plot in the format of site-year-crop-number.

9. The method for segmenting breeding field instances by fusing UAV orthophotos and digital surface models according to claim 7, characterized in that, Also includes: S7. Extract field-level phenotypic parameters based on the instance segmentation mask. Under the mask constraint of each breeding field, calculate at least one of the following parameters from the RGB pixels and digital surface model pixels within the field: 1) The area of ​​the field is the product of the number of pixels within the mask and the square of the ground sampling distance; 2) Canopy coverage is the ratio of the number of vegetation pixels within the canopy to the total number of pixels within the canopy; 3) Over-green index, which is the average value of the normalized red, green and blue components of each pixel in the mask, calculated by subtracting the red component from the blue component by twice the green component. 4) Plant height, which is the difference between the high quantile value of the digital surface model inside the mask and the low quantile value of the digital surface model outside the buffer zone of the field boundary; 5) The canopy volume is obtained from the canopy coverage, the plant height, and the field area.

10. A breeding field instance segmentation system integrating UAV orthophotos and digital surface models, characterized in that, include: S1, Data Acquisition and Registration Module, is used to acquire UAV RGB orthophotos and digital surface models of the same breeding test area, so that the two have the same spatial reference system and complete pixel-level alignment, and divide the aligned RGB orthophotos and digital surface models into corresponding RGB image blocks and digital surface model image blocks. S2, a dual-stream coding module, including a mutually parameter-independent RGB coding branch and a digital surface model coding branch, used to perform multi-level feature extraction on the RGB image block and the digital surface model image block respectively, to obtain RGB multi-level features and high-level multi-level features. S3, a cross-modal fusion module, is used to perform cross-modal fusion of the RGB multi-level features and the height multi-level features to obtain multi-level fused features that fuse RGB spectral texture information and digital surface model canopy height structure information at each level; the cross-modal fusion module includes a hierarchical bimodal fusion submodule for bidirectional information exchange between shallow and middle layers, an attention submodule for multi-scale spatial-channel attention recalibration of the fused features, and a boundary attention submodule for enhancing the boundary region based on modal gradient; S4, Instance Decoding Module, is used to input the multi-level fusion features into a query-based instance decoder to predict and output instance segmentation masks for each breeding field.

Images