Multi-scale leaf instance segmentation method and system for complex forest stand

Abstract

The invention belongs to the technical field of image processing, and discloses a multi-scale leaf instance segmentation method and system for complex forest stands. The method comprises the following steps: constructing a leaf segmentation data set of a forestry scene; generating an initial multi-scale feature pyramid through multi-scale feature extraction; recalibrating the features along the channel dimension by using a squeezing and incentive attention mechanism to obtain a channel enhanced feature pyramid; carrying out multi-scale feature fusion by utilizing the BiFPN to generate a multi-scale feature pyramid; performing spatial dimension recalibration by using a spatial attention sub-module to obtain a final refinement feature pyramid; and a cascade region convolutional neural network is adopted as a solution wharf, and a positioning and pixel-level segmentation result of each detection blade is obtained through iterative optimization. According to the method, adaptive fusion of multi-scale blade features and directional suppression of complex background noise are realized, so that instance distinguishing of pixel-level precision is realized in dense and seriously shielded scenes.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a multi-scale leaf instance segmentation method and system for complex forest stands. Background Technology

[0002] Forests are the core of terrestrial ecosystems, and their health and productivity directly affect carbon sequestration capacity, biodiversity conservation, and timber resource security. Achieving precise management and sustainable operation of forest resources hinges on efficient and accurate monitoring and assessment of tree growth. Leaves, as the primary organs for photosynthesis, transpiration, and responses to environmental stresses, contain key phenotypic traits reflecting the health, productivity, and environmental adaptability of individual and group trees, including their quantity, morphology, color, and physiological state. Therefore, achieving high-throughput, automated phenotypic analysis of tree leaves is crucial for accelerating the selection of superior tree species, implementing precision irrigation, and providing early warning of biotic and abiotic stresses.

[0003] Traditional leaf trait measurements rely primarily on manual sampling and laboratory analysis. This method is inefficient, destructive, prone to subjective errors, and difficult to achieve continuous spatiotemporal monitoring at the canopy scale. In recent years, the widespread adoption of UAV remote sensing and near-ground sensing technologies has made it possible to rapidly acquire high-resolution forest stand image data. However, automatically and accurately segmenting each leaf instance from these complex natural scene images and further extracting quantitative traits remains a core challenge in the field. This is mainly due to the inherent complexity of the natural forest stand environment: structural complexity: leaves exhibit severe overlap and occlusion, resulting in blurred boundaries; morphological variability: leaf shape, size, and orientation show high heterogeneity both within individuals and between species; environmental interference: changes in light, shadows, and complex backgrounds (soil, dead branches, weeds) strongly interfere with image features. Existing general image segmentation models (such as Mask R-CNN and YOLO series) and models trained on standard datasets (such as COCO) often suffer from problems such as instance loss, boundary missegmentation, and inability to distinguish adhered leaves when facing the above-mentioned forestry-specific challenges, resulting in insufficient accuracy and robustness.

[0004] Despite the revolutionary advancements in computer vision brought about by deep learning, particularly the significant advantages of the Transformer architecture in global context modeling, its direct application to forestry leaf analysis still faces significant challenges. Firstly, there is a lack of domain-specific data; most publicly available datasets target indoor environments or simple backgrounds, lacking large-scale, finely labeled segmentation data of leaf instances in natural forest scenes. Secondly, model architectures lack targeted optimization; existing models fail to effectively integrate dedicated modules to handle core issues such as multi-scale forestry leaf analysis, high occlusion, and background clutter. Finally, there is a gap between "segmentation" and "trait" analysis; most studies stop at improving segmentation accuracy, failing to build an automated analysis pipeline directly from pixel-level segmentation results to usable agronomic parameters (such as stress indices). Developing a complete technical solution encompassing dedicated data construction, customized model design, and practical phenotypic analysis has become an urgent need to advance forestry phenomics from research to application and achieve precise decision-making in smart forestry. This study addresses this need by proposing an innovative and systematic solution. Summary of the Invention

[0005] This invention aims to address the problems of insufficient accuracy, limited practicality, and fragmented process in existing technologies for leaf phenotypic analysis in complex natural forest environments, and proposes a multi-scale leaf instance segmentation method and system for complex forest stands.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A multi-scale leaf instance segmentation method for complex forest stands includes the following steps: Using UAV remote sensing technology, image data of target trees are acquired to construct a leaf segmentation dataset for forestry scenarios; Based on the aforementioned leaf segmentation dataset, an improved Swing Transformer backbone network is used for multi-scale feature extraction, generating an initial multi-scale feature pyramid consisting of four layers of feature maps suitable for multi-scale leaf feature parsing. ; Based on the initial multi-scale feature pyramid The features are recalibrated along the channel dimension using a squeezing and incentive attention mechanism to obtain a channel-enhanced feature pyramid. ; Feature Pyramid Based on Channel Enhancement By utilizing an improved bidirectional weighted feature pyramid network (BiFPN), multi-scale feature fusion is performed to generate a set of enhanced multi-scale feature pyramids that incorporate information from all scales. ; The spatial attention submodule of the convolutional block attention module is used to process the multi-scale feature pyramid output by BiFPN. Each scale feature in the pyramid is recalibrated in terms of spatial dimensions to obtain the final refined feature pyramid. ; Based on the refined feature pyramid A cascaded regional convolutional neural network is used as the decoding head, and the localization and pixel-level segmentation results of each detection blade are obtained through iterative optimization.

[0007] Preferably, the improved Swing Transformer backbone network employs a window-based self-attention mechanism, including window-based multi-head self-attention (W-MSA) and shift-window-based multi-head self-attention (SW-MSA), as well as downsampling layers interspersed between the two. The switching between W-MSA and SW-MSA is achieved through a mask matrix.

[0008] Preferably, the features are recalibrated along the channel dimension using a squeezing and excitation attention mechanism to obtain a channel-enhanced feature pyramid. include: For the initial multi-scale feature pyramid Feature map in Global average pooling is used to compress the spatial information of each channel to generate channel descriptors; By utilizing a two-layer fully connected network with a bottleneck structure, the channel descriptor learns the nonlinear interactions between channels, thereby obtaining the channel weight vector; The channel weight vector is multiplied channel by channel with the original feature map to obtain the calibrated feature map; the initial multi-scale feature pyramid is then processed. Each layer of features is recalibrated to obtain a channel-enhanced feature pyramid. .

[0009] Preferably, the improved bidirectional weighted feature pyramid network BiFPN constructs a bidirectional information propagation graph and utilizes bidirectional cross-scale information flow and learnable adaptive weights to achieve the fusion of features at different resolutions, thereby simultaneously capturing fine leaf vein texture and macroscopic leaf morphology. The update process of the bidirectional information propagation graph is divided into a top-down path and a bottom-up path. Among them, in the top-down path, features The generation combines the input of the current layer with the higher-resolution features of the previous layer, and in the bottom-up path, the output features... Further integrate the intermediate features from the top-down path with the lower-resolution features of the next layer; fusion function for: ; BiFPN achieves iteration through cross-layer connections.

[0010] Preferably, the spatial attention submodule of the convolutional block attention module is used to process the multi-scale feature pyramid output by BiFPN. The spatial dimension recalibration of each scale feature includes: First, the multi-scale feature pyramid For each scale feature in the image, global max pooling and global average pooling are performed along the channel dimension to obtain two spatial description maps. and Then, the two are concatenated along the channel dimension and integrated through a standard convolutional layer to generate a two-dimensional spatial importance map. Finally, the spatial importance map is multiplied element-wise with the input feature map to obtain the spatially refined feature map. .

[0011] Preferably, the method for obtaining the localization and pixel-level segmentation results of each detected leaf by using a cascaded region convolutional neural network as the decoding head includes: For refined feature pyramids Using the Region Proposal Network (RPN) to refine the feature pyramid Generate an initial set of proposal boxes. Subsequently, the Each cascade stage is the output proposal box of the previous stage. As input, perform more refined detection and segmentation: , In the formula, Represents the learnable detection head function for the m-th stage, with parameters as follows: , The operation extracts a fixed-size feature region corresponding to each proposal box from the feature pyramid; function Internally, three subtasks are executed in parallel: bounding box regression, object classification, and mask prediction. Each subtask uses a progressively increasing IoU threshold. The positive and negative samples are defined; the bounding box of the mask prediction output is combined with the mask that has been bilinearly upsampled to the original image resolution to obtain the localization and segmentation result of each detected leaf instance.

[0012] The present invention also provides a multi-scale leaf instance segmentation system for complex forest stands. The system applies the aforementioned method and includes an image data acquisition unit, a multi-scale feature extraction unit, a calibration unit, a fusion unit, a recalibration unit, and an optimization unit. The image data acquisition unit is used to acquire image data of target trees using UAV remote sensing technology and construct a leaf segmentation dataset for forestry scenarios. The multi-scale feature extraction unit is used to extract multi-scale features based on the leaf segmentation dataset using an improved SwinTransformer backbone network, generating an initial multi-scale feature pyramid consisting of four layers of feature maps suitable for multi-scale feature parsing of leaves. ; The calibration unit is used to base the initial multi-scale feature pyramid. The features are recalibrated along the channel dimension using a squeezing and incentive attention mechanism to obtain a channel-enhanced feature pyramid. ; The fusion unit is used for feature pyramids based on channel enhancement. By utilizing an improved bidirectional weighted feature pyramid network (BiFPN), multi-scale feature fusion is performed to generate a set of enhanced multi-scale feature pyramids that incorporate information from all scales. ; The recalibration unit is used to process the multi-scale feature pyramid output by BiFPN using the spatial attention submodule of the convolutional block attention module. Each scale feature in the pyramid is recalibrated in terms of spatial dimensions to obtain the final refined feature pyramid. ; The optimization unit is used to optimize based on the refined feature pyramid. A cascaded regional convolutional neural network is used as the decoding head, and the localization and pixel-level segmentation results of each detection blade are obtained through iterative optimization.

[0013] The beneficial effects of this invention are as follows: This invention provides a multi-scale leaf instance segmentation method and system for complex forest stands, which has the following technical effects: This invention constructs a systematic and innovative framework from data and algorithms to applications, fundamentally improving the accuracy and practicality of leaf phenotypic analysis in complex natural forest environments. Unlike methods that rely on general models or simple improvements to existing architectures, the LeafSeg-T algorithm proposed in this invention at the model level does not simply apply an existing architecture, but achieves several key breakthroughs through deep customization. Its improved Swing algorithm... The Transformer backbone network, combined with learnable relative position biases, accurately captures the long-range spatial contextual relationships between overlapping leaves. The innovative cascaded attention mechanism and the synergistic design of the bidirectional weighted feature pyramid network enable adaptive fusion of multi-scale leaf features and targeted suppression of complex background noise. This achieves pixel-level precision instance differentiation even in dense, heavily occluded scenes. Crucially, this invention transcends the limitations of traditional research focused solely on segmentation output, seamlessly integrating high-precision segmentation with direct forestry analysis. The developed LIDA module and LDSI index enable detailed analysis from images to morphological parameters and physiological stress states, providing direct decision-making support for early screening in forest tree breeding and precise irrigation management. Comprehensive experiments demonstrate that this technology not only achieves leading-edge segmentation accuracy but also exhibits excellent robustness and generalization ability in zero-sample transfer, marking a key step towards the practical application and intelligentization of forestry phenotypic analysis. Attached Figure Description

[0014] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a schematic diagram of the process of a multi-scale leaf instance segmentation method for complex forest stands according to the present invention; Figure 2 This is a schematic diagram of the contour extracted from image data and the data information extracted from the leaf in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Example 1 This embodiment provides a multi-scale leaf instance segmentation method for complex forest stands. First, at the two-dimensional image level, an improved visual Transformer backbone network extracts multi-scale depth features to simultaneously capture the global morphological contour and local structural details of the leaves, overcoming the limitations of traditional methods in representing overlapping and occluded leaves. Next, a progressive optimization path integrating channel attention, bidirectional weighted feature fusion, and spatial attention is designed and constructed. A unified multi-task loss function is used to dynamically supervise and globally optimize this path, guiding the model to focus on learning to distinguish difficult samples and avoiding the model getting trapped in local optima or generating redundant predictions. Finally, through cascaded decoding and refinement, pixel-level leaf instance segmentation is achieved, and quantitative phenotypic parameters and stress indices that can be directly used for breeding screening and precision irrigation are automatically output, thereby comprehensively improving the completeness, accuracy, and practical application value of the leaf phenotypic analysis process.

[0017] The following is combined Figure 1 The following is a detailed explanation of the specific steps in an embodiment of the present invention: S1. Using UAV remote sensing technology, image data of target trees are acquired, and a leaf segmentation dataset for forestry scenarios is constructed.

[0018] In this embodiment, a drone platform equipped with a high-resolution camera is used to collect two-dimensional sequence image data of the forest stand in a low-altitude multi-angle flight mode, and a leaf segmentation dataset covering forestry scenes under complex natural conditions is constructed.

[0019] S2. Based on the leaf segmentation dataset, an improved Swing Transformer backbone network is used to extract leaf image features at multiple scales, generating an initial multi-scale feature pyramid consisting of four layers of feature maps suitable for multi-scale leaf feature parsing. .

[0020] In this embodiment, a hierarchical visual backbone network with a layered design, the Swing Transformer, is used as the core feature extraction backbone. The core innovation of this backbone network lies in its shifted window multi-head self-attention mechanism, SW-MSA. This mechanism can efficiently unify local feature extraction and global dependency modeling with linear computational complexity, thereby adaptively capturing multi-scale leaf features and long-range spatial context information.

[0021] The advantage of the Swin Transformer lies in its efficient window-based multi-head self-attention mechanism, W-MSA, which is expressed as: , For the feature map output by the preceding Swing Transformer The query, key, and value tensors are generated through learnable linear projections: , , , Where X is the feature map input to the current network layer, output by the preceding Swin Transformer; H and W are the height and width of the feature map X, respectively; and C is the number of channels in the feature map. , , These are the linear projection weight matrices that can be learned during network training.

[0022] Self-attention calculation is summarized by the following unified formula, which encapsulates two operation modes: window-based multi-head self-attention (W-MSA) and shift-window-based multi-head self-attention (SW-MSA): , , In the formula, Representatives are divided in the first Each attention point, size is Token projection within a local window This represents the concatenated result of all attention head outputs. For the total number of attention heads, This represents the output projection weight matrix of head self-attention. This represents the dimension of the keys in each attention head. The key innovation lies in representing relative position offset through a compact, learnable parameter table. Perform efficient parameterization (where (where M represents the set of real numbers and M represents the size of the local window) This injects translational isomorphism into the model, enabling it to accurately perceive the internal structural elements of the blades and the relative orientations between overlapping blades. Mask matrix Used to implement W-MSA ( ) and SW-MSA( The switching between the two methods involves the latter establishing cross-window connections through cyclic shifting and masking mechanisms to model the long-range dependency between the occluded blade portion and its global context, while maintaining linear computational complexity. Mask matrix Derived from the Swing Transformer, the parameter m takes the value 0 or 1, corresponding to the attention calculation constraints of W-MSA and SW-MSA respectively. In W-MSA, attention is restricted to be calculated only within the window, while in SW-MSA, cyclic shifting is used to mask invalid regions, retaining only valid cross-window connections, thus maintaining linear computational complexity while modeling long-range dependencies.

[0023] To construct a pyramid representation suitable for multi-scale leaf feature parsing, downsampling layers are interspersed between attention modules. The downsampling layers perform a PatchMerging operation to halve the spatial resolution and double the channel dimension. The process is formalized as follows: For the multi-scale leaf image features output by the Swin Transformer backbone network... ( (representing the set of real numbers), dividing it into Non-overlapping blocks, within each block After concatenating the 3D features, the dimensionality is reduced to 1000 dimensionality using linear projection. : .

[0024] in, This represents four sub-feature blocks within a 2×2 non-overlapping window divided from the input feature map F. This represents the learnable linear projection weight matrix used for dimensionality reduction in the Patch Merging operation. It represents the set of real numbers.

[0025] By alternately stacking the shifted window attention blocks and the PatchMerging layer, the SwinTransformer backbone network outputs an initial multi-scale feature pyramid consisting of four feature maps, suitable for multi-scale feature parsing of leaves. Its spatial resolution decreases step by step while its semantic information becomes increasingly rich.

[0026] S3. Based on the initial multi-scale feature pyramid The features are recalibrated along the channel dimension using a squeezing and incentive attention mechanism to obtain a channel-enhanced feature pyramid. .

[0027] Background noise in complex natural environments (such as soil and fallen leaves) and variations in leaf color and texture across different tree species and growth stages can cause certain feature channels to be dominated by irrelevant information. The Squeeze-and-Excitation (SE) attention mechanism adaptively enhances discriminative feature channels relevant to leaf recognition by explicitly modeling the interdependencies between channels, while suppressing redundant or interfering channels, thereby improving the signal-to-noise ratio of features in the early stages.

[0028] In this embodiment, for the initial multi-scale feature pyramid Feature map in First, SE compresses the spatial information of each channel through a global average pooling operation to generate channel descriptors. , its first The elements are: , in, Represents the feature map P of the l-th layer l Spatial height, Represents the feature map P of the l-th layer l Space width, Represents the feature map P of the l-th layer l The feature value of the c-th channel at spatial location (i,j).

[0029] This channel descriptor The channel weight vector is generated by passing the channel through a two-layer fully connected network with a bottleneck structure to learn the nonlinear interactions between channels.

[0030] The learned channel weight vectors are then compared with the initial multi-scale feature pyramid. The feature maps in the original data are multiplied channel by channel to obtain the calibrated feature maps: .

[0031] In the formula, and For learnable weight matrix, Represents the ReLU activation function. The compression ratio is set to 16 in this paper to balance performance and model complexity. This represents the Sigmoid function, which normalizes the output to the (0,1) interval, representing the importance weight of each channel. Represents the feature map P of the l-th layer l The number of channels. This represents a channel-dimensional broadcast multiplication. SE is applied independently to each layer of features output from the backbone network. The feature pyramid with enhanced channels is obtained. .

[0032] S4. Feature Pyramid Based on Channel Enhancement By utilizing an improved bidirectional weighted feature pyramid network (BiFPN), multi-scale feature fusion is performed to generate a set of enhanced multi-scale feature pyramids that incorporate information from all scales. .

[0033] Images of young trees simultaneously contain millimeter-sized new leaflets and centimeter-sized mature leaves, exhibiting extremely significant scale heterogeneity.

[0034] To address this issue and effectively integrate multi-scale feature information filtered by channels, this embodiment introduces an improved Bidirectional Weighted Feature Pyramid Network (BiFPN). BiFPN achieves efficient fusion of features at different resolutions through bidirectional (top-down and bottom-up) cross-scale information flow and learnable adaptive weights, ensuring that the model can simultaneously capture fine leaf vein textures and macroscopic leaf morphology.

[0035] The core of BiFPN is to construct a bidirectional information propagation graph, whose update process can be characterized by the following two ensemble formulas. For the first... BiFPN, in its top-down path, uses intermediate features obtained by fusing the input of the current layer with higher-resolution features from the previous layer. The generation combines the current layer input and the features from the previous layer (higher resolution): , In the formula, This indicates an upsampling operation to align the spatial scale. In the first iteration ( )for Subsequent iterations will incorporate the output from the previous round. This represents a feature fusion function used to adaptively fuse input feature maps of different resolutions or paths. In a bottom-up path, the output features... Further integrate intermediate features from the top-down path with features from the next layer (lower resolution): , In the formula, This indicates a downsampling operation.

[0036] Core fusion function Rapid normalization fusion and feature refinement were achieved: .

[0037] In this formula, and These represent the two input features to be fused. Learnable weights. The features are trained using unbounded values ​​and normalized using the sigmoid function during inference to adaptively measure the importance of features at different resolutions for leaf prediction at the current scale. This represents a very small positive number, used to prevent the denominator from being zero and to ensure numerical stability. This represents a learnable scaling factor used to control the contribution strength of the depthwise separable convolution (DWConv) residual terms. This design allows the model to dynamically adjust feature contributions: for severely occluded or blurred leaves, high-level semantic features (…) The weight of ) Increased to provide stronger contextual cues; for isolated leaves with well-defined edges, low-level detail features ( The weight of ) Increased to achieve precise segmentation. The residual term on the right side of the equation. Depthwise separable convolutions are introduced to enhance input features with minimal increase in computational cost. The spatial context modeling capability is crucial for distinguishing closely adjacent blade edges. BiFPN achieves iterative refinement through cross-layer connections, using the output of the previous round of BiFPN as part of the input of the current round. In the formula, A learnable scaling factor. This is achieved by stacking two layers (...). The aforementioned fusion units enable deep interaction of multi-scale features. Ultimately, BiFPN outputs an enhanced multi-scale feature pyramid that incorporates information from all scales. .

[0038] S5. Utilize the spatial attention submodule of the convolutional block attention module to process the multi-scale feature pyramid output by BiFPN. Each scale feature in the pyramid is recalibrated in terms of spatial dimensions to obtain the final refined feature pyramid. .

[0039] In this embodiment, a spatial attention submodule of the convolutional block attention module is introduced to recalibrate the spatial dimension of each scale feature output by BiFPN, further focusing on the image region where the leaf instance is located.

[0040] For the first BiFPN output Layer feature map The spatial attention weights are generated to highlight foreground pixels belonging to the leaf. First, global max pooling and global average pooling are performed along the channel dimension to obtain two spatial description maps. and They focus on the most salient feature responses and the overall feature distribution, respectively. Then, they are concatenated along the channel dimension and the information is integrated through a standard convolutional layer. , In the formula, Indicates the core size is (Set to 7 in this paper) Convolution operation is used to integrate local neighborhood information to perceive the local shape and edges of the leaf. The Sigmoid function is used to generate a two-dimensional spatial importance graph. The high-value regions precisely correspond to the pixel positions of the foreground leaves (including partially occluded leaves). Finally, the spatial importance map is element-wise multiplied with the input feature map to obtain the spatially refined feature map: , In the formula, This represents element-wise multiplication. By independently applying this spatial attention operation to each layer of features output by BiFPN, the final refined feature pyramid is obtained. This step complements the preceding SE step: SE performs "channel-dimensional feature selection" in the early stage, while CBAM spatial attention performs "spatial-dimensional region focusing" after fusion. The two work together to significantly improve the feature map's ability to distinguish foreground leaves, providing the instance segmentation head with highly discriminative and target-specific feature input.

[0041] S6. Based on refined feature pyramid A cascaded regional convolutional neural network is used as the decoding head, and the localization and pixel-level segmentation results of each detection blade are obtained through iterative optimization.

[0042] In this embodiment, a cascaded regional convolutional neural network is used as the decoding head. This architecture is iteratively optimized through a series of detection stages with progressively increasing quality. Its core lies in gradually increasing the intersection-over-union (IoU) threshold to resample training samples, so that the regressor and classifier can adaptively focus on positive samples that are more difficult to distinguish (such as severely overlapping leaves), thereby significantly improving the localization and segmentation accuracy in complex scenes.

[0043] In this embodiment, the cascaded optimization process can be characterized by a recursively defined formula. The refined feature pyramid is based on the S5 output. The final feature pyramid output .

[0044] First, an initial set of proposal boxes is generated on the feature pyramid using a Region Proposal Network (RPN). Subsequently, the first Cascaded stages ( ) Output proposal box from the previous stage As input, perform more refined detection and segmentation: In the formula, Represents the learnable detection head function for the m-th stage, with parameters as follows: . This represents the refined feature pyramid output by S5. The operation extracts a fixed size (e.g., from the feature pyramid) corresponding to each proposal box. Feature region. Function Internally, three subtasks are executed in parallel: 1) Bounding box regression to predict refined relative offsets. Iteratively update the bounding box ,in 1) Coordinate transformation applied according to standard R-CNN parameterization; 2) Target classification, outputting class probabilities. 3) Mask prediction, generating a low-resolution (e.g., Binary segmentation graph Each stage uses a progressively increasing IoU threshold. (like To define positive and negative samples, thus enabling the detector sequence... It specializes in handling proposals with increasing quality, which is crucial for accurately separating visually similar and closely adjacent blades. The entire cascaded structure is trained end-to-end by minimizing a multi-task loss function: In the formula, , and These are based on the current stage threshold. Assign a ground truth class label, bounding box regression target, and pixel-level ground truth segmentation mask to each proposal box. The standard cross-entropy classification loss is used. To smooth the L1 regression loss, The pixel-wise binary cross-entropy segmentation loss is calculated only on positive sample proposals. To balance the weighting coefficients of the various losses, the model can progressively calibrate and refine the initial region proposals through the aforementioned cascaded, multi-task optimization framework. Finally, the bounding boxes output by the third stage (the third detection and refinement stage in the cascaded region convolutional neural network R-CNN, i.e., m=3, IoU=0.7) are... With the mask that has been bilinearly upsampled to the original image resolution This combination provides precise localization and pixel-level segmentation results for each detected leaf instance, such as... Figure 2 As shown.

[0045] As shown in Table 1, quantitative comparisons on the P-leaf validation set demonstrate that the LeafSeg-T model for multi-scale leaf instance segmentation, based on the present invention, achieves the best performance in the joint task of instance segmentation and object detection. In the segmentation task, its mAP reaches 67.59; AP50 and AP75 are 87.35 and 68.12 respectively, indicating that the model can generate high-quality segmentation masks with accurate boundaries. In the detection task, its bounding box mAP is 64.77, also outperforming dedicated detection frameworks such as BoxInst. Compared with YOLOv8, LeafSeg-T improves segmentation mAP by 2.49 while maintaining comparable detection accuracy; compared with MaskDino, it comprehensively leads in both segmentation and detection metrics. The results verify the effectiveness of the proposed architecture in multi-task collaborative optimization and module combination, and confirm that LeafSeg-T has superior performance and robustness in complex leaf segmentation tasks.

[0046] Table 1 Example 2 This second embodiment provides a multi-scale leaf instance segmentation system for complex forest stands, mainly including an image data acquisition unit, a multi-scale feature extraction unit, a calibration unit, a fusion unit, a recalibration unit, and an optimization unit. Specifically: The image data acquisition unit is used to acquire image data of target trees using UAV remote sensing technology and construct a leaf segmentation dataset for forestry scenarios.

[0047] The multi-scale feature extraction unit is used to extract multi-scale features based on the leaf segmentation dataset using an improved Swing Transformer backbone network, generating an initial multi-scale feature pyramid consisting of four layers of feature maps suitable for multi-scale feature parsing of leaves. .

[0048] The calibration unit is used based on the initial multi-scale feature pyramid. The features are recalibrated along the channel dimension using a squeezing and incentive attention mechanism to obtain a channel-enhanced feature pyramid. .

[0049] Fusion unit for channel-enhanced feature pyramid By utilizing an improved bidirectional weighted feature pyramid network (BiFPN), multi-scale feature fusion is performed to generate a set of enhanced multi-scale feature pyramids that incorporate information from all scales. .

[0050] The recalibration unit is used to refine the multi-scale feature pyramid output by BiFPN by utilizing the spatial attention submodule of the convolutional block attention module. Each scale feature in the pyramid is recalibrated in terms of spatial dimensions to obtain the final refined feature pyramid. .

[0051] The optimization unit is used for refining the feature pyramid. A cascaded regional convolutional neural network is used as the decoding head, and the localization and pixel-level segmentation results of each detection blade are obtained through iterative optimization.

[0052] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multi-scale leaf instance segmentation method for complex forest stands, characterized in that, Includes the following steps: Using UAV remote sensing technology, image data of target trees are acquired to construct a leaf segmentation dataset for forestry scenarios; Based on the aforementioned leaf segmentation dataset, an improved Swing Transformer backbone network is used for multi-scale feature extraction, generating an initial multi-scale feature pyramid consisting of four layers of feature maps suitable for multi-scale leaf feature parsing. ; Based on the initial multi-scale feature pyramid The features are recalibrated along the channel dimension using a squeezing and incentive attention mechanism to obtain a channel-enhanced feature pyramid. ; Feature Pyramid Based on Channel Enhancement By utilizing an improved bidirectional weighted feature pyramid network (BiFPN), multi-scale feature fusion is performed to generate a set of enhanced multi-scale feature pyramids that incorporate information from all scales. ; The spatial attention submodule of the convolutional block attention module is used to process the multi-scale feature pyramid output by BiFPN. Each scale feature in the pyramid is recalibrated in terms of spatial dimensions to obtain the final refined feature pyramid. ; Based on the refined feature pyramid A cascaded regional convolutional neural network is used as the decoding head, and the localization and pixel-level segmentation results of each detection blade are obtained through iterative optimization.

2. The method according to claim 1, characterized in that: The improved Swing Transformer backbone network employs a window-based self-attention mechanism, including window-based multi-head self-attention (W-MSA) and shift-window-based multi-head self-attention (SW-MSA), as well as downsampling layers interspersed between the two. The switching between W-MSA and SW-MSA is achieved through a mask matrix.

3. The method according to claim 1, characterized in that: The features are recalibrated along the channel dimension using a squeeze and stimulated attention mechanism to obtain a channel-enhanced feature pyramid. include: For the initial multi-scale feature pyramid Feature map in Global average pooling is used to compress the spatial information of each channel to generate channel descriptors; By utilizing a two-layer fully connected network with a bottleneck structure, the channel descriptor learns the nonlinear interactions between channels, thereby obtaining the channel weight vector; The channel weight vector is multiplied channel by channel with the original feature map to obtain the calibrated feature map; the initial multi-scale feature pyramid is then processed. Each layer of features is recalibrated to obtain a channel-enhanced feature pyramid. .

4. The method according to claim 1, characterized in that: The improved bidirectional weighted feature pyramid network BiFPN, by constructing a bidirectional information propagation graph, utilizes bidirectional cross-scale information flow and learnable adaptive weights to achieve the fusion of features at different resolutions, thereby simultaneously capturing fine leaf vein texture and macroscopic leaf morphology. The update process of the bidirectional information propagation graph is divided into a top-down path and a bottom-up path. Among them, in the top-down path, features The generation combines the input of the current layer with the higher-resolution features of the previous layer, and in the bottom-up path, the output features... Further integrate the intermediate features from the top-down path with the lower-resolution features of the next layer; fusion function for: ； BiFPN achieves iteration through cross-layer connections.

5. The method according to claim 1, characterized in that: The spatial attention submodule of the convolutional block attention module is used to process the multi-scale feature pyramid output by BiFPN. The spatial dimension recalibration of each scale feature includes: First, the multi-scale feature pyramid For each scale feature in the image, global max pooling and global average pooling are performed along the channel dimension to obtain two spatial description maps. and Then, the two are concatenated along the channel dimension and integrated through a standard convolutional layer to generate a two-dimensional spatial importance map. Finally, the spatial importance map is multiplied element-wise with the input feature map to obtain the spatially refined feature map. .

6. The method according to claim 1, characterized in that: Methods for obtaining the localization and pixel-level segmentation results of each detected leaf by using a cascaded region convolutional neural network as the decoding head include: For refined feature pyramids Using the Region Proposal Network (RPN) to refine the feature pyramid Generate an initial set of proposal boxes. Subsequently, the Each cascade stage is the output proposal box of the previous stage. As input, perform more refined detection and segmentation: ， In the formula, Represents the learnable detection head function for the m-th stage, with parameters as follows: , The operation extracts a fixed-size feature region corresponding to each proposal box from the feature pyramid; function Internally, three subtasks are executed in parallel: bounding box regression, object classification, and mask prediction. Each subtask uses a progressively increasing IoU threshold. The positive and negative samples are defined; the bounding box of the mask prediction output is combined with the mask that has been bilinearly upsampled to the original image resolution to obtain the localization and segmentation result of each detected leaf instance.

7. A multi-scale leaf instance segmentation system for complex forest stands, wherein the system applies the method described in any one of claims 1-6, characterized in that, It includes an image data acquisition unit, a multi-scale feature extraction unit, a calibration unit, a fusion unit, a recalibration unit, and an optimization unit; The image data acquisition unit is used to acquire image data of target trees using UAV remote sensing technology and construct a leaf segmentation dataset for forestry scenarios. The multi-scale feature extraction unit is used to extract multi-scale features based on the leaf segmentation dataset using an improved SwinTransformer backbone network, generating an initial multi-scale feature pyramid consisting of four layers of feature maps suitable for multi-scale feature parsing of leaves. ; The calibration unit is used to base the initial multi-scale feature pyramid. The features are recalibrated along the channel dimension using a squeezing and incentive attention mechanism to obtain a channel-enhanced feature pyramid. ; The fusion unit is used for feature pyramids based on channel enhancement. By utilizing an improved bidirectional weighted feature pyramid network (BiFPN), multi-scale feature fusion is performed to generate a set of enhanced multi-scale feature pyramids that incorporate information from all scales. ; The recalibration unit is used to process the multi-scale feature pyramid output by BiFPN using the spatial attention submodule of the convolutional block attention module. Each scale feature in the pyramid is recalibrated in terms of spatial dimensions to obtain the final refined feature pyramid. ; The optimization unit is used to optimize based on the refined feature pyramid. A cascaded regional convolutional neural network is used as the decoding head, and the localization and pixel-level segmentation results of each detection blade are obtained through iterative optimization.

Images