A plant leaf fine segmentation method based on two-layer convolutional network and mask refinement

Abstract

The present invention discloses a method for fine segmentation of plant leaves based on a double-layer convolutional network and mask refinement, which realizes end-to-end fine segmentation of plant leaves obtained in a natural environment. The network of the present invention consists of three main parts: 1) a bottom-up path enhancement network, which transfers more shallow feature information to deeper features to generate higher-quality feature maps. 2) a double-layer convolutional network based on Mask R-CNN, which is used to decouple occluding leaves and occluded leaves; 3) a mask refinement module, which refines the leaf edges through a point selection strategy and a multi-layer perceptron. The present invention combines the feature-enhanced double-layer convolutional network with the mask refinement module, effectively alleviating the problems of inaccurate segmentation caused by occlusion overlap and complex leaf edge shapes in natural environments.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a plant leaf fine segmentation method based on a double-layer convolutional network and mask refinement. Background Art

[0002] Plant phenotype refers to the physical, physiological, and biochemical properties that reflect the structural and functional characteristics of plant cells, tissues, organs, plants, and populations. Plant phenotypes have deeply explored the inherent relationship between "genotype, phenotype, and environmental type," revealing the mechanisms underlying plant trait formation and significantly advancing the development of functional genomics, crop molecular breeding, and efficient cultivation. Traditional methods of acquiring phenotypic data are time-consuming and labor-intensive. With the advancement of imaging technology, non-invasive, rapid, and high-precision plant phenotypic image acquisition has become mature. However, for high-throughput plant phenotypic data, various feature extraction methods need to be developed.

[0003] In phenotypic feature extraction, fine segmentation of plant leaves remains a challenging task. This is particularly true in natural environments with complex backgrounds. Leaf segmentation presents numerous difficulties, such as interference from factors like lighting and shadows, occlusion between leaves, and blurred leaf edges. These factors all contribute to variations in leaf color, shape, and size depending on the time and angle of image acquisition. Therefore, new solutions are urgently needed for the fine segmentation of single plant leaves captured in natural environments. Summary of the Invention

[0004] In response to the problems existing in the prior art, the present invention provides a plant leaf fine segmentation method based on a double-layer convolutional network and mask refinement, which realizes end-to-end segmentation of plant leaves and improves the accuracy of plant leaf segmentation obtained in a natural environment.

[0005] The purpose of the present invention is achieved through the following technical solutions.

[0006] A plant leaf fine segmentation method based on a double-layer convolutional network and mask refinement includes the following steps:

[0007] 1) A bottom-up path enhancement network that transfers shallow feature information to deeper features to generate higher quality feature maps;

[0008] 2) A two-layer convolutional network based on Mask R-CNN to decouple occluding and occluded leaves;

[0009] 3) Mask refinement module, which refines leaf edges through point selection strategy and multi-layer perceptron.

[0010] Step 1) specifically involves using the Faster R-CNN object detector to detect and generate leaf bounding boxes. A mask branch is then used to predict a foreground mask within each detection box. A ResNet-based FPN network is then added after the FPN network with a bottom-up path enhancement. This path enhancement starts at the bottom layer, P2, and gradually downsamples P2 through P5 to obtain {N2, N3, N4, N5}, where N2 and P2 are the same feature map. During this process, no more than 10 layers of feature transfer are performed.

[0011] The step 2) specifically includes: modeling the occluding instance and the occluded instance through a two-layer convolutional network, and explicitly decoupling the boundaries of the occluding instance and the occluded instance: for the region of interest after ROI Align, one of the occlusion-aware branches detects the occlusion contour and mask, explicitly models the occluded area, extracts the shape and position information of the occluder, and guides another occlusion-aware branch to detect the contour and mask of the occluded area through the residual structure, wherein each occlusion-aware branch consists of four 3×3 convolutional layers with a stride of 1.

[0012] The step 3) introduces an edge refinement module, which first iteratively upsamples the uncertain pixels and then predicts the category of the pixels. The edge refinement module includes three parts: point sampling, point-by-point representation, and nodding.

[0013] The step 3) specifically includes:

[0014] 3-1) During point sampling training, a non-iterative random sampling method is used to train the model. The points are selected using oversampling rate and importance. During inference, an iterative upsampling method is used to classify pixels from coarse to fine. In each iteration, the previously predicted mask is first upsampled using bilinear interpolation. Then, the N most uncertain points are selected on the higher-resolution image. The point selection strategy is the same as that during training. The point-by-point representation of the pixels is then calculated and their categories are predicted. The above operation is repeated five times to obtain a set of high-resolution feature maps containing leaf edge details, which are 56×56, 112×112, 224×224, 448×448, and 896×896 respectively.

[0015] 3-2) In the point-by-point representation, the edge refinement module classifies uncertain pixels by combining fine-grained and coarse-grained features to form a point feature vector. The fine-grained features come from the N2 feature map in the bottom-up path enhancement, which contains rich detail information, while the coarse-grained features come from the mask of the occluded object predicted by the double-layer occlusion module, providing additional position information.

[0016] 3-3) The edge refinement module in the nod uses a simple multi-layer perceptron for point-by-point prediction and classification, and the multi-layer perceptron is a globally shared multi-layer perceptron.

[0017] Compared with the existing technology, the advantages of the present invention are: 1. A new two-stage instance segmentation network is constructed, called LSNet (Leaf Segmentation Net).

[0018] 2. By transferring more shallow features to the deep network through a bottom-up path enhancement network, the accuracy of leaf segmentation can be improved.

[0019] 3. After the network extracts the ROI, the occluding leaves and the occluded leaves are segmented at the same time, and the occluding leaves and the occluded leaves are decoupled to improve the segmentation accuracy of the leaves.

[0020] 4. Mask Refining Module: This module adaptively selects uncertain pixels in the instance object and combines them with fine-grained features in the feature pyramid to refine the leaf mask, especially the leaf edge. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the network structure of the present invention.

[0022] Figure 2 It is a schematic diagram of the bottom-up path enhancement structure.

[0023] Figure 3 It is a schematic diagram of point sampling form. DETAILED DESCRIPTION

[0024] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] A plant leaf fine segmentation method based on a double-layer convolutional network and mask refinement includes the following steps:

[0026] Step 1: Bottom-up path enhancement network, transfers more shallow feature information to deeper features, and generates higher quality feature maps. The Faster R-CNN target detector is used to detect the bounding box of the generated leaf, and then the mask branch is used to predict the foreground mask within each detection box. In the FPN network based on ResNet, during the bottom-up transmission of features, after dozens or even hundreds of convolutional layers, shallow features are often lost or offset. Therefore, this paper adds a bottom-up path enhancement (BPA) after FPN. The structure starts from the bottom layer P2, and gradually downsamples P2 to P5 to obtain {N2, N3, N4, N5}, where N2 and P2 are the same feature map. In this process, the number of layers of feature transmission is less than 10, which can reduce the loss of shallow feature information during the transmission process. The specific generation method of the feature map {N3, N4, N5} is as follows Figure 2 As shown in the figure, Ni passes through a 3×3 convolutional layer with a stride of 2, reducing the feature map to half of its original size. The resulting feature map is then element-wise added to Pi+1, and the resulting features are passed through another 3×3 convolutional layer with a stride of 1, resulting in the feature map Ni+1.

[0027] Step 2: A two-layer convolutional network based on Mask R-CNN is used to decouple occluding leaves and occluded leaves; an end-to-end two-stage instance segmentation model, wherein step 2) specifically includes: modeling occluding instances and occluded instances through a two-layer convolutional network, and explicitly decoupling the boundaries of occluding instances and occluded instances: for the region of interest after ROI Align, one occlusion-aware branch detects the occlusion contour and mask, explicitly models the occluded area (occluder), extracts the shape and position information of the occluder, and guides another occlusion-aware branch to detect the contour and mask of the occluded area through a residual structure, wherein each occlusion-aware branch consists of four 3×3 convolutional layers with a stride of 1.

[0028] Step 3: Mask Refinement Module: This module refines the leaf edges using a point selection strategy and a multilayer perceptron (MLP). This module introduces an edge refinement module that iteratively upsamples uncertain pixels and then predicts their categories, achieving high-quality mask segmentation. This module consists of three main components: point selection, point-wise representation, and point head.

[0029] Step 3-1: When training point sampling, first set a group of waiting points {Seti}, this set is used to store the most uncertain points (for example, the probability of the binary mask is close to 0.5). Since the iterative upsampling method is not easy to train, a non-iterative random sampling method is used to train the model during training. The points are selected using the oversampling rate k (k>1) and the importance β (0<β<1). For each instance, kN points are selected to form the optional point set U i , and then from U i Select βN most uncertain points from the U, and the remaining (1-β)N points from the remaining U i During inference, iterative upsampling is used to classify pixels from coarse to fine. In each iteration, bilinear interpolation is first used to upsample the previously predicted mask, and then the most N uncertain points are selected on the higher resolution image. The point selection strategy is the same as that during training. Then the point-by-point representation of the pixels is calculated and their categories are predicted. Repeating the above operation five times, a set of high-resolution feature maps containing leaf edge details can be obtained, which are 56×56, 112×112, 224×224, 448×448 and 896×896 respectively. The sampling form is as follows Figure 3 shown.

[0030] Step 3-2: In the point-by-point representation, the edge refinement module classifies uncertain pixels by combining fine-grained features and coarse-grained predicted features to form a point feature vector. The fine-grained features come from the N2 feature map in the bottom-up path enhancement, which contains rich detail information, while the coarse-grained features come from the mask of the occluded object predicted by the double-layer occlusion module, providing additional position information.

[0031] Step 3-3: The edge refinement module in the nod uses a simple multi-layer perceptron (MLP) for point-by-point prediction and classification, which is a globally shared MLP.

[0032] Furthermore, for the end-to-end two-stage instance segmentation model, the multi-task loss function defined by the present invention during training is:

[0033] Total_Loss=Loss rpn_cls +Loss rpn _ loc +Loss cls +Loss box_reg +Loss mask +Loss mask_point (1)

[0034] Loss mask =Loss Occluder +LossOc (2)

[0035] Loss Occluder =λ1Loss′ Occ_B +λ2Loss (3)

[0036] Loss Occludcc =λ3Loss Occ_B +λ4Loss (4)

[0037] Among them, Loss rpn_cls 、Loss rpn_loc 、Loss cls and Loss box_reg The definition is the same as Mask R-CNN, Loss mask_point Use binary cross entropy loss. Loss' Occ_B 、Loss' Occ_S 、Loss Occ_B and Loss Occ_S They represent the edge detection loss for occluded instances, the mask segmentation loss for occluded instances, the edge detection loss for occluded instances, and the mask segmentation loss for occluded instances, respectively. All are implemented using binary cross-entropy loss. To balance the various loss functions, the hyperparameters are designed as follows: λ1 = 0.5, λ2 = 0.25, λ3 = 0.5, and λ4 = 1.0.

[0038] The above descriptions are only some embodiments of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principles of the present invention. These improvements should be considered as within the scope of protection of the present invention.

Claims

1. A plant leaf fine segmentation method based on a double-layer convolutional network and mask refinement, characterized in that: The following steps are involved: 1) A bottom-up path enhancement network that transfers shallow feature information to deeper features to generate higher quality feature maps; 2) A two-layer convolutional network based on Mask R-CNN to decouple occluding and occluded leaves; 3) Mask refinement module, which refines leaf edges through point selection strategy and multi-layer perceptron; The step 3) specifically includes: 3-1) During point sampling training, a non-iterative random sampling method is used to train the model. The points are selected using oversampling rate and importance. During inference, iterative upsampling is used to classify pixels from coarse to fine. In each iteration, the previously predicted mask is first upsampled using bilinear interpolation, and then the N most uncertain points are selected on the higher resolution image. The point selection strategy is the same as that during training. The point-by-point representation of the pixels is calculated and their categories are predicted. After five iterations, a set of high-resolution feature maps containing leaf edge details can be obtained, which are 56×56, 112×112, 224×224, 448×448 and 896×896 respectively; 3-2) In the point-by-point representation, the edge refinement module judges and classifies uncertain pixels by combining the point feature vector composed of fine-grained features and coarse-grained features. The fine-grained features come from the N2 feature map in the bottom-up path enhancement, which contains rich detail information, while the coarse-grained features come from the mask of the occluded object predicted by the double-layer occlusion module, providing additional position information; 3-3) The edge refinement module in the nod uses a simple multi-layer perceptron for point-by-point prediction and classification, and the multi-layer perceptron is a globally shared multi-layer perceptron.

2. The plant leaf fine segmentation method based on a double-layer convolutional network and mask refinement according to claim 1, characterized in that: Step 1) specifically includes: using the Faster R-CNN object detector to detect and generate the bounding box of the leaf, then using the mask branch to predict the foreground mask within each detection box, and adding a bottom-up path enhancement after the ResNet-based FPN network. The path enhancement starts from the bottom layer P2 and gradually downsamples P2 to P5 to obtain {N2, N3, N4, N5}, where N2 and P2 are the same feature map. In this process, the number of layers of feature transfer does not exceed 10.

3. The plant leaf fine segmentation method based on a double-layer convolutional network and mask refinement according to claim 1, characterized in that: The step 2) specifically includes: modeling the occluding instance and the occluded instance through a two-layer convolutional network, and explicitly decoupling the boundaries of the occluding instance and the occluded instance: for the region of interest after ROI Align, one of the occlusion-aware branches detects the occlusion contour and mask, explicitly models the occluded area, extracts the shape and position information of the occluder, and guides another occlusion-aware branch to detect the contour and mask of the occluded area through the residual structure, wherein each occlusion-aware branch consists of four 3×3 convolutional layers with a stride of 1.

4. The plant leaf fine segmentation method based on a double-layer convolutional network and mask refinement according to claim 1, characterized in that: The step 3) introduces an edge refinement module, which first iteratively upsamples the uncertain pixels and then predicts the category of the pixels. The edge refinement module includes three parts: point sampling, point-by-point representation, and nodding.

Images