Tobacco stem and leaf image segmentation method

Abstract

The invention provides a tobacco stem and leaf image segmentation method, and relates to the technical field of image processing, and the method comprises the steps: obtaining tobacco stem and leaf image data based on control environment collection and field environment collection; preprocessing the obtained tobacco stem and leaf image, and dividing the tobacco stem and leaf image into a training set, a test set and a verification set; constructing an improved deep learning model based on U-Net, and training, testing and verifying the improved deep learning model by using the training set, the test set and the verification set; and segmenting the tobacco stem and leaf image by using the trained, tested and verified deep learning model. The invention solves the problems in the prior art in the aspects of precise segmentation of slender structures (such as stalks) and handling of serious class imbalance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a tobacco stem and leaf image segmentation method. BACKGROUND

[0002] Under the background of modern agricultural informatization, accurately obtaining the phenotype information of tobacco plants is of great significance for optimizing planting management and improving yield and quality. In particular, the leaf and stem ratio, leaf area index, and stem thickness of tobacco are directly related to the growth state assessment and final quality of tobacco. Accurate extraction of these features has become a key link in the precise management of tobacco.

[0003] However, tobacco plant segmentation faces unique challenges. Tobacco stem pixels usually account for only 2%-3% of the total image pixels, much lower than the proportion of leaf and background pixels. This extreme imbalance makes it easy for the model to overlook the key stem structure. The morphological structure of tobacco plants also increases the difficulty of segmentation. The stem is slender and vertical, and the leaf is wide and often irregular in shape, with the two being distributed in space. The plant undergoes significant morphological changes during growth, with the stem developing from a single main stem in the seedling stage to a complex branching system in the mature stage, requiring the segmentation model to have the ability to adapt to multi-scale structures. Field environmental factors such as uneven light, shadow effects, background complexity, and plant overlap make the segmentation task more complex. The color of tobacco stems and leaves is similar under certain lighting conditions, and the boundary is blurred, making it difficult to accurately distinguish between them relying on simple color features. These challenges make tobacco stem and leaf segmentation a research topic in the field of computer vision that has both scientific significance and practical application value. Accurate tobacco stem and leaf segmentation not only supports early detection of diseases and pests, dynamic monitoring of growth status, but also provides key data support for breeding research, yield prediction, and intelligent agricultural machinery development, with significant economic and scientific value.

[0004] Deep learning technology has provided powerful capabilities for agricultural image analysis, and the U-Net model, as a classic encoder-decoder architecture, has been successful in various plant segmentation tasks. However, there is relatively little research on the segmentation of tobacco plant leaves and stems. In the field of tobacco image analysis, improvements have been made to the U-Net model for shape feature extraction of tobacco leaf main veins, but the stem has not been handled independently. This research gap reflects the unique difficulty of tobacco stem and leaf segmentation, especially when dealing with such low-proportion slender structures as stems.

[0005] Current semantic segmentation network architecture has achieved remarkable results in the field of agriculture, among which U-Net is particularly suitable for processing detailed medical and agricultural images due to its symmetrical encoder-decoder structure and skip connection mechanism. However, the standard U-Net has limitations in processing tobacco images. Its relatively simple encoder structure is difficult to extract the complex multi-level morphological features of tobacco plants. The non-differentiability of the traditional ReLU activation function at zero causes gradient jumps, hindering the learning of subtle texture features. The conventional cross-entropy loss function is prone to bias towards the dominant class when dealing with class-imbalanced tobacco images. The standard U-Net also lacks effective modeling of spatial location and directional features, making it difficult to distinguish between stems and leaf structures with directional differences. These limitations make the existing method's performance in tobacco leaf and stem segmentation, especially in slender stem segmentation, far from meeting the requirements of practical applications. SUMMARY

[0006] In view of the above deficiencies in the prior art, the present application provides a tobacco stem and leaf image segmentation method. The method improves the segmentation accuracy of targets such as stems and leaves in tobacco images, especially in complex backgrounds and class imbalance conditions, while enhancing the robustness and generalization ability of the model, solving the problem of accurately segmenting slender structures such as stems and dealing with severe class imbalance in the prior art.

[0007] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a tobacco stem and leaf image segmentation method, comprising the following steps: S1, based on the control environment acquisition and the field environment acquisition, acquiring tobacco stem and leaf image data; S2, preprocessing the acquired tobacco stem and leaf image, and dividing it into a training set, a test set and a validation set; S3, constructing an improved deep learning model based on U-Net, and training, testing and verifying the improved deep learning model using the training set, the test set and the validation set; S4, using the trained, tested and verified deep learning model to segment the tobacco stem and leaf image.

[0008] Further, the improved deep learning model includes an encoder, a decoder connected to the encoder via a skip connection, and an output layer connected to the decoder; The encoder is used for hierarchical feature extraction of the tobacco stem and leaf image; The decoder is used to restore the spatial resolution of the feature map based on the feature extraction result of the encoder; The output layer is used to map the feature map output by the decoder into three channels and generate a tobacco stem and leaf image segmentation probability map consistent with the original size to obtain the final segmentation result, wherein the three channels correspond to background, tobacco leaf and stem respectively.

[0009] Further, the encoder adopts ResNet50 as a backbone network, includes five encoding stages of a first encoding stage Stage0 to a fifth encoding stage Stage4 for downsampling, and the output ends of the five encoding stages are all connected with CBAM attention modules, wherein the first encoding stage Stage0 is used as an input port to obtain tobacco stem leaf image features through downsampling, the second encoding stage Stage1 to the fifth encoding stage Stage4 perform deep feature extraction on the tobacco stem leaf image by stacking residual bottleneck modules, and the first encoding stage Stage0 to the fifth encoding stage Stage4 are connected in sequence; The decoder includes four upsampling stages, wherein the input of each upsampling stage includes the upsampled result of the previous sampling stage of the current upsampling stage and the tobacco stem leaf feature map enhanced by the CBAM attention module, and the high-dimensional tobacco stem leaf image feature formed by the jump splicing, wherein the convolution block containing 3*3 convolution and 1*1 convolution is used to fuse the spliced high-dimensional tobacco stem leaf image feature, and the Mish activation function is arranged in the convolution block.

[0010] Further, the first encoding stage Stage0 includes a 7*7 convolution kernel and a maximum pooling layer group; the second encoding stage Stage1 to the fifth encoding stage Stage4 respectively include 3 residual bottleneck modules, 4 residual bottleneck modules, 6 residual bottleneck modules and 3 residual bottleneck modules.

[0011] Further, the CBAM attention module is sequentially connected with a channel attention submodule and a spatial attention submodule; The expression of the channel attention submodule is as follows: ; Wherein, represents a channel attention map, represents the final channel attention weight generated by the Sigmoid function, represents the input tobacco stem leaf feature map, represents the Sigmoid function, represents the multi-layer perception operation, represents the global average pooling operation, represents the global maximum pooling operation; The expression of the spatial attention submodule is as follows: ; Wherein, represents the final spatial attention weight generated by the Sigmoid function, represents the channel weighted feature map generated by the channel attention submodule, Spatial attention map represents a 7*7 convolution operation.

[0012] Further, the expression of the Mish activation function is as follows: ; ; wherein, represents the output of the Mish activation function, represents a self-regularization non-monotonic activation function, represents the input tobacco stem leaf feature map, represents a tangent function, represents a Softplus function, represents an exponential function with the natural constant e as the base.

[0013] Further, the expression of the loss function of the improved deep learning model is as follows: ; ; ; ; wherein, represents the loss function of the deep learning model, and both represent weight coefficients, represents the Dice loss, represents the focal loss, represents the Dice coefficient, i represents each pixel in the tobacco stem leaf image, represents the probability that the deep learning model predicts that the pixel i belongs to the foreground, represents the real label of the pixel i, represents the prediction probability of the deep learning model for the real class, represents a focal parameter.

[0014] The beneficial effects of the present application are: (1) The innovation of the present application lies in the organic combination of Unet architecture, powerful ResNet50 backbone network, advanced CBAM attention module, efficient Mish activation function and mixed loss function of Dice Loss and Focal Loss optimized for class imbalance. This combination is specially designed for the characteristics of tobacco stem and leaf segmentation task (such as target morphology, long and thin structure difficult to segment, class ratio imbalance, etc.). Through the synergistic effect of these key technologies, the accuracy and robustness of tobacco stem and leaf image segmentation are significantly improved, especially in accurately identifying and segmenting small stems, which provides effective technical support for crop phenotype analysis and growth state monitoring in precision agriculture.

[0015] (2) The present application creates a tobacco segmentation dataset containing 2001 high-quality labeled images through the construction of a special tobacco stem and leaf segmentation dataset, covering diverse data from laboratory controlled environment to field complex scene, providing basic resources for tobacco image analysis research.

[0016] (3) The present application designs a segmentation architecture for tobacco morphological characteristics, and proposes an improved Unet framework, which uses ResNet50 as the backbone network and integrates CBAM attention module and Mish activation function, forming a segmentation system optimized for tobacco plant features.

[0017] (4) The present application develops an optimization strategy for class imbalance. A mixed loss function combining Dice Loss and Focal Loss is designed, which effectively alleviates the problem of extreme class imbalance in tobacco images and significantly improves the stem recognition accuracy.

[0018] (5) The present application realizes high-precision segmentation of slender structures. Through experimental verification, the proposed framework shows high precision and effectiveness in the segmentation of slender structures such as stems, proving its potential in solving the problem of slender structure segmentation in complex agricultural scenes.

[0019] (6) The present application establishes the theoretical correspondence between morphological features and network components, systematically analyzes the correspondence between the morphological features of tobacco plants and new network components (such as ResNet50 backbone network, CBAM attention module), and explains why this combination can effectively solve the problem of tobacco segmentation, providing a methodological framework for similar crop segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0020] Fig. 1 The flowchart of the method of the present application.

[0021] Fig. 2 The improved deep learning model structure diagram based on U-Net in the present application. DETAILED DESCRIPTION

[0022] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0023] EMBODIMENT The present application significantly improves the accuracy and robustness of tobacco stem and leaf segmentation through the synergistic effect of Unet-based model architecture and ResNet50 backbone network and other key technologies. The present application is mainly applied to a deep learning platform and can run in a server or high-performance computing environment with GPU acceleration function. The main configuration of the hardware environment includes: GPU: NVIDIA series (such as Tesla V100 or RTX 3080) for accelerating the training and inference process of deep learning models.

[0024] CPU: Intel Xeon series for data processing and model inference.

[0025] Storage device: SSD hard disk for storing corn seed image dataset and training model parameters.

[0026] The implementation logic is as shown in Figs. 1-2 The present application provides a tobacco stem and leaf image segmentation method, and the implementation method is as follows: S1, based on control environment acquisition and field environment acquisition, acquiring tobacco stem and leaf image data; In this embodiment, the tobacco stem and leaf image data acquisition adopts a double-environment complementary strategy, including control environment acquisition and field environment acquisition. The control environment data is obtained under laboratory conditions, the tobacco plants are placed according to different arrangement combinations, a high-resolution mobile phone camera (≥ 30 million pixels) is used to shoot under standard lighting conditions (400-700 lux), and the image clarity and perspective diversity are ensured. Field environment data is directly collected in tobacco planting fields, covering different growth stages, various lighting conditions (overcast, sunny, evening, etc.), and complex background environments (such as other vegetation, soil, etc.).

[0027] S2, preprocessing the acquired tobacco stem and leaf image, and dividing it into training set, test set and validation set; In this embodiment, the preprocessing includes normalization, size adjustment and data enhancement processing of the tobacco stem leaf image. The normalization is used to standardize the pixel value, and the adjustment of the size of the tobacco stem leaf image ensures that the input model has a consistent size. Data augmentation (such as flipping, rotation, scaling, etc.) is used to expand the data set, increase sample diversity, and improve the generalization ability of the model.

[0028] In this embodiment, the preprocessed data is divided into a training set, a validation set and a test set, and the typical ratio is 7:2:1. The training set is used to train the deep learning model, the validation set is used to monitor the performance of the deep learning model, and the test set is used for final evaluation.

[0029] S3, an improved deep learning model based on U-Net is constructed, and the training set, the test set and the validation set are used to train, test and verify the improved deep learning model; In this embodiment, the improved deep learning model includes an encoder, a decoder connected with the encoder by jumping, and an output layer connected with the decoder; The encoder is used for hierarchical feature extraction of the tobacco stem leaf image; The decoder is used to restore the spatial resolution of the feature map based on the feature extraction result of the encoder; The output layer is used to map the feature map output by the decoder into three channels, and generate a tobacco stem leaf image segmentation probability map consistent with the original size to obtain the final segmentation result, wherein the three channels correspond to background, tobacco leaf and stem respectively.

[0030] In this embodiment, the encoder uses ResNet50 as the backbone network, including five encoding stages of down-sampling composed of the first encoding stage Stage 0 to the fifth encoding stage Stage 4, and the output ends of the five encoding stages are all connected with CBAM attention modules, wherein the first encoding stage Stage 0 is used as the input port, the second encoding stage Stage 1 to the fifth encoding stage Stage 4 are used to extract deep features of the tobacco stem leaf image by stacking residual bottleneck modules, and the first encoding stage Stage 0 to the fifth encoding stage Stage 4 are connected in turn; The decoder includes four up-sampling stages, wherein the input of each up-sampling stage includes the up-sampling result of the previous up-sampling stage and the tobacco stem leaf feature map enhanced by the CBAM attention module, and the high-dimensional tobacco stem leaf image feature formed by the jump splicing, wherein the convolution block containing 3x3 convolution and 1x1 convolution is used to fuse the spliced high-dimensional tobacco stem leaf image feature, and the Mish activation function is set in the convolution block.

[0031] In this embodiment, the improved deep learning model based on U-Net is specifically optimized for the special requirements of tobacco leaf and stem segmentation task. The framework integrates ResNet50 backbone network, transfer learning strategy, CBAM (Convolutional Block Attention Module) attention mechanism, Mish activation function and mixed loss strategy of Dice and Focal loss function, forming an end-to-end efficient segmentation system, as shown in Fig. 2

[0032] In this embodiment, the improved deep learning model retains the encoder-decoder symmetric structure of the classic U-Net, but makes important improvements in key components. As shown in Fig. 2

[0033] The encoding path uses ResNet50 as the backbone network, which is composed of five stages (first encoding stage Stage 0 to fifth encoding stage Stage 4) and is responsible for hierarchical feature extraction. The first encoding stage Stage 0 as the input port includes a 7x7 large convolution kernel (step size 2) and a max pooling layer, which is used for preliminary down-sampling and capturing basic features. The second encoding stage Stage 1 to the fifth encoding stage Stage 4 gradually increases the number of channels (from 64 to 256, 512, 1024, 2048) and reduces the resolution of the feature map by stacking residual bottleneck modules, achieving deep feature extraction from low-level texture to high-level semantics. According to the standard design of ResNet50, the second encoding stage Stage 1 to the fifth encoding stage Stage 4 respectively contains 3 residual bottleneck modules, 4 residual bottleneck modules, 6 residual bottleneck modules and 3 residual bottleneck modules. A key improvement is that CBAM attention modules are connected after the output of each encoding stage to enhance the network's perception of key feature channels and spatial positions. This deep structure can provide rich multi-scale feature representation for accurate segmentation of tobacco leaves and stems. Among them, the classic U-net is used as the basic segmentation framework, and the pre-trained ResNet50 is used as a powerful backbone network to effectively extract complex features in tobacco images through its deep residual structure. The combination of encoder-decoder structure and skip connection helps to fuse multi-scale information and preserve spatial details, laying the foundation for accurate segmentation.

[0034] ​​The decoding path is symmetrical to the encoding path and contains four upsampling stages, aiming to progressively restore the spatial resolution of the feature maps and refine segmentation details. The input to each decoding stage is formed by concatenating the upsampling result from the previous stage with feature maps from the corresponding stage of the encoder, enhanced by the CBAM attention module. For example... Fig. 2 As shown, the number of channels after concatenation are 1024+1024, 512+512, 256+256, and 64+64, respectively. Subsequently, the concatenated high-dimensional features are fused and refined using convolutional blocks containing 3×3 and 1×1 convolutions, reducing the number of channels to 1024, 512, 256, and 64. In these convolutional blocks, the Mish activation function is used instead of the traditional ReLU to improve the network's nonlinear modeling ability and the smoothness of gradient flow.

[0035] Fig. 2 The key modules and symbols are explained below: The CBAM attention module consists of two concatenated sub-modules: Channel Attention and Spatial Attention. Its workflow is as follows: The input tobacco stem and leaf feature map first passes through the channel attention submodule, which learns the importance weights of different channels and weights the original tobacco stem and leaf feature map accordingly. Then, the result is fed into the spatial attention submodule, which learns the importance of different spatial locations and further enhances the features. In this way, the CBAM attention module achieves focused attention on both the channel and spatial dimensions. Here, Input represents the input, Output represents the output, and the + symbol indicates element-wise addition in the residual connection, which adds the output of the CBAM attention module to the input, ensuring that the attention module enhances rather than completely replaces the original features.

[0036] Fig. 2 The numbers (3, 64, 256, 512, 1024, and 2048) represent the number of channels in the tobacco stem and leaf feature map at different network depths. Convolution 3x3 / 1x1 indicates that the size of the convolution kernel is 3x3 or 1x1. Max pooling 2x2 / upsampling 2x2 indicate that the window size of pooling / upsampling is 2x2, respectively. Yellow squares indicate block duplication, red arrows indicate skip connections, ResNet50 indicates the backbone network, Stage 0 to Stage 4 indicate the first encoding stage to the fifth encoding stage, respectively, and CBAM indicates the CBAM attention module.

[0037] Particularly noteworthy is that the deep learning model employs a multi-level skip connection mechanism between the encoder and the decoder. As shown by the red arrows in FIG. 1, Fig. 2 As shown by the red arrows in FIG. 1,

[0038] In this embodiment, in the U-Net and its variant architecture, the "corresponding" relationship between the encoder and the decoder at the corresponding stage is mainly realized through the core mechanism of skip connection. The specific description is as follows: 1. Function definition: skip connection is a structure that directly transmits and fuses (usually splices) the tobacco stem and leaf feature maps of different levels in the encoder to the corresponding up-sampling stage in the decoder.

[0039] 2. Theoretical basis: the encoder gradually extracts high-level, abstract semantic features during the down-sampling process through a series of convolution and pooling operations, which is crucial for understanding the content of the tobacco stem and leaf image. However, this process inevitably loses a large amount of low-level spatial detail information, such as the precise boundaries of the target, texture, etc. The task of the decoder is to gradually up-sample the highly abstract semantic feature maps to the resolution of the original tobacco stem and leaf image, thereby realizing pixel-level prediction. If only relying on the deepest features of the encoder, the decoder will have difficulty in reconstructing accurate details.

[0040] 3. Core role: the introduction of skip connection aims to solve the above information loss problem. It effectively fuses the shallow features in the encoder that retain rich spatial details with the deep features in the decoder that contain rich semantic information after up-sampling. This combination of multi-scale features enables the decoder to fully utilize the fine positioning clues from the encoder while restoring the image size, thereby significantly improving the accuracy of the segmentation results, especially in the prediction of target edges.

[0041] Therefore, this "corresponding" relationship is the key to the high-precision semantic segmentation of the U-Net architecture, which ensures that the network can simultaneously consider both "what" (semantic information) and "where" (spatial information) when making the final output.

[0042] In this embodiment, the encoder contains five stages, while the decoder contains four stages of asymmetric design, which is a classic and efficient configuration in the U-Net architecture. The corresponding relationship is not a simple layer-by-layer mapping, but is developed around the bottleneck layer at the deepest part of the network. The specific corresponding way is as follows: 1. Encoder (Contracting Path): This path consists of five feature extraction stages (e.g., the first encoding stage Stage 0 to the fifth encoding stage Stage 4 based on ResNet50). As the stage deepens, the spatial resolution of the tobacco stem feature map is systematically reduced (down-sampling), while its channel dimension is correspondingly increased, forming a hierarchical feature pyramid. The tobacco stem feature maps output by the five encoding stages are denoted as E1, E2, E3, E4, and E5 in turn.

[0043] 2. Bottleneck layer (Bottleneck): The output E5 of the fifth encoding stage Stage 4 of the encoder constitutes the bottleneck layer of the network, which contains the highest degree of abstract semantic information. It does not directly correspond to any decoder layer through a jump connection, but serves as the starting input for the entire decoding path.

[0044] 3. Asymmetric correspondence between decoder and encoder: The four up-sampling stages of the decoder correspond one-to-one to the first four stages (first encoding stage Stage 0 to third encoding stage Stage 4) of the encoder through jump connections, in the order from deep to shallow: First decoding stage of the decoder: First, the tobacco stem feature map of the bottleneck layer E5 is up-sampled, and then it is spliced and fused with E4 output from the fourth encoding stage Stage 3 of the encoder. Second decoding stage of the decoder: The tobacco stem feature map output by the first decoding stage is up-sampled and spliced and fused with E3 output from the third encoding stage Stage 2 of the encoder.

[0045] Third decoding stage of the decoder: The tobacco stem feature map output by the second decoding stage is up-sampled and spliced and fused with E2 output from the second encoding stage Stage 1 of the encoder.

[0046] Fourth decoding stage of the decoder: The tobacco stem feature map output by the third decoding stage is up-sampled and spliced and fused with E1 output from the first encoding stage Stage 0 of the encoder.

[0047] In this way, the last layer of the encoder (the bottleneck layer) initiates the decoding process, while each of the remaining layers of the encoder provides critical contextual information for the recovery of details at the corresponding level of the decoder. This structure ensures a gradual, smooth recovery from the most abstract semantic concepts to the finest spatial details.

[0048] Finally, the output layer of the deep learning model uses a 1x1 convolution to map the 64-channel feature map output by the decoder to 3 channels (corresponding to background, tobacco leaf, and stem, respectively) and generate a segmentation probability map consistent with the original input size, resulting in the final segmentation result. This deep learning model architecture combines the powerful feature extraction capabilities of ResNet50, the dual-dimensional attention enhancement of CBAM attention modules, the optimized nonlinear activation of Mish, and the feature fusion of multiple layers of skip connections, forming an end-to-end system highly optimized for tobacco leaf and stem segmentation tasks.

[0049] In this embodiment, the present application uses ResNet50 as the feature extraction backbone network, replacing the relatively simple encoder structure in the original U-Net. ResNet50 is a residual network with a depth of 50 layers, and its core advantage lies in effectively solving the gradient vanishing and network degradation problems in the training process of deep networks by introducing "residual connections" (Residual Connection). Its mathematical expression is: ; where, represents the weight parameter set of all layers in the residual mapping. Here, the subscript represents the th network layer, so represents all weights in the residual block that need to be learned through training, represents the input tobacco stem and leaf feature map, represents the output tobacco stem and leaf feature map, is a residual function composed of three convolutional layers (1x1, 3x3, 1x1). The input is directly added to the output of the residual function , which constitutes a "shortcut connection" (Shortcut Connection) that allows gradients to flow directly in the backpropagation, ensuring effective training of deep networks.

[0050] Relationship between ResNet50 and encoder: In this framework, the ResNet50 backbone network itself is used as the encoder. The five encoding stages (first encoding stage Stage 0 to fifth encoding stage Stage 4) inside it constitute the down-sampling path of the U-Net. Specifically, the input image is first processed by the first encoding stage Stage 0, and the output thereof is taken as the input of the second encoding stage Stage 1; the output of the second encoding stage Stage 1 is taken as the input of the third encoding stage Stage 2, and so on, which is a sequential processing relationship rather than a fusion of features of multiple encoding stages Stage. The output of each encoding stage Stage is sent to the corresponding level of the decoder through the skip connection (after the CBAM attention module) while being sent to the next encoding stage Stage.

[0051] In this embodiment, in order to enable the deep learning model to pay more attention to important information in the tobacco stem leaf image, the present application introduces a CBAM (Convolutional Block Attention Module) attention module. Unlike attention mechanisms that only pay attention to channel or spatial information, CBAM realizes collaborative attention in two dimensions through sequentially connected channel attention sub-modules (Channel Attention Module) and spatial attention sub-modules (Spatial Attention Module).

[0052] The channel attention sub-module aims to identify which “channels” in the input tobacco stem leaf feature map are more important. It first performs global average pooling (AvgPool) and global maximum pooling (MaxPool) on the input tobacco stem leaf feature map respectively to generate two different channel descriptors. Then, the two descriptors are sent to a shared multi-layer perceptron (MLP) for processing, and the two activation vectors output are added element by element. Finally, a Sigmoid function is used to generate the final channel attention weight . This process can be summarized as: ; wherein, represents a channel attention map (Channel Attention Map), which is a weight vector with dimensions Cx1x1, wherein C represents the number of channels of the input tobacco stem leaf feature map F, represents the final channel attention weight generated by the Sigmoid function, represents the input tobacco stem leaf feature map, represents the Sigmoid function, represents the multi-layer perceptron operation, denotes a global average pooling operation, denotes a global max pooling operation.

[0053] the resulting final channel attention weight will be broadcasted and multiplied element-wise with the original tobacco stalk feature map to generate a channel-weighted tobacco stalk feature map .

[0054] The spatial attention sub-module follows the channel attention sub-module, aiming to identify which “spatial locations” in the feature map are more worthy of attention. It takes the channel-weighted tobacco stalk feature map output by the channel attention sub-module as input, first performs average pooling and max pooling along the channel dimension to generate two 2D spatial description maps. Then, the two maps are concatenated and fused through a 7x7 convolutional layer. Finally, the final spatial attention weight is generated through a Sigmoid function. The process can be summarized as: ; where, denotes the final spatial attention weight generated through the Sigmoid function, denotes the channel-weighted feature map generated by the channel attention sub-module, denotes the spatial attention map, which is a weight map with dimensions 1xHxW, where H and W represent the height and width of the tobacco stalk feature map, respectively. This weight map assigns an importance weight to each spatial location (pixel) of the tobacco stalk feature map . The channel-weighted tobacco stalk feature map is first averaged and maximized along the channel dimension to generate two 2D maps. Then, the two maps are concatenated and fused through a 7x7 convolutional layer , and finally, the Sigmoid activation function (σ) is applied to generate denotes the 7x7 convolution operation.

[0055] The resulting final spatial attention weight will be multiplied element-wise with the input tobacco stalk feature map to obtain the final double-optimized feature map .

[0056] In this embodiment, the present application uses Mish as the network activation function, instead of the traditional ReLU activation function. Mish is a self-regularizing non-monotonic activation function, which is defined as follows: ; where the softplus function is defined as: ; where, represents the output of the Mish activation function, Mish is a self-regularizing, non-monotonic activation function, which has a small gradient in the negative value region, which helps to avoid the problem of gradient disappearance, and its smooth characteristics make it more stable in the optimization process, represents a self-regularizing non-monotonic activation function, represents the input tobacco stem leaf feature map (representing each pixel value in the feature map, the Mish function performs a smooth non-linear transformation on all values, since the Mish function itself is a smooth, continuous and derivable curve everywhere, so no matter what the pixel value in the feature map is (positive, negative or zero), it can smoothly calculate a new value, which makes the gradient flow smoothly during backpropagation, which is beneficial to model optimization), in a neural network, this is usually the output of the previous layer of neurons or an element value in the feature map, represents the tangent function, represents the Softplus function, which can be regarded as a smooth approximation of the ReLU activation function, whose output is always positive, and the derivative is the Sigmoid function, represents the exponential function with the natural constant e as the base.

[0057] Mish as a network activation function, it smooths the transition from negative to positive. The combination of Tanh and softplus functions makes Mish an activation function that is self-regularizing, with multiple key properties that contribute to its successful application in deep learning tasks. The main advantage of Mish activation function compared to traditional ReLU activation function is its smoothness and non-monotonicity, which helps better gradient flow and improved learning dynamics. Unlike ReLU, which is a piecewise linear function, Mish activation function has continuous derivatives of all orders (denoted as ), thus providing a smoother optimization space. This is particularly important in deep networks, as ReLU activation function ( , represents the input tobacco stem and leaf feature map (representing the value of each pixel in the feature map, when the value of a certain pixel in the tobacco stem and leaf feature map is exactly equal to 0, the ReLU function forms a “sharp corner” or “kink” here, and in mathematics, the derivative of this point is undefined, which is the meaning of “non-smoothness”, which refers to the mathematical properties of the function when dealing with individual element values, not the entire tobacco stem and leaf feature map x equals 0.), in a single neuron of the neural network, The non-smoothness of the input value, which is usually the result of the weighted sum of all the neuron outputs of the previous layer plus a bias term (bias), i.e. the output of the linear calculation, is then sent to an activation function (such as ReLU or Mish) for non-linear transformation before being passed as the final output of the current neuron to the next layer), can cause optimization problems. The smoothness of the Mish activation function makes optimization smoother, reduces training instability, and promotes more effective learning in deep networks.

[0058] The present application integrates the CBAM (Convolutional Block Attention Module) attention module, enabling the deep learning model to adaptively learn and focus on key spatial regions (such as small stems) and important channel features in the image, suppressing background noise and irrelevant information. At the same time, the Mish activation function is used instead of the traditional activation function to improve the gradient flow in the network, which helps the model converge faster and achieve better performance.

[0059] In this embodiment, a hybrid loss function combining Dice loss and focal loss is used to address the problem of extremely uneven number of stem, leaf, and background pixels in tobacco images. The Dice loss directly optimizes the overlap metric (IoU) between the segmentation result and the true label, and is sensitive to foreground segmentation. The focal loss focuses on reducing the contribution of easily classified samples (such as a large number of background or leaf pixels) to the loss, so that the model pays more attention to the few difficult-to-classify samples (such as stem pixels), effectively alleviating the class imbalance problem and significantly improving the segmentation accuracy of small targets. The overall formula of the hybrid loss function is defined as: ; Wherein, represents the loss function of the deep learning model, and both represent weight coefficients, represents the Dice loss, represents the focal loss.

[0060] The first part of the Dice loss directly optimizes the overlap metric between the segmentation result and the true label, and is sensitive to foreground (such as tobacco stems) segmentation. Its definition is derived from the Dice coefficient (D), and the formula is defined as: ; ; wherein, D i represents the Dice coefficient, i represents each pixel in the tobacco stem and leaf image, P i represents the probability that the deep learning model predicts that pixel i belongs to the foreground, Y i represents the true label of pixel i, usually 1 for foreground and 0 for background.

[0061] The second part of the focal loss focuses on reducing the contribution of easily classified samples (such as a large number of background or leaf pixels) to the loss, so that the model pays more attention to the minority class samples (such as stem pixels) that are difficult to classify. Its formula is defined as: ; wherein, P i represents the prediction probability of the deep learning model for the true class, F i represents the focusing parameter. When , the loss weight of easily classified samples can be effectively reduced.

[0062] By combining the Dice loss sensitive to the shape of the segmentation region and the focal loss that can effectively handle the class imbalance problem, the hybrid loss function of the present application can significantly alleviate the training difficulties brought by sample imbalance, thereby effectively improving the segmentation accuracy of small targets (such as tobacco stems).

[0063] In this embodiment, the training process of the deep learning model adopts the following specific hyperparameter configuration and strategy: (1) Optimizer and learning rate strategy. The Adam optimizer is selected to update the weights of the deep learning model. The initial learning rate is set to 5e-5. In order to achieve more effective convergence and avoid falling into local optimum during training, the cosine annealing (Cosine Annealing) learning rate decay strategy is adopted, and the learning rate will be smoothly decreased from the initial value to the minimum learning rate 5e-7 with the training round.

[0064] (2) Training rounds and batch size. The total training process is set to a total of 100 rounds (TotalEpochs). The batch size used for each parameter update is 2. The size of the tobacco stem and leaf image input into the deep learning model is uniformly processed to 512x512 pixels.

[0065] (3) Regularization and training method. In this training configuration, the weight decay (Weight Decay) parameter is set to 0, i.e. no L2 regularization is used. At the same time, the freeze training strategy (Freeze Train) is set to False, indicating that all deep learning model layers, including the pre-trained backbone network, participate in parameter update learning during the entire training process, and no layer is frozen.

[0066] (4) Data loading. In order to accelerate the reading and preprocessing process of training data, 4 data loading worker threads (Num Workers) are configured.

[0067] In this embodiment, the test evaluation is to apply the final deep learning model to the test set after completing the training and verification of the deep learning model, and evaluate the actual performance of the deep learning model. The evaluation indicators include loss value, average intersection over union, etc.

[0068] S4, using the trained, tested and verified deep learning model, tobacco stem and leaf image segmentation.

[0069] Through the above design, the present application provides a complete tobacco segmentation solution, enabling ordinary technical personnel to achieve efficient and accurate operation of tobacco segmentation in the existing computing hardware environment through the method of the present application.

Claims

1. A tobacco stalk image segmentation method, characterized by, The method comprises the following steps: S1, based on the control environment acquisition and field environment acquisition, obtaining tobacco stem and leaf image data; S2, preprocessing the obtained tobacco stem and leaf image, and dividing it into a training set, a test set and a validation set; S3, constructing an improved deep learning model based on U-Net, and training, testing and verifying the improved deep learning model using the training set, the test set and the validation set; S4, using the trained, tested and verified deep learning model to segment the tobacco stem and leaf image.

2. The tobacco stem image segmentation method according to claim 1, characterized in that, The improved deep learning model comprises an encoder, a decoder connected to the encoder by jumping, and an output layer connected to the decoder; The encoder is used for hierarchical feature extraction of the tobacco stem and leaf image; The decoder is used for recovering the spatial resolution of the feature map based on the feature extraction result of the encoder; The output layer is used for mapping the feature map output by the decoder into three channels and generating a tobacco stem and leaf image segmentation probability map consistent with the original size to obtain the final segmentation result, wherein the three channels correspond to background, tobacco leaf and stem, respectively.

3. The tobacco stem image segmentation method according to claim 2, characterized in that, The encoder uses ResNet50 as the backbone network, includes five encoding stages of down-sampling composed of a first encoding stage Stage 0 to a fifth encoding stage Stage 4, and the output ends of the five encoding stages are all connected to CBAM attention modules, wherein the first encoding stage Stage 0 is used as an input port to obtain tobacco stem and leaf image features through down-sampling, the second encoding stage Stage 1 to the fifth encoding stage Stage 4 extract deep features of the tobacco stem and leaf image by stacking residual bottleneck modules, and the first encoding stage Stage 0 to the fifth encoding stage Stage 4 are connected in turn; The decoder includes four up-sampling stages, wherein the input of each up-sampling stage includes the up-sampling result of the previous up-sampling stage and the CBAM attention module enhanced tobacco stem and leaf feature map, and the high-dimensional tobacco stem and leaf image feature formed by jumping and splicing, wherein the convolution block containing 3×3 convolution and 1×1 convolution is used to fuse the spliced high-dimensional tobacco stem and leaf image feature, and the Mish activation function is set in the convolution block.

4. The tobacco stem image segmentation method according to claim 3, characterized in that, The first encoding stage Stage 0 includes a 7×7 convolution kernel and a maximum pooling layer group; the second encoding stage Stage 1 to the fifth encoding stage Stage 4 respectively contain 3 residual bottleneck modules, 4 residual bottleneck modules, 6 residual bottleneck modules and 3 residual bottleneck modules.

5. The tobacco stem image segmentation method according to claim 3, characterized in that, The CBAM attention module is connected in turn with a channel attention submodule and a spatial attention submodule; The expression of the channel attention submodule is as follows: ； wherein, represents a channel attention map, represents a final channel attention weight generated by a Sigmoid function, represents an input tobacco stalk feature map, represents a Sigmoid function, represents a multi-layer perceptron operation, represents a global average pooling operation, represents a global max pooling operation; The expression of the spatial attention submodule is as follows: ； wherein, denotes the final spatial attention weights generated by the Sigmoid function, denotes the channel-weighted feature maps generated by the channel attention sub-module, denotes the spatial attention map, denotes a 7x7 convolution operation.

6. The tobacco stem image segmentation method according to claim 3, characterized in that, The expression of the Mish activation function is as follows: ； ； wherein, denotes the output of a Mish activation function, denotes a self-regularizing non-monotonic activation function, denotes an input tobacco leaf feature map, denotes a tangent function, denotes a Softplus function, denotes an exponential function with base the natural constant e.

7. The tobacco stem image segmentation method according to claim 1, characterized in that, The expression of the loss function of the improved deep learning model is as follows: ； ； ； ； wherein, represents a loss function of the deep learning model, and each represents a weight coefficient, represents a dices loss, represents a focal loss, represents a dices coefficient, i represents each pixel in the tobacco stalk leaf image, represents a probability that the deep learning model predicts that pixel i belongs to the foreground, represents a true label of pixel i, represents a predicted probability of the deep learning model for the true class, represents a focal parameter.

Images