Tobacco leaf vein segmentation method and device based on improved UNet model

By improving the UNet model, combining the spatial attention mechanism and the generative adversarial network, the problem of insufficient accuracy and robustness in tobacco leaf vein segmentation is solved, and efficient leaf vein segmentation under complex conditions is achieved, which significantly improves segmentation accuracy and stability , provides technical support for the quality evaluation and processing of tobacco leaves.

CN119068192AActive Publication Date: 2024-12-03ZHENGZHOU TOBACCO RES INST OF CNTC

Patent Information

Application Number
CN202411120073.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-12-03
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

The prior art has problems of low accuracy and poor robustness in tobacco leaf vein segmentation, especially when dealing with irregularly shaped vein edge details such as fractures, crosses or fusion parts.

Method used

The vein segmentation method of tobacco leaf based on the improved UNet model is adopted to integrate the spatial attention mechanism and the generative adversarial network, and the SA-UNet network is constructed through hyperspectral imaging and principal component analysis, and the generative adversarial network is used for adversarial training, and the model is optimized. Adaptability and segmentation accuracy under complex conditions.

Benefits of technology

It significantly improves the accuracy and robustness of tobacco leaf vein segmentation, ensures stable performance under different conditions, provides efficient technical solutions, and provides reliable technical support for tobacco leaf quality evaluation and processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068192A_ABST
    Figure CN119068192A_ABST
Patent Text Reader

Abstract

The invention provides a tobacco leaf vein segmentation method and device based on an improved UNet model, and belongs to the technical field of tobacco leaf classification and identification. The tobacco leaf vein segmentation method comprises the following steps: acquiring a hyperspectral image of a tobacco leaf sample and constructing a sample set of the hyperspectral image of the tobacco leaf; constructing a vein segmentation model based on a UNet network, wherein the vein segmentation model at least comprises an encoder, a space attention module and a decoder; constructing a generative adversarial network model, inputting the sample set into the vein segmentation model, and training the vein segmentation model by using the generative adversarial network model; and performing vein segmentation on the hyperspectral image of the tobacco leaf to be detected through the trained vein segmentation model. According to the tobacco leaf vein segmentation method and device provided by the invention, stable segmentation performance can be kept under different conditions, an efficient technical solution is provided for tobacco leaf quality evaluation and processing, and the segmentation precision and robustness are remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of tobacco leaf classification and recognition, and in particular to a tobacco leaf vein segmentation method and device based on an improved UNet model. Background Art

[0002] Veins are an important part of leaves, containing a large amount of physiological information, genetic information and quality information of plants. Tobacco is one of the important economic crops in my country. The characteristics of tobacco leaf veins are of great importance in the whole process of tobacco breeding, cultivation and processing. The content of leaf veins is an important indicator for the economic consideration of tobacco breeding and processing. The shape and width of leaf veins are important factors for judging the leaf position in the tobacco leaf grading process. The existing detection of tobacco leaf veins is mainly through manual stem extraction, which has limited sample size, poor sample representativeness, cumbersome operation and low accuracy. Therefore, it is of great practical significance to achieve accurate and lossless segmentation of tobacco leaf veins.

[0003] In recent years, there have been a lot of research in the field of image segmentation in computer vision. One technical method is to generate a vein label map by manually annotating the vein information, pass it into the U-Net network for training, optimize the model parameters, and predict the leaf image, and quantitatively compare it with the manually annotated label map to obtain the performance evaluation results of the model. The encoder stage in U-Net uses multiple convolution and pooling layers, which can effectively extract abstract features in the image and capture key information such as the shape, texture and position of the veins.

[0004] However, for irregularly shaped leaf vein edge details, such as broken, crossed or fused parts, the above cannot accurately segment them.

[0005] In view of this, the inventor, based on many years of production design experience in this field and related fields, has designed a tobacco leaf vein segmentation method and device based on an improved UNet model after repeated experiments, in order to solve the problems existing in the prior art. Summary of the invention

[0006] The object of the present invention is to provide a tobacco leaf vein segmentation method based on an improved UNet model, which can effectively improve the robustness and accuracy of tobacco leaf vein segmentation.

[0007] To achieve the above object, the present invention proposes a tobacco leaf vein segmentation method based on an improved UNet network model, the tobacco leaf vein segmentation method comprising:

[0008] Collecting hyperspectral images of tobacco leaf samples and constructing a sample set of hyperspectral images of tobacco leaves;

[0009] Constructing a leaf vein segmentation model based on a UNet network, wherein the leaf vein segmentation model at least includes an encoder, a spatial attention module, and a decoder;

[0010] Constructing a generative adversarial network model, inputting the sample set into the leaf vein segmentation model and using the generative adversarial network model to train the leaf vein segmentation model;

[0011] The vein segmentation is performed on the hyperspectral image of the tobacco leaf to be tested by using the trained vein segmentation model.

[0012] Compared with the prior art, the present invention has the following characteristics and advantages:

[0013] The present invention proposes a tobacco leaf vein segmentation method and device based on an improved UNet model, which integrates a spatial attention mechanism, enhances the model's ability to recognize vein details, and optimizes the model's adaptability under complex conditions through adversarial training of a generative adversarial network. By combining hyperspectral imaging with principal component analysis, the separability of features is effectively improved, while ensuring fast reasoning and meeting the needs of real-time processing. The generalization ability test of the model on a variety of tobacco leaf samples shows that it can maintain stable segmentation performance under different conditions, providing an efficient technical solution for tobacco leaf quality evaluation and processing, and significantly improving segmentation accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings described herein are only for explanation purposes and are not intended to limit the scope of the present invention in any way. In addition, the shapes and proportional dimensions of the various components in the figures are only schematic, used to help understand the present invention, and are not specifically limited to the shapes and proportional dimensions of the various components of the present invention. Those skilled in the art can select various possible shapes and proportional dimensions to implement the present invention according to the teachings of the present invention.

[0015] Figure 1 It is a schematic diagram of the steps of the leaf vein segmentation method of the present invention;

[0016] Figure 2 It is a schematic diagram of the hyperspectral tobacco leaf segmentation process of the present invention;

[0017] Figure 3 It is a schematic diagram of the SA-UNet network structure of the present invention;

[0018] Figure 4 Schematic diagram of the overall network structure of GAN-SA-UNet of the present invention;

[0019] Figure 5 It is the PCA dimension reduction graph of the present invention;

[0020] Figure 6 This is a tobacco leaf vein marking effect diagram of the present invention;

[0021] Figure 7 This is a tobacco leaf vein marking effect diagram of the present invention;

[0022] Figure 8 It is a segmentation loss value variation curve diagram of the present invention;

[0023] Fig. 9 Visualization result diagrams of different models of the present invention. DETAILED DESCRIPTION

[0024] The details of the present invention can be more clearly understood with reference to the accompanying drawings and the description of the specific embodiments of the present invention. However, the specific embodiments of the present invention described herein are only used for the purpose of explaining the present invention and cannot be understood as limiting the present invention in any way. Under the guidance of the present invention, technicians can conceive of any possible variations based on the present invention, which should all be considered to belong to the scope of the present invention.

[0025] like Figure 1 As shown, the present invention proposes a tobacco leaf vein segmentation method based on an improved UNet model, and the tobacco leaf vein segmentation method comprises: collecting hyperspectral images of tobacco leaf samples and constructing a sample set of tobacco leaf hyperspectral images; constructing a vein segmentation model based on a UNet network, the vein segmentation model at least comprising an encoder, a spatial attention module and a decoder; constructing a generative adversarial network model, inputting the sample set into the vein segmentation model and training the vein segmentation model using the generative adversarial network model; performing vein segmentation on the hyperspectral image of the tobacco leaf to be tested by using the trained vein segmentation model.

[0026] The tobacco leaf vein segmentation method based on the improved UNet model proposed in the present invention integrates the spatial attention mechanism, enhances the model's ability to recognize vein details, and optimizes the model's adaptability under complex conditions through adversarial training of the generative adversarial network. By combining hyperspectral imaging with principal component analysis, the separability of features is effectively improved, while ensuring fast reasoning and meeting the needs of real-time processing. The generalization ability test of the model on a variety of tobacco leaf samples shows that it can maintain stable segmentation performance under different conditions, providing an efficient technical solution for tobacco leaf quality evaluation and processing, and significantly improving segmentation accuracy and robustness.

[0027] In an optional embodiment of the present invention, in the encoder, a dropout convolution block is used to replace the original convolution block. The structured dropout convolution block used in the method replaces the traditional convolution block. While maintaining the effective extraction of image features by the convolution operation, the dropout convolution block can randomly close some neurons in the network by introducing a random dropout mechanism in the convolution layer, thereby preventing the model from overfitting the training data during the training process. The dropout convolution block randomly discards the output in the convolution layer according to a predetermined probability, which prompts the network to learn different feature representations at each forward propagation, effectively improves the generalization ability of the model for the training data, reduces the risk of overfitting, and enables the model to maintain high segmentation accuracy and robustness when facing new and unseen data. The introduction of structured dropout ensures the spatial consistency of the feature map, making the feature map output by the encoder more stable in quality, and providing a solid foundation for the decoder to generate accurate segmentation maps. In addition, this improved encoder structure does not significantly increase the computational burden while improving the segmentation accuracy, and maintains the high efficiency of the model, which is of great significance for achieving real-time tobacco leaf vein segmentation.

[0028] Specifically, the U-Net network is a deep learning architecture for semantic segmentation tasks. It is a basic model in the field of image segmentation, has groundbreaking significance, and is widely used in the field of medical image segmentation. U-Net will suffer from gradient decay, structural information loss or loss after going through convolutional layers and pooling layers. In order to solve these problems, a spatial and channel attention mechanism module is introduced, which can infer the attention map along the spatial dimension and multiply it with the input feature map to achieve adaptive feature refinement. In addition, the network adopts a structured dropout convolution block instead of the original convolution block in U-Net to prevent the network from overfitting. These measures together improve the performance of the network and effectively address challenges such as gradient decay and structural information loss.

[0029] In an optional embodiment of the present invention, the spatial attention module receives the output of the encoder as input. Specifically, the spatial attention module is designed to further enhance the model's ability to recognize key areas in the image. The spatial attention module receives the output of the encoder as input, and the high-level feature map output by the encoder contains image information that has undergone multi-level abstract processing. The spatial attention module analyzes and processes these feature maps in the spatial dimension through a series of calculation steps. The introduction of the spatial attention mechanism can significantly improve the model's recognition accuracy of tobacco leaf vein details, especially in areas with fine veins or complex textures. By adaptively adjusting the spatial weights of the feature maps, the model can focus more on the key information in the image, which not only improves the quality of the segmentation results, but also enhances the model's generalization ability for different tobacco leaf samples. The application of the spatial attention module also helps to reduce the interference of irrelevant background information, making the segmentation results clearer and more accurate.

[0030] In an optional example of this embodiment, in the leaf vein segmentation model, its last layer uses 1×1 convolution and Sigmoid activation function to generate an output segmentation map. The key step of finally generating the segmentation map involves the last layer of the model. This layer uses a 1×1 convolution operation, followed by a Sigmoid activation function. 1×1 convolution is an efficient feature mapping technique that can compress feature maps and adjust the number of channels without increasing the number of parameters and computational complexity. In the present invention, 1×1 convolution is used to convert the feature map output by the encoder and the spatial attention module into a feature representation of the same size as the original image, while reducing the number of channels of the feature map to prepare for the final segmentation result. The Sigmoid activation function is applied to these adjusted feature maps to compress the range of feature values ​​from any real number to the (0,1) interval, so that the output obtained can be interpreted as the probability that each pixel belongs to a leaf vein. Finally, based on this probability map, the position of the leaf vein can be determined and a clear leaf vein segmentation map can be generated. By using 1×1 convolution and Sigmoid activation function in the last layer of the leaf vein segmentation model: the introduction of 1×1 convolution significantly improves the computational efficiency of the model because it reduces the dimension of the feature map while avoiding a significant increase in the number of parameters; the application of the Sigmoid activation function enables the model to output the probability that each pixel belongs to a leaf vein, which provides intuitive and continuous output for subsequent segmentation decisions.

[0031] In an optional example, in a leaf vein segmentation model, the loss function uses a cross entropy function, whose formula is:

[0032]

[0033] Where: mn is the input image size, representing the number of rows and columns of the input image respectively; x ij Represents a pixel in the image; yij is the correct segmentation result, which is the value of the corresponding pixel in the actual segmentation label (0 or 1); g(x ij ) is the SA-UNet segmentation result, that is, the probability that the pixel predicted by the model belongs to the leaf vein. In the leaf vein segmentation model of the present invention, the loss function adopts the cross entropy function to measure the difference between the segmentation map predicted by the model and the actual segmentation label. This loss function quantifies the overall prediction error by calculating the cross entropy for each pixel and then taking the average. The cross entropy loss function penalizes wrong predictions, which makes the model more accurate in distinguishing leaf veins from background, especially at the edges of leaf veins and complex texture areas. In the model training process of the present invention, by minimizing the loss function, the model can be prompted to learn more accurate segmentation features.

[0034] In an optional embodiment of the present invention, in the leaf vein segmentation model, as Figure 2 As shown, the hyperspectral image of the tobacco leaf sample is input into the encoder, and the encoder converts the hyperspectral image into high-level semantic features and outputs a high-level feature map; the high-level feature map is input into the spatial attention module, and the spatial attention module adjusts the spatial information weight of the high-level feature map and outputs the adjusted feature map; the adjusted feature map is input into the decoder, and the decoder generates a segmentation map with tobacco leaf vein markers through the adjusted feature map. In the vein segmentation model of the present invention, the hyperspectral image of the tobacco leaf sample is first input into the encoder. The encoder is the initial processing stage of the model, which is responsible for converting the original hyperspectral image data into more abstract and advanced semantic features. The high-level semantic feature extraction of the encoder enables the model to capture more subtle and abstract features in the image, and through these features, it can capture key information in the image, such as shape, texture and color, thereby improving the accuracy of segmentation. The encoder extracts and compresses features through a series of convolutional layers and pooling layers, and finally outputs a high-level feature map. Then, these high-level feature maps are sent to the spatial attention module. The role of the spatial attention module is to adjust the weight of the spatial information in the feature map. The introduction of the spatial attention module significantly enhances the model's ability to recognize key areas of the image, especially in vein details and complex texture areas. This not only improves the accuracy of segmentation, but also enhances the model's adaptability to tobacco leaf images under different types and conditions. In this way, the model can more effectively identify and distinguish between veins and other parts. Finally, the adjusted feature maps are input to the decoder, which uses these feature maps to generate the final segmentation map, which has clear tobacco leaf vein markings and achieves accurate segmentation of the veins. The segmentation map with tobacco leaf vein markings generated by the decoder is clear and accurate, and can provide a reliable basis for subsequent tobacco leaf quality evaluation and processing.

[0035] In an optional embodiment of the present invention, the generative adversarial network model includes a generator capable of game adversarial training, which discriminates whether the data source is real data; and a discriminator, which fits the real data to confuse the discriminator so that the generated data is discriminated as true. The generative adversarial network (GAN) model proposed in the present invention includes two core components: a generator and a discriminator. The function of the generator is to use random noise as input to generate data through a deep neural network, and the generated data should be as close to the real data as possible visually. Specifically, the data generation ability learned by the generator in the present invention through adversarial training is significantly improved, and the generated data is almost indistinguishable from the real data visually, which is very important for background generation and detail enhancement in tasks such as image segmentation. The discriminator is responsible for distinguishing the true and false data, distinguishing the data generated by the generator from the actual real data. And because the data generation ability of the generator is significantly improved after adversarial training, the true and false discrimination ability of the discriminator is also strengthened, and the generated data and real data can be more accurately identified and distinguished. In this adversarial training process, the generator and the discriminator conduct game adversarial training. The generator continuously learns how to generate more realistic data to confuse the discriminator, while the discriminator continuously learns how to more accurately identify true and false data. This adversarial training promotes the gradual improvement of the quality of the data generated by the generator until the discriminator has difficulty distinguishing between true and false data, thereby generating more realistic and accurate segmentation results in practical applications, such as tobacco leaf vein segmentation.

[0036] In an optional example of this implementation, the training process of the generative adversarial network model is:

[0037]

[0038] Where: x is the real data, which conforms to P data (x) distribution; z is noise data, which conforms to P z (z) distribution; D(x) is the probability that x is judged to be true; G(z) is the data generated by the generator; D(G(z)) is the probability that the data generated by the generator is judged to be true. In the generative adversarial network (GAN) model of the present invention, the training process follows a specific game adversarial mechanism. This process is achieved by minimizing the loss function of the generator G and maximizing the loss function of the discriminator D. Specifically, the loss function V(D,G) consists of two parts: the first part Calculate the logarithmic expected value of the probability that the discriminator judges the real data X as true; Part II Calculate the logarithmic expectation of the probability that the noise data Z generated by the generator is judged as false by the discriminator. During the training process, the generator G significantly improves the quality of the generated data by continuously generating more and more realistic data, making it closer to the real data in terms of visual and statistical characteristics, so that the generator G tries to generate more and more realistic data to confuse the discriminator D, while the discriminator D continuously learns to more accurately distinguish between real data and generated data. In this way. The generator and the discriminator compete with each other during the training process, and this adversarial training mechanism promotes the adaptability of the model in dealing with complex data distributions, which not only improves the ability to generate background and details in tasks such as image segmentation, but also provides an effective technical means for other fields that require high-quality data generation.

[0039] In an optional example of this embodiment, the vein segmentation model is used as a generator and the multi-layer convolutional network is used as a discriminator. In the generative adversarial network (GAN) model of the present invention, the vein segmentation model is used as a generator, and the purpose of the generator is to generate realistic vein segmentation images by learning the features of real vein segmentation data. The vein segmentation model captures the key features of the image through its deep learning architecture and generates highly detailed segmentation results. The multi-layer convolutional network acts as a discriminator and is responsible for evaluating the authenticity of the generated segmented image. The multi-layer convolutional network uses its in-depth feature extraction capabilities to analyze the images generated by the generator and determine their similarity with the real vein segmentation images.

[0040] In an optional example, the generator generates fake tobacco leaf vein images and label vein images, and the discriminator captures the feature information in the fake tobacco leaf vein images and label vein images, and calculates the authenticity of the fake tobacco leaf vein images and label vein images. The generator uses random noise as input through multi-layer convolution operations to learn and simulate the distribution characteristics of real tobacco leaf vein images. The generated fake tobacco leaf vein images and label vein images are then provided as input to the discriminator. The discriminator is composed of a multi-layer convolutional network, and its task is to capture the feature information of these input images and evaluate their authenticity. The discriminator performs in-depth feature analysis on the fake images, judges their similarity with the real tobacco leaf vein images, and outputs a authenticity score, which reflects the probability that the fake image is judged as real data. This adversarial training mechanism strengthens the game between the generator and the discriminator, pushing the performance of both to a higher level. In the end, the generator can produce highly realistic tobacco leaf vein images, while the discriminator has excellent authenticity identification capabilities, improving the accuracy and robustness of tobacco leaf vein segmentation.

[0041] Further, a random vector or noise is input into the generator, and the generator processes the random vector and noise to generate a fake tobacco leaf vein image, which is visually similar to the labeled vein image; the fake vein image and the labeled vein image are simultaneously input into the discriminator, and the discriminator outputs a probability value, which represents the probability that the input image is a real tobacco leaf vein image. Specifically, the input of the generator is a random vector or noise, and these input data are mathematically extracted from a certain probability distribution. The generator uses these random data and generates fake tobacco leaf vein images that are visually similar to real tobacco leaf vein images through deep neural network processing. The generated images are intended to simulate the characteristics of real tobacco leaf veins, including shape, texture, and structure. Subsequently, these fake vein images are fed into the discriminator together with the corresponding labeled vein images. The discriminator is a trained multi-layer convolutional network that evaluates the characteristics of the input image and outputs a probability value that reflects the possibility that the input image is a real tobacco leaf vein image. This method improves the accuracy of leaf vein segmentation, enhances the model's recognition ability for complex images, and provides an effective training strategy for image generation tasks.

[0042] In an optional embodiment of the present invention, in the generative adversarial network model, the loss function adopts the cross entropy function and is optimized using the Adam method, and the calculation formula is:

[0043] loss sum =loss SA-UNet +loss uD(z) +loss Dc(z)

[0044] Where: loss SA-UNet is the segmentation loss; loss uD(z) is the discriminator loss; loss Dc(z) is the feature loss. In the generative adversarial network (GAN) model of the present invention, the loss function is designed using the cross entropy function and trained using the Adam optimization algorithm. The loss function consists of three parts: segmentation loss, SA-UNet , discriminator loss loss uD(z) And feature loss loss Dc(z) . Segmentation loss SA-UNet It measures the difference between the fake tobacco leaf vein image output by the generator and the real label image. uD(z) It measures the accuracy of the discriminator in distinguishing fake images generated by the generator. Dc(z) It measures the discriminator's ability to distinguish the features of generated images and real images. The sum of these three losses constitutes the total loss of the GAN model. sum, which is used to guide the training of the network. In this way, the generator learns to generate more realistic vein images, while the discriminator learns how to distinguish between real and fake images more accurately. The use of the Adam optimization algorithm further improves the efficiency and stability of model training, allowing the network parameters to converge to the optimal solution more quickly. Ultimately, this technical solution makes the generated fake tobacco leaf vein images visually highly similar to real images, while the discriminator has stronger discrimination capabilities, thereby achieving excellent performance in the tobacco leaf vein segmentation task.

[0045] The specific implementation process of the present invention is now described in detail in conjunction with an embodiment:

[0046] 1 Materials and methods

[0047] 1.1 Materials

[0048] This example uses tobacco as the main research object, and collects tobacco samples of different varieties, parts, and grades from major tobacco producing areas across the country. Including Sichuan K326 upper tobacco B1F, B2F, B3F, B4F, B1L, B2L, B3L, B4L; middle tobacco C2F, C3F, C4F, C2L, C3L, C4L; lower tobacco X1F, X2F, X3F, X4F, X1L, X2L, X3L, X4L, Fujian Nanping Cui Bi No. 1, Yunnan Yuxi Yun 87, Yunnan Chuxiong Yun 8542 grade, Yueyan 98, Henan Sanmenxia Mianchi Zhongyan 100C3F, cigar wrapper tobacco. The tobacco dataset contains 340 pieces of flue-cured tobacco, and the dataset is randomly divided into a training set and a test set at a ratio of 8:2, where the training set contains 272 pieces of tobacco leaves and the validation set contains 68 pieces of tobacco leaves. The tobacco leaf samples used are shown in Table 1.

[0049] Table 1:

[0050]

[0051] The instrument uses a hyperspectral imaging system (1000-2500nm, 288 effective bands), SpecView software, envi 5.6 software, pycharm2023 high-performance computer (graphics card), and a constant temperature and humidity chamber.

[0052] 1.2 Data Collection Method

[0053] Before data collection, tobacco leaves need to be balanced in an environment with a relative humidity of 70% and an ambient temperature of (25±2)℃ for 48 hours to facilitate flattening of tobacco leaves in subsequent experiments and reduce breakage. After the tobacco leaves are flattened, they are placed in the hyperspectral imaging system for data collection. The exposure time of the hyperspectral camera is set to 12ms and the scanning speed is 1.5cm / s, which helps to collect data with accuracy and stability.

[0054] 1.3 Image Correction Method

[0055] Before the hyperspectral imaging system collects data, it is necessary to use the black and white plate correction method to eliminate the uneven distribution of light intensity in each band and the noise caused by the dark current in the lens. Collect a standard white calibration plate with a reflectivity of 99%, obtain a white plate calibration hyperspectral image, turn off the light source and cover the lens cap to obtain a black calibration hyperspectral image with a reflectivity of 0%. The corrected hyperspectral image R can be obtained by the formula. The correction formula is:

[0056]

[0057] Where: I is the original hyperspectral data; B is the hyperspectral data of the black calibration plate; W is the hyperspectral data of the white calibration plate.

[0058] 2 Leaf vein segmentation model construction method

[0059] 2.1SA-UNet Network Generator

[0060] The U-Net network is a deep learning architecture for semantic segmentation tasks. It is a basic model in the field of image segmentation. It is groundbreaking and widely used in the field of medical image segmentation. U-Net suffers from problems such as gradient decay and loss or loss of structural information after convolutional and pooling layers. To address these problems, a spatial and channel attention mechanism module is introduced, which infers the attention map along the spatial dimension and multiplies it with the input feature map to achieve adaptive feature refinement. In addition, the network uses a structured dropout convolution block instead of the original convolution block in U-Net to prevent the network from overfitting. The above measures together improve the performance of the network and effectively address challenges such as gradient decay and loss of structural information.

[0061] SA-UNet is a neural network with a U-shaped encoder-decoder structure, such as Figure 3 As shown. Its encoder contains structured dropout convolution blocks and maxpooling operations. Each convolution block is followed by DropBlock, normalization layer and ReLU activation function, and then downsampled by maximum pooling to increase the number of feature channels. The decoder includes deconvolution for upsampling, which halves the number of feature channels, concatenation of corresponding feature maps in the autoencoder, and structured dropout convolution blocks. The spatial attention module is located between the encoder and decoder, and the last layer uses 1×1 convolution and Sigmoid activation function to generate the output segmentation map. The loss function uses the cross entropy function, and its formula is as follows:

[0062]

[0063] Where: mn is the input image size; y ijis the correct segmentation result; g(x ij ) is the SA-UNet segmentation result.

[0064] SA-UNet introduces a spatial attention mechanism, which enables the model to pay more attention to the importance of different areas in the image, which helps to better capture key spatial features in the tobacco leaf vein segmentation task and improve segmentation accuracy. Compared with the traditional U-Net, the spatial attention mechanism can make SA-UNet pay more attention to the detail information in the image, which is very important for segmenting the fine structures and edges in the tobacco leaf veins. However, the introduction of the attention mechanism increases the computational complexity of the model, resulting in slower training and inference speeds; if the hyperparameters are not adjusted properly, the introduced attention mechanism causes the model to overfit on the training data, resulting in poor performance on unseen data. In response to the above problems, this embodiment uses a generative adversarial network to optimize the SA-UNet network model.

[0065] 2.2 Generative Adversarial Networks

[0066] The Generative Adversarial Network (GAN) model consists of a generator and a discriminator, which continuously optimize each other. The discriminator determines whether the data source is real data or not. The generator fits the real data to confuse the discriminator so that the generated data is identified as real. The two sides conduct adversarial training. The training process of GAN is as follows:

[0067]

[0068] Where: x is the real data, which conforms to P data (x) distribution; z is noise data, which conforms to P z (z) distribution; D(x) is the probability that x is judged to be true; G(z) is the data generated by the generator; D(G(z)) is the probability that the data generated by the generator is judged to be true.

[0069] At the beginning of training, the generator is untrained and the data it generates is of low quality. Therefore, the discriminator can accurately distinguish the source of the data, thereby achieving high accuracy and a large log(1-D(G(z))) value. In order to improve the quality of the generator, it is necessary to reduce the probability that the discriminator will judge the data generated by the generator as false, thereby reducing the log(1-D(G(z))) value, which allows the generator to obtain a larger gradient for model updates. The generative network is continuously optimized by gradually increasing the similarity between the generated data and the real data, and the gradient of the discriminator will also change, enhancing the ability to judge the source of the data. The discriminator and the generator are trained simultaneously, forming an adversarial relationship, and they continuously compete and optimize each other.

[0070] This embodiment uses SA-UNet as a generator and a multi-layer convolutional network as a discriminator. The discriminator takes the generated tobacco leaf vein image and the label vein image as input, uses an adaptive convolution kernel for convolution, continuously reduces the image size, and extracts features. The fully connected layer calculates the discriminator value of the generated tobacco leaf vein image and the label vein image. During the training process, the discriminator continuously improves its ability to generate images as false, and the generator continues to optimize after receiving feedback, improving the discriminator's ability to classify them as true. The two models are in a competitive training process, constantly challenging each other. After training, the accuracy of the generator's image segmentation results is significantly improved, and accurate image segmentation results can be obtained in testing and practical applications.

[0071] 2.3GAN-SA-UNet tobacco leaf vein segmentation method

[0072] In order to further improve the network model's ability to segment tobacco leaf vein images, this embodiment uses the adversarial idea of ​​generative adversarial networks to optimize the SA-UNet network model, and proposes a tobacco leaf vein image segmentation method GAN-SA-UNet based on generative adversarial network optimization. In the construction of the generator and discriminator of the generative adversarial network, this embodiment uses SA-UNet as the generator, and uses the discriminator to make the two networks in the model compete, thereby imposing higher-level constraints on the SA-UNet segmentation network. The model iterates through adversarial training between the generator and the discriminator, and finally achieves the optimal segmentation model.

[0073] like Figure 4 As shown in the figure, the architecture of the GAN-SA-UNet model is shown. The left side of the model structure diagram is the segmentation network, and the right side is the adversarial network. Through the adversarial training of the two networks, the discriminant network can capture the high-level feature information of the tobacco leaf vein image and distinguish the predicted output image and label image of the segmentation network. If the output of the generator is classified as a true value in the discriminator, then this indicates the effectiveness of the generator network. The cross entropy form is used to obtain the loss value of the model optimization loss uD(z) The discriminator network extracts features from the generated segmentation results and segmentation label images, calculates the difference based on these feature maps, and then calculates the cross entropy loss between the images. Dc(z) The loss function of the model is optimized using the Adam method.

[0074] loss sum =loss SA-UNet +loss uD(z) +loss Dc(z)

[0075] Where: loss SA-UNet is the segmentation loss; loss uD(z)is the discriminator loss; loss Dc(z) is the feature loss.

[0076] In the adversarial learning between the generator and the discriminator, the loss function of the discriminator, features and SA-UNet is calculated to promote network updates and optimize the entire network. GAN-SA-UNet introduces adversarial training to make the generated segmentation results more realistic and accurately reflect the details and texture of the tobacco leaf veins. The robustness of the generative adversarial network improves the segmentation effect of the model on images under different conditions and enhances the generalization ability. Through global adversarial training, the generator can better understand the overall image structure of the tobacco leaf veins, making the segmentation results more consistent and reasonable. GAN-SA-UNet, which integrates the generation and discriminant networks, can adapt to complex tobacco leaf veins, effectively solve the problem of broken veins, produce smooth and clear segmentation results, and provide a reliable solution for tobacco leaf image processing.

[0077] result

[0078] The first three principal components of the spectral dimension of the hyperspectral image are synthesized into a three-channel image, such as Figure 5 As shown in the figure, compared with the ordinary RGB image, the difference between the leaf surface and the veins of the tobacco leaf image after spectral PCA dimensionality reduction is more significant, and the vein features are more prominent. The reason is that PCA dimensionality reduction helps to highlight the main change directions related to the leaf surface and veins in the tobacco leaf hyperspectral data, which will help improve the accuracy of subsequent segmentation tasks.

[0079] High-quality data annotation of tobacco leaf veins is the key to accurate segmentation. The original hyperspectral data is synthesized into a pseudo-color image after PCA dimensionality reduction. The pixel function in the Region of Interest (ROI) tool in ENVI 5.6 software is used to import the synthesized pseudo-color image and manually annotate the main vein and branch vein areas of the tobacco leaf. The tobacco leaf veins are annotated pixel by pixel to form an XML file, and the XML file is converted to obtain the annotated map of the tobacco leaf veins. The annotation results are shown in the figure below. Figure 6 shown.

[0080] In order to improve the robustness, migration ability and more accurate segmentation of the network model, this embodiment selects typical varieties and grades of tobacco leaves, and performs random flipping, Gaussian blurring, affine transformation, image cropping and padding on the data set so that the data set can better fit the network of this paper. This data enhancement operation helps the model better learn the different features of the data, improves the generalization ability of the model, and enables it to better process tobacco leaf images in various practical application scenarios.

[0081] 3.2 Model Improvement and Optimization Training

[0082] In the tobacco leaf vein segmentation task, the basic model is U-Net, on which improvements and optimizations are made. Specifically, the following two main strategies are used to improve the model performance: adding a spatial attention mechanism and GAN optimization. In order to verify the performance difference between different networks and the impact of adding GAN optimization to the model on the tobacco leaf vein segmentation effect, some network training hyperparameters such as optimization method, number of training cycles, and dropout rate are set as shown in Table 2.

[0083] Table 2:

[0084]

[0085] 3.3 Application Verification

[0086] like Figure 8 As shown in the figure, the segmentation visualization results of the network model used in this embodiment are shown. The spatial attention mechanism introduced by GAN-SA-UNet can better capture key spatial features, focus on the fine structures and edges in the veins of tobacco leaves, accurately capture complex texture features, and the segmentation results are more accurate, the edges are clearer, and closer to the Ground Truth.

[0087] To further verify the robustness and generalization ability of the GAN-SA-UNet vein segmentation model, the optimal model weights trained were used to verify samples outside the training and test sets, and a large number of tests were conducted on tobacco leaves from different origins (Yunnan, Fujian, Sichuan, Henan), different parts (upper, middle, lower), different varieties (K326, Yun 87, Yun 85, Yueyan 98, Cuibi No. 1), and different types (flue-cured tobacco, cigar tobacco). Some segmentation results are shown above. Fig. 9 As shown in the figure, it can be seen that the model has a certain tolerance for diverse tobacco leaf images, and still maintains stable segmentation performance in the face of uncertainty and complex environments, showing strong generalization ability. Some of the segmented leaf veins in the tobacco leaf have broken leaves, that is, the veins are not continuous. The reason is that the broken tobacco leaves themselves and the folds on the leaf surface cause the occlusion, which makes it impossible to fully expose the veins. This phenomenon is difficult to avoid in actual complex application scenarios. The next step will be to use the image restoration technology in deep learning to reconstruct the broken leaf veins after segmentation to ensure the continuity of the whole tobacco veins.

[0088] 4. Conclusion

[0089] In order to achieve accurate segmentation of fine leaf veins under complex leaf texture conditions, based on hyperspectral imaging, the first three principal components are PCA-reduced into a three-channel image. The PCA explanation rate of the first three principal components of the vein and leaf surface spectra is over 95.5%, and the spectral features of the two after dimension reduction show obvious separability. The synthesis of the first three principal components can highlight the difference between the leaf surface and the veins, making the vein features more prominent.

[0090] The GAN-SA-UNet model helps to capture key spatial features in the tobacco leaf vein segmentation task. The introduction of GAN network optimization training enables the network to learn a wider range of contextual information. Compared with the traditional methods with slow convergence speed, unstable training process, blurred vein edge segmentation and missed segmentation, it has achieved a breakthrough. The IoU achieved is 66.23%, and the inference time for a single image is 4ms.

[0091] The model has been tested on tobacco leaves of different origins, parts, grades, and types. It has maintained stable segmentation performance in the face of uncertainty and complex environments, reflecting the details and texture of the veins, achieving accurate segmentation, and showing strong generalization ability, meeting the performance requirements of industrial applications. This method provides a new idea and solution for tobacco leaf vein segmentation, and provides technical support and guarantee for tobacco leaf quality evaluation, tobacco leaf classification, tobacco leaf processing and other application scenarios.

[0092] The detailed explanation of the above-mentioned embodiments is only intended to explain the present invention so as to facilitate a better understanding of the present invention. However, these descriptions cannot be interpreted as limitations on the present invention for any reason. In particular, the various features described in different embodiments may also be arbitrarily combined with each other to form other embodiments. Unless there is a clear description to the contrary, these features should be understood to be applicable to any embodiment and are not limited to the described embodiments.

Claims

1. A tobacco leaf vein segmentation method based on an improved UNet network model, characterized in that: The tobacco leaf vein segmentation method comprises: Collecting hyperspectral images of tobacco leaf samples and constructing a sample set of hyperspectral images of tobacco leaves; Constructing a leaf vein segmentation model based on a UNet network, wherein the leaf vein segmentation model at least includes an encoder, a spatial attention module, and a decoder; Constructing a generative adversarial network model, inputting the sample set into the leaf vein segmentation model and using the generative adversarial network model to train the leaf vein segmentation model; The vein segmentation is performed on the hyperspectral image of the tobacco leaf to be tested by using the trained vein segmentation model.

2. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 1, characterized in that: In the encoder, the dropout convolution block is used to replace the original convolution block.

3. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 1, characterized in that: The spatial attention module receives as input the output of the encoder.

4. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 3, characterized in that: In the leaf vein segmentation model, its last layer uses 1×1 convolution and Sigmoid activation function to generate the output segmentation map.

5. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 4, characterized in that: In the leaf vein segmentation model, the loss function adopts the cross entropy function, and its formula is: Where: mn is the input image size; x ij Represents a pixel in the image; y ij is the correct segmentation result; g(x ij ) is the SA-UNet segmentation result.

6. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 1, characterized in that: In the leaf vein segmentation model, Inputting a hyperspectral image of a tobacco leaf sample into the encoder, the encoder converting the hyperspectral image into a high-level semantic feature and outputting a high-level feature map; Inputting the high-level feature map into the spatial attention module, the spatial attention module adjusts the spatial information weight of the high-level feature map and outputs the adjusted feature map; The adjusted feature map is input into the decoder, and the decoder generates a segmentation map with tobacco leaf vein markings through the adjusted feature map.

7. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 1, characterized in that: The generative adversarial network model includes: The generator determines whether the data source is real or not; The discriminator is fitted with real data to confuse the discriminator so that the generated data is identified as real.

8. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 7, characterized in that: The training process of the generative adversarial network model is: Where: x is the real data, which conforms to P data (x) distribution; z is noise data, which conforms to P z (z) distribution; D(x) is the probability that x is judged to be true; G(z) is the data generated by the generator; D(G(z)) is the probability that the data generated by the generator is judged to be true.

9. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 7, characterized in that: The leaf vein segmentation model is used as the generator and the multi-layer convolutional network is used as the discriminator.

10. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 9, characterized in that: The generator generates a fake tobacco leaf vein image and a label vein image, and the discriminator captures feature information in the fake tobacco leaf vein image and the label vein image, and calculates the authenticity of the fake tobacco leaf vein image and the label vein image.

11. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 10, characterized in that: Inputting a random vector or noise into the generator, the generator processing the random vector and noise to generate the fake tobacco leaf vein image, wherein the fake tobacco leaf vein image is visually similar to the label vein image; The fake vein image and the label vein image are simultaneously input into the discriminator, and the discriminator outputs a probability value, wherein the probability value indicates the probability that the input image is a real tobacco leaf vein image.

12. The tobacco leaf vein segmentation method based on the improved UNet network model according to claim 1, characterized in that: In the generative adversarial network model, the loss function adopts the cross entropy function and is optimized using the Adam method, and its calculation formula is: loss sum =loss SA-UNet +loss uD(z) +loss Dc(z) Where: loss SA-UNet is the segmentation loss; loss uD(z) is the discriminator loss; loss Dc(z) is the feature loss.

Citation Information

Patent Citations

  • A semi-supervised image instance segmentation method based on stepwise adversarial learning

    CN109949317A

  • Remote sensing image semantic segmentation method based on W divergence adversarial network

    CN112733756A

  • Gastric lesion segmentation method and system based on generative adversarial network

    CN114266786A

  • Breast medical image segmentation method based on improved U-net network

    CN116580202A

  • Buckwheat grain image segmentation method based on ResCo-UNet

    CN117893763A

Cited By

  • Tobacco stem and leaf image segmentation method

    CN120976549A