A deep learning-based tobacco vein extraction and number counting method

Abstract

The application discloses a tobacco green vein extraction and number counting method based on deep learning, which comprises the following steps: S1, collecting green vein tobacco original pictures through a green vein tobacco screening device to obtain a green vein tobacco dataset; S2, performing semantic segmentation labeling on the collected green vein tobacco dataset; S3, performing Albumentations data enhancement on the labeled green vein dataset to increase training samples; S4, transmitting the data-enhanced green vein tobacco dataset into a deep learning instance segmentation Yolact network model for training; S5, collecting green vein tobacco images in real time, inputting the obtained real-time green vein tobacco images into a trained tobacco green vein extraction and segmentation model for inference and prediction to obtain a tobacco green vein region; and S6, counting and counting the number of green veins through a green vein number counting and counting method and outputting the number of green veins through a communication mode. The application can effectively extract the green vein region in tobacco and realize automatic detection and extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision deep learning, and particularly relates to a tobacco vein extraction and number counting method based on deep learning. BACKGROUND

[0002] The tobacco leaf industry occupies a very important position in China. In the process of tobacco leaf curing production, if the curing time of tobacco leaf is not appropriate, it is easy to cause the phenomenon of green and yellow veins, which is collectively referred to as the green vein phenomenon. In most existing tobacco leaf curing production lines, green vein tobacco still relies on manual observation for screening and detection. Human detection has strong subjectivity, and at the same time, due to the limited energy of the staff, it is impossible to maintain detection precision and quality, so it is necessary to use a machine to replace the human visual detection method.

[0003] At present, machine vision is mainly divided into traditional machine vision and deep learning-based visual detection method. The main detection area of green vein tobacco is the tobacco vein part. Compared with the foreground tobacco, the target object is relatively small, and the area edge is relatively not obvious. The traditional machine vision has relatively poor compatibility, and the detection precision is insufficient, which is easy to cause missed detection and over detection, so as to not meet the high-precision and high-efficiency extraction of tobacco green veins. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a tobacco vein extraction and number counting method based on deep learning.

[0005] The technical scheme of the present application is as follows:

[0006] A tobacco vein extraction and number counting method based on deep learning, comprising the following steps:

[0007] S1, collecting green vein tobacco original pictures through a green vein tobacco screening device to obtain a green vein tobacco data set;

[0008] S2, performing semantic segmentation annotation on the collected green vein tobacco data set;

[0009] S3, performing Albumentations data enhancement on the annotated green vein data set to increase the training samples;

[0010] S4, inputting the data enhanced green vein tobacco data set into a deep learning instance segmentation Yolact network model for training;

[0011] S5, collecting green vein tobacco images in real time, inputting the obtained real-time green vein tobacco images into the trained tobacco vein extraction and segmentation model for inference and prediction to obtain the tobacco vein area;

[0012] S6, count the number of green veins by the green vein number counting method, and output the number of green veins through communication.

[0013] In step S1, the green vein tobacco screening device comprises a tobacco image acquisition mechanism and a tobacco green vein extraction algorithm platform.

[0014] The tobacco image acquisition mechanism comprises a camera, a lens, a light source, and a light source controller.

[0015] The camera adopts the model HS-E54200-10GM, the lens adopts the model HS-FA162016A, the light source adopts the model HS-BAR-237-38-WK, and the light source controller adopts the model HS-APS2460B-4TD.

[0016] The tobacco green vein extraction algorithm platform comprises an HSVersion visual software.

[0017] In step S3, the Albumentations data enhancement comprises synchronous preprocessing of green vein tobacco image and labeled mask image, and the preprocessing comprises four image processing functions of RandomCrop, HorizontalFlip, VerticalFlip and ShiftScaleRotate.

[0018] Step S4 specifically comprises:

[0019] S41, load the COCO dataset pre-training model weight to realize transfer learning and reduce the training amount of the dataset;

[0020] S42, the picture enters the Backbone main network part to perform deep network feature extraction;

[0021] S43, the convolution feature layer enters the FPN network to generate a mask region with higher quality and higher robustness;

[0022] S44, the multi-scale fusion convolution layer features are transmitted into the Protonet network to generate a mask region, and a plurality of scale convolution layer features are transmitted into the Prediction Head network in parallel to perform class confidence, position offset and mask coefficient;

[0023] S45, the class confidence, position offset and mask coefficient are spliced, and the spliced result is subjected to threshold processing to finally generate a green vein mask region.

[0024] In step S41, the COCO dataset pre-training model weight is a model parameter obtained by pre-training a Yolact main network Resnet50 based on a COCO dataset, and the COCO dataset includes more than 80 categories, 40890 training pictures and 5000 test pictures.

[0025] In step S42, the main network includes five 3*3 convolution modules, and the number of channels of the five 3*3 convolution modules is 64, 128, 256 and 512 respectively.

[0026] In step S6, the green vein number counting method is to count the number of instance objects by outputting a mask coefficient through a Prediction Head, and the mask coefficient is a green vein corresponding instance object, and the corresponding coefficient is output and calculated, so that the specific number of green veins of the tobacco leaf is obtained.

[0027] Compared with the prior art, the present application has the following advantages:

[0028] 1. The present application adopts a Yolcat instance segmentation model to realize automatic extraction of the tobacco green vein area, count the number of green veins, distinguish green tobacco from normal tobacco, and has the technical effects of precision, effectiveness and practicality.

[0029] 2. The present application uses deep learning instance segmentation technology to realize tobacco green vein extraction and number counting, solves the problems of difficult tobacco green vein extraction and insufficient human selection efficiency, and solves the problems of relatively poor compatibility of traditional machine vision, insufficient detection accuracy and easy to cause missed detection and over detection. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0031] Figure 1 A flowchart of a tobacco green vein extraction and number counting method based on deep learning provided by the present application;

[0032] Figure 2 A green tobacco dataset instance diagram in step S2 of the present application;

[0033] Figure 3 An Albumentations data enhancement flowchart in step S3 of the present application;

[0034] Figure 4 The Albumentations data enhancement effect diagram for step S3 in the application;

[0035] Figure 5 The Yolact network model structure diagram for step S4 in the application;

[0036] Figure 6 The Backbone backbone network structure diagram for step S4 in the application;

[0037] Figure 7 The green vein number counting statistical method flow chart for step S6 in the application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the application clearer and more understandable, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0039] In order to illustrate the technical scheme described in the application, the following will be described through specific examples.

[0040] EMBODIMENT

[0041] The embodiment provides a tobacco green vein extraction and number counting method based on deep learning, as shown in the figure, comprising the following steps: Figure 1

[0042] S1, collecting green vein tobacco original pictures through a green vein tobacco screening device to obtain a green vein tobacco dataset;

[0043] The green vein tobacco screening device comprises a tobacco image acquisition mechanism and a tobacco green vein extraction algorithm platform.

[0044] The tobacco image acquisition mechanism comprises a camera, a lens, a light source and a light source controller. In the embodiment, the model of the camera is HS-E54200-10GM, the model of the lens is HS-FA162016A, the model of the light source is HS-BAR-237-38-WK, and the model of the light source controller is HS-APS2460B-4TD.

[0045] The tobacco green vein extraction algorithm platform comprises an HSVersion visual software, which realizes image preprocessing, tobacco green vein extraction and counting, and output of green vein tobacco results on the green vein tobacco images collected by the tobacco image acquisition mechanism.

[0046] S2, performing semantic segmentation annotation on the collected green vein tobacco dataset;

[0047] In which, as​Figure 2 As shown, the green vein tobacco dataset includes green vein tobacco and qualified tobacco, and in this embodiment, 1200 tobacco leaves are selected as the model training dataset, the model training dataset is divided into a dataset in a ratio of 8:2, including 960 training sets and 240 validation datasets, and the input image size is 544*544;

[0048] S3, the labeled green vein dataset is subjected to Albumentations data enhancement to increase the training samples;

[0049] The Albumentations data enhancement is a professional image processing library, which can process multiple data types, including RGB images, multispectral images, grayscale images, segmentation mask regions, key points, and bounding boxes. Compared with the processing methods provided by PyTorch and TensorFlow, it is faster and can be better embedded and applied in detection projects to achieve fast processing.

[0050] The Albumentations data enhancement includes synchronous preprocessing of green vein tobacco images and labeled mask images. Specifically, the Random_params generated by marking the green vein dataset Albumentations data enhancement function is used to perform image pixel-level transformation or spatial-level transformation on the original image, and simultaneously perform image pixel-level transformation or spatial-level transformation on the labeled mask image, and uniformly name it, thereby achieving data set augmentation, solving the tedious and laborious problem of segmentation dataset labeling, and the preprocessing includes RandomCrop, HorizontalFlip, VerticalFlip, and ShiftScaleRotate four image processing functions.

[0051] The specific implementation steps are as shown in Figure 3 The green vein tobacco dataset is labeled to obtain green vein tobacco images and labeled mask images, and the Albumentations image processing function is used to simultaneously perform data augmentation operations on the green vein tobacco images and the labeled mask images, thereby achieving green vein tobacco dataset augmentation and improving the fitting effect and generalization ability of the model training.

[0052] The Albumentations data enhancement is as shown in Figure 4 It includes random cropping, vertical rotation, horizontal flipping, and random rotation four image processing effects.

[0053] S4, the data enhanced green vein tobacco dataset is transmitted to the deep learning instance segmentation Yolact network model for training, specifically:

[0054] S41. Load the pre-trained model weights from the COCO dataset to achieve transfer learning and reduce the amount of training data.

[0055] S42. The image enters the Backbone network for deep network feature extraction. The backbone network includes five 3*3 convolutional modules, and the number of channels corresponding to the five 3*3 convolutional modules are 64, 128, 256 and 512 respectively.

[0056] S43. The convolutional feature extraction layer is fed into the FPN network to generate a higher quality and more robust Mask region;

[0057] S44. Input the multi-scale fused convolutional layer features into the Protonet network to generate the mask region, and input the multi-scale convolutional layer features into the Prediction Head network in parallel to perform class confidence, position offset, and mask coefficients.

[0058] S45. The category confidence, position offset, and mask coefficient are spliced ​​together, and the splicing result is subjected to a cropping threshold to finally generate the blue vein mask region.

[0059] Among them, such as Figure 5 As shown, the Yolact network model structure is mainly divided into three parts: the Backbone network, FPN, Prediction Head, and Protonet. The image input is processed by the Backbone network for feature extraction, and the output is an effective feature layer with a downsampling factor of 3 (C3), 4 (C4), and 5 (C5), which is then fed into the FPN structure. The FPN outputs five effective feature layers (P3-P7) of different depths, which are then fed into the Prediction Head structure. P3 is fed into the Protonet structure for pixel-level segmentation. The Prediction Head output is processed by Fast NMS to filter the obtained ROI positions, and then matrix multiplication is performed with the Protonet structure output to obtain the corresponding instance mask. Finally, foreground and background processing is performed using Crop and Threshold.

[0060] The backbone network portion of the Backbone uses a ResNet 50 network structure, such as... Figure 6 As shown, it includes multiple Bottleneck Layers. Each Bottleneck Layer includes a main branch and a shortcut. The main branch includes a 1*1 convolution, a 3*3 convolution, and a 1*1 convolution, which realizes dimensionality reduction feature convolution and improves the computation speed.

[0061] The FPN structure is specifically that the C3, C4 and C5 effective feature layers output by the Backbone main network are fused in multiple sizes, the C5 is extracted to obtain an 18*18 feature layer (P5) through convolution, the P5 is respectively up-sampled and down-sampled, the up-sampling refers to that the P5 is up-sampled to obtain a 35*35 feature layer through bilinear interpolation, and the P4 layer is obtained by adding the feature layer after convolution of the C4, the P3 layer is obtained by the same method, and the down-sampling refers to that the P5 is down-sampled to obtain a P6 through convolution, and the P7 is obtained by down-sampling the P6 through convolution;

[0062] The deep learning neural network training parameters are specifically set as follows:

[0063] The confidence threshold is set to 0.5, the NMS_iou is set to 0.3, the anchor size is [24, 48, 96, 192, 384], the batch size is set to 16, the training round is set to 200 rounds, the learning rate is 0.0001, and the SGD optimizer is adopted. The loss function Loss is composed of a class confidence Loss, a position offset Loss and a mask loss;

[0064] The SGD optimizer is a model parameter updating method, only one sample gradient is calculated for each parameter update, and in the case of limited training samples, the optimal solution can be effectively iterated to;

[0065] The class confidence Loss and the position offset Loss both adopt the Smooth L1 Loss, and the mask loss adopts a pixel-level binary cross-entropy;

[0066] The Smooth L1 loss is a smooth change L1, and is specifically:

[0067]

[0068] The pixel-level binary cross-entropy is specifically:

[0069] BinaryCrossEntropy=-P*logQ-(1-P)*log(1-q), P is a prediction result corresponding to a blue vein, and Q is a background prediction result;

[0070] The COCO dataset pre-training model weight is a model parameter obtained by pre-training a Yolact main network Resnet50 based on a COCO dataset, and the COCO dataset includes more than 80 categories, 40890 training pictures and 5000 test pictures;

[0071] S5, real-time collection of green vein tobacco image, input the obtained real-time green vein tobacco image into the trained tobacco green vein extraction segmentation model for inference prediction, and obtain the tobacco green vein area;

[0072] The real-time green vein tobacco image is an original tobacco image without training, and includes green vein tobacco and normal tobacco.

[0073] S6, counting and statistics are performed through the green vein number counting and statistics method, and the green vein number is output through a communication mode.

[0074] The green vein number counting and statistics method is as shown in Figure 7 The mask coefficient is the green vein corresponding instance object, and the corresponding coefficients are superimposed and calculated to obtain the specific green vein number of the green vein tobacco, and the green vein tobacco result is output by comparing with the set threshold value.

[0075] Through the test of inputting 200 real-time collected green vein tobacco images into the tobacco green vein extraction segmentation model, the green vein threshold number is set to 3, and the pixel accuracy (Pixel Accuracy), accuracy (Accuracy) are used to judge the green vein area segmentation ability of the model and the green vein tobacco detection effect.

[0076] The results show that the pixel accuracy is 83%, the accuracy is 89%, and the test frame rate is 16 fps. It can be seen that the tobacco green vein extraction ability of the method is better, the accuracy is high, and the method meets the agricultural tobacco detection demand.

[0077] The above is only a preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A deep learning-based tobacco vein extraction and number counting method, characterized in that, The method comprises the following steps: S1, collecting original pictures of green vein tobacco leaves through a green vein tobacco leaf screening device to obtain a green vein tobacco leaf dataset; S2, performing semantic segmentation annotation on the collected green vein tobacco leaf dataset; S3, performing Albumentations data enhancement on the annotated green vein dataset to increase training samples; The Albumentations data enhancement comprises synchronous preprocessing of green vein tobacco leaf images and annotation mask images, and the preprocessing comprises four image processing functions of RandomCrop, HorizontalFlip, VerticalFlip and ShiftScaleRotate; S4, transmitting the data-enhanced green vein tobacco leaf dataset into a deep learning instance segmentation Yolact network model for training; Step S4 specifically comprises: S41, loading COCO dataset pre-training model weights to realize transfer learning and reduce dataset training amount; S42, the picture enters a Backbone main network part to perform deep network feature extraction; S43, the convolutional feature layer enters an FPN network to generate mask regions with higher quality and better robustness; S44, the multi-scale fusion convolutional layer features are transmitted into a Protonet network to generate mask regions, and the multi-scale convolutional layer features are transmitted into a Prediction Head network in parallel to perform class confidence, position offset and mask coefficient; S45, the class confidence, position offset and mask coefficient are spliced, and the spliced result is subjected to threshold processing to finally generate green vein mask regions; S5, real-time collection of green vein tobacco leaf images, input of the obtained real-time green vein tobacco leaf images into the trained tobacco green vein extraction and segmentation model for inference and prediction to obtain tobacco green vein regions; S6, counting and statistics through a green vein number counting and statistics method, and output of the green vein number through a communication mode.

2. The method according to claim 1, wherein, In step S1, the green vein tobacco leaf screening device comprises a tobacco leaf image collection mechanism and a tobacco green vein extraction algorithm platform.

3. The method according to claim 2, wherein, The tobacco leaf image collection mechanism comprises a camera, a lens, a light source and a light source controller.

4. The method according to claim 3, wherein, The camera adopts a model of HS-E54200-10GM, the lens adopts a model of HS-FA162016A, the light source adopts a model of HS-BAR-237-38-WK, and the light source controller adopts a model of HS-APS2460B-4TD.

5. The method according to claim 2, wherein, The tobacco green vein extraction algorithm platform comprises an HSVersion visual software.

6. The method according to claim 1, wherein, In step S41, the COCO dataset pre-training model weights are model parameters obtained by pre-training a Yolact main network Resnet50 based on a COCO dataset, and the COCO dataset comprises more than 80 categories, 40890 training pictures and 5000 test pictures.

7. The method according to claim 1, wherein, In step S42, the main network comprises five 3*3 convolution modules, and the channel numbers of the five 3*3 convolution modules are 64, 128, 256 and 512 respectively.

8. The method according to claim 1, wherein, In step S6, the number of green veins counting method is to count the number of instances by outputting the mask coefficient through the prediction head. The mask coefficient corresponds to the instance object of the green veins. By outputting and calculating the corresponding coefficient, the specific number of green veins of the tobacco leaves can be obtained.

Images