Online detection method for threshed tobacco stems in threshing and redrying process

Abstract

The invention relates to an online tobacco stem detection method after threshing in a threshing and redrying process, which comprises the following steps of: erecting an image acquisition and analysis device on a production line, collecting data according to a preset mode, preprocessing the data, training and testing a preset tobacco stem segmentation technical model; the model is used for actual detection, and a detection result is processed by an area ratio and mass ratio association method to predict the tobacco stem mass ratio. The method has the advantages that the area ratio of the tobacco stems in the pictures of the outermost surface of the tobacco leaf and tobacco stem mixture is associated with the mass ratio of the tobacco stems in the tobacco leaf and tobacco stem mixture, and the association is quantified in a specific scene by using an artificial intelligence method; the effect of predicting the mass ratio by solving the area ratio is achieved, and various defects of directly or indirectly measuring the mass ratio traditionally are overcome.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to an online detection method for tobacco stems after threshing in a threshing and re-roasting process. Background Art

[0002] Leaf beating and re-baking is an important step in the production process of cigarettes. In this step, the stem content in the leaves is a key processing quality control indicator. Tobacco stems entering the cigarettes can easily cause the cigarettes to be punctured, leak, or even burst at the burning end, which not only affects the combustibility of the cigarettes, the stability of physical test indicators, but also affects the sensory quality of the entire cigarette.

[0003] The stem content in leaves is a measure of the mass proportion of tobacco stems in a mixture of tobacco leaves and stems. Its traditional measurement generally requires separating the leaves and stems first, then weighing the leaves and stems separately, and finally calculating. The separation of leaves and stems is the most important and difficult step. Traditional measurement methods include manual weighing and wind-separated weighing. Manual weighing method: 3-5kg is sampled each time, and free leaves, stem-containing leaves, and free stems are manually selected and weighed to obtain the ratio. Wind-separated weighing method: Sample a certain amount of material, repeatedly crush it, completely separate the stems and leaves, and then perform wind separation to blow the material up. The weight of the stems and leaves is different. According to the different suspension speeds, the light leaves are blown away and the heavy stems are left. The ratio is calculated by weighing.

[0004] However, both methods have problems such as long detection time and inconvenient operation.

[0005] The computer vision method only requires the installation of an image capture device on the leaf threshing and re-roasting line, and the acquired images can be processed almost instantly. With the rapid development of computers, the processing time for a single image or even a video image is extremely short and the processing result can be obtained. Therefore, using computer vision methods to detect and separate leaves and stems can effectively alleviate these problems. After segmenting the leaf stems, the area ratio of leaves to stems can be calculated, which can also provide data support for the subsequent calculation of the stem content in leaves through images.

[0006] In order to measure the above-mentioned area ratio, there are two main methods in the existing technology. The first is to directly measure the area ratio. X-rays are mostly used to collect images for the identification and detection of leaves and stems. Different grayscale images will be obtained according to the different density characteristics of tobacco stems and tobacco leaves. The tobacco stems are then identified and detected through image denoising, edge detection, threshold segmentation and other methods. The disadvantage of this method is that the radiation generated by X-rays is worrying and there is a risk of affecting health.

[0007] The second is to indirectly measure the area ratio. This method is to detect the data of thinned or separately separated leaf stems. However, this method of calculating the area ratio of leaves and stems requires calculating the area of ​​the complete leaves and stems without any omissions. This leads to a big disadvantage: the operator is required to sample the mixture on the production line, and then completely separate the tobacco leaves and stems and spread them flat on a plane. In order to ensure the integrity of the area display of leaves and stems, there cannot be any stacking or obstruction in this separation process. This operation does not actually omit the manual sampling and measurement process. The machine must be stopped for sampling, which interrupts the production process and takes up a lot of manual time for separation and spreading operations.

[0008] Therefore, a new method is needed to detect the weight proportion of tobacco stems in actual production. Summary of the invention

[0009] In order to overcome the problems in the background technology: traditional manual detection and wind analysis detection have the problems of long detection time and inconvenient operation; the direct measurement method using X-rays in the current computer vision method has health and safety risks, and the indirect measurement method of thinning or individually separating the leaf stalks has the problem of manual sampling and manual separation and thinning that consumes manpower and time.

[0010] The present invention proposes an online detection method for tobacco stems after threshing in a threshing and re-roasting process, comprising: an online detection method for tobacco stems after threshing in a threshing and re-roasting process, comprising setting up an image acquisition and analysis device on a production line, and also comprising:

[0011] Collect data in a preset way, pre-process the data, and divide it into training set and test set;

[0012] Using the training set to train a preset tobacco stem segmentation technology model, and using the test set to test the preset tobacco stem segmentation technology model;

[0013] The preset tobacco stem segmentation technology model is used for actual testing, and the test results are processed by the area ratio and mass ratio correlation method to predict the tobacco stem mass ratio.

[0014] Preferably, the image acquisition and analysis device includes an industrial camera, a light source system, structural parts, a data processing device, an acquisition card, a data transmission line, and a high-performance processor.

[0015] Preferably, the collecting of data in a preset manner, preprocessing of the data, and dividing the data into a training set and a test set comprises:

[0016] Obtaining a surface image of a mixture of tobacco leaves and tobacco stems on a production line and an artificially annotated tobacco stem mask image corresponding to the surface image;

[0017] Screening whether the surface image and the mask image are representative;

[0018] Perform data augmentation on the filtered images;

[0019] The data-augmented images are divided into training sets and test sets according to a preset ratio.

[0020] Preferably, the preset tobacco stem segmentation technology model is a U-Net++ model.

[0021] Preferably, the processing of the detection results by the method of associating area ratio with mass ratio to predict the tobacco stem ratio comprises:

[0022] Count the number of tobacco stem pixels in a single image:

[0023] s n =Sum(Img n (x,y)==255) (1)

[0024] Where: S n is the number of pixels of the tobacco stem in the nth image, n is the nth image, and Img(x, y) is the value of the coordinate (x, y) in the image;

[0025] Calculate the pixel ratio of tobacco stems in a single image:

[0026]

[0027] Calculate the weight percentage of tobacco stems in a single image:

[0028]

[0029] Where: ρ1 is the average density of tobacco stems, ρ1 is the average density of tobacco leaves, V1 is the volume of tobacco stems, and V2 is the volume of tobacco leaves;

[0030] Calculate the minimum number of cumulative images:

[0031]

[0032] Where: m true is the actual weight proportion of tobacco stems during the cumulative time measured offline;

[0033] According to formula (4), the minimum n is obtained, and the value is N, where N is the minimum number of accumulated images in a steady-state output state. The steady-state output state is that the average value of the weight percentage of tobacco stems in a single image calculated cumulatively is very close to the average value of the weight percentage of real tobacco stems measured cumulatively offline, and the very close means that the error between the above two is less than or equal to 5%;

[0034] That is: when When , it is considered that formula (4) holds;

[0035] Then find the minimum cumulative time:

[0036] T=N / f (5)

[0037] Where: f is the camera shooting frequency

[0038] The beneficial technical effects of the present invention are:

[0039] (1) It overcomes the problems of traditional manual detection and wind separation detection, such as long detection time, labor consumption, and delayed results. It can monitor while production is in progress, achieving a significant improvement in efficiency.

[0040] (2) The present invention creatively associates the area ratio of the tobacco stems in the surface photo of a pile of tobacco leaf and tobacco stem mixture with the mass ratio of the tobacco stems in the pile of tobacco leaf and tobacco stem mixture, and uses artificial intelligence methods to quantify this association in a specific scenario, thereby achieving the effect of predicting the mass ratio by calculating the area ratio, and solving the various disadvantages of traditional direct or indirect measurement of mass ratio: the present invention can directly use an industrial camera to collect images and analyze them in real time to obtain results. This process does not use any X-ray device, eliminating health risks; the present invention can directly sample the tobacco stems and tobacco leaf mixture on the production line in the actual measurement link, without stopping the machine for sampling, thus ensuring the continuity of production and significantly reducing the detection cost. At the same time, it is no longer necessary to manually separate or thin the tobacco stems and tobacco leaf mixture in the sampling, thus saving a lot of labor; the present invention uses the U-Net++ model through experiments, which achieves relatively better results in the application scenario of the present invention, with richer details of the labeled tobacco stems, thereby improving the final detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 Yes: Schematic diagram of the U-Net++ model structure.

[0043] Figure 2 Yes: Schematic diagram of data augmentation.

[0044] Figure 3 : The loss curves of the training set and the validation set of the embodiment.

[0045] Figure 4Yes: Example training set and validation set IoU curve diagram

[0046] Figure 5 They are: original tobacco stem image, manually annotated mask, and output prediction images of each model.

[0047] Figure 6 Yes: flow chart of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention.

[0050] like Figure 1-6 As shown, in order to solve the problems in the background technology:

[0051] (1) The traditional manual weighing method and wind weight weighing method have problems such as long detection time and inconvenient operation.

[0052] (2) The direct measurement of area ratio in traditional machine vision methods will cause worrying X-ray radiation and pose a risk to health; the indirect measurement of area ratio has the disadvantage that the machine must be stopped for sampling, interrupting the production process and taking up a lot of man-hours for separation and averaging operations.

[0053] A method for online detection of tobacco stems after threshing in a threshing and re-roasting process, comprising: setting up an image acquisition and analysis device on a production line; collecting data in a preset manner, preprocessing the data, and dividing the data into a training set and a test set; using the training set to train a preset tobacco stem segmentation technology model, and using the test set to test the preset tobacco stem segmentation technology model; using the model to perform actual detection, and processing the detection results by an area ratio and mass ratio correlation method to predict the tobacco stem ratio.

[0054] Specifically, the image acquisition and analysis device includes an industrial camera, a light source system, structural parts, a data processing device, including an acquisition card, a data transmission line, and a high-performance processor. These devices ensure that the production line can be imaged in real time, and the acquired images can be immediately entered into the data processing device for processing.

[0055] Specifically, the data is collected in a preset manner, preprocessed, and divided into a training set and a test set, including:

[0056] The data preprocessing includes: in the actual production process, the industrial camera instantly obtains the surface images of the mixture of tobacco leaves and tobacco stems on the production line. These images are as follows: Figure 2 The original tobacco stem image shown in (a) is subjected to a mask operation to obtain a corresponding mask image. The mask operation is to mark the outline of the tobacco stem using open source annotation software to form a contour line, and then convert it into a mask image through image processing, such as Figure 2 As shown in (d), the image is a gray image, the tobacco stems are white, and the rest are black. Since the mask image is manually labeled, different people may have different visual perceptions and understandings of tobacco stems. For example, some people may label smaller tobacco stems, while others may not. Therefore, screening and unified processing are required to avoid data confusion, which results in the disadvantage that the final model has good recognition but low accuracy. Therefore, the present invention screens the surface image and the mask image to see if they are representative.

[0057] The representative screening operation is: since the pictures are taken online and actual measurement is impossible, manual empirical judgment is required. In actual operation, the main vein and primary branches are retained, and the remaining tiny tobacco stems are not marked.

[0058] More specifically, in order to expand the data set, the present invention performs horizontal and vertical flipping operations on the original tobacco stem image, such as Figure 2 As shown in (b), (c), (e) and (f), the total number of data sets will be relatively sufficient, which is conducive to subsequent model training and testing. The training set images in the present invention reach 1728 and the test set images reach 216.

[0059] Specifically, the preset tobacco stem segmentation technology model is a U-Net++ model, and the structure diagram of the model is as follows: Figure 1 The U-Net++ model encoder and decoder also have multiple convolution blocks. Each convolution block obtains deep feature information of the image and changes the number of channels of the feature map through two consecutive 3×3 or 5×5 convolution kernels and ReLu activation functions, thereby enhancing the nonlinear learning ability of the model.

[0060] More specifically, the U-Net++ model connects the feature maps of multiple decoder layers to further improve the model's accurate segmentation of the target. The encoder consists of multiple convolution blocks, each of which contains a continuous 3×3 convolution layer, a batch normalization layer (BN) and a ReLu activation function. The encoder part constitutes the backbone network of the model. The present invention selects the VGG16 network as the backbone network of the U-Net++ model. The decoder consists of multiple convolution blocks and upsampling operations. The decoder convolution block is similar to the encoder part. The decoder uses deconvolution for upsampling operations, and each decoder convolution block is connected to the output of the corresponding encoder convolution block and the previous decoder convolution block. The detailed information is restored by upsampling step by step, and dense jump connections are made with the low-level features of the encoder to better capture the multi-scale features in the tobacco stem image, thereby improving the model's perception of the details and shapes of the tobacco stem. At the end of the encoder, the size of the feature map is restored to the original image size, and the feature dimension reduction is also performed using a 1×1 convolution kernel, which is mapped to the number of channels that matches the number of task categories, and the pixels are classified using the Softmax function.

[0061] More specifically, U-Net++ has a stronger generalization ability and can better adapt to different data sets and changes in tobacco stem images, so as to obtain better performance in different application scenarios and better preserve the boundary details and shapes of tobacco stems. The loss function is used here to evaluate the U-Net++ model selected by the present invention. The loss function is a crucial component in the tobacco stem image segmentation task, and it plays a key role in the network training process. Common loss functions in segmentation tasks include cross entropy loss function, Dice loss function, and structural similarity index (SSIM) loss function. Because tobacco stem images have the characteristics of uneven color and texture, slender and curved shapes, incomplete boundaries, diversity and variability, background interference and low contrast. These characteristics pose a higher challenge to the tobacco stem image segmentation algorithm, so the present invention adopts a loss function that combines binary cross entropy (Binary Cross Entropy) and Dice similarity, namely BCEDiceLoss. The binary cross entropy loss can effectively deal with the problem of class imbalance between the foreground and background of tobacco stem images, while the Dice coefficient loss can measure the similarity of the segmentation results. Combining these two loss functions can comprehensively consider the classification and segmentation accuracy at the pixel level and improve the quality of tobacco stem image segmentation. The loss function formula is defined as (1), where Y b For manual annotation masks, is the output predicted by the model.

[0062]

[0063] More specifically, the evaluation indicators selected by the present invention are precision, recall, F1-score and IoU (Intersection over Union) as evaluation indicators. IoU is used to evaluate the degree of overlap between the tobacco stem segmentation results of the model and the manually labeled true segmentation mask. The value range of IoU is 0 to 1, where 1 indicates complete overlap and 0 indicates no overlap. The evaluation indicator definition is shown in formula (2-5):

[0064]

[0065] More specifically, parameter setting plays a key role in the tobacco stem image segmentation experiment. Correctly selecting and adjusting parameters can significantly affect the accuracy and performance of tobacco stem segmentation results. In order to achieve high-quality tobacco stem image segmentation, the present invention conducted a series of experiments, and the experimental parameter settings used are shown in Table 1:

[0066] Table 1 Experimental parameter settings

[0067]

[0068] More specifically, whether the tobacco stem segmentation model has overfitting can be tested by the change curve of the loss value (Loss) and IoU during the training process. Figure 3 The training loss function curve and validation set loss function curve show the change of the model's loss value in each training iteration. It can be observed from the figure that as the model training progresses, the loss value gradually decreases and tends to stabilize. This shows that the tobacco stem segmentation model gradually adapts to the task during the training process and can effectively segment tobacco stems.

[0069] IoU is a commonly used indicator for evaluating image segmentation results. It reflects the degree of overlap between the prediction results of the model and the actual mask segmentation. The experimental results of the IoU indicator are as follows: Figure 4 As shown in the figure, in the initial stage, the IoU value is low because the model has not yet fully learned the characteristics and complexity of the tobacco stems in the image. As the training progresses, the IoU value gradually increases and the performance of the model gradually improves. Finally, the IoU value reaches a stable state with an average value of 0.2366, indicating that the model can accurately segment the tobacco stems.

[0070] Depend on Figure 5 As shown, by comparing multiple traditional models, the present invention finds that the U-Net++ model exhibits higher accuracy and robustness, that is, the (f) U-Net++ model is closer to the (c) SegNet model, (d) FCN model and (e) U-Net model and (b) Mask, and is therefore relatively more suitable for the application scenario of the present invention.

[0071] When the U-Net++ model used in the present invention is trained, the model will obtain a mask image generated by the model based on real-time photos taken on the actual production line, and process it according to the area ratio and mass ratio association method proposed in the present invention, so as to infer the mass ratio of tobacco stems in the mixture of tobacco stems and tobacco leaves in the photo.

[0072] Specifically, the method for associating area ratio with mass ratio includes:

[0073] Count the number of tobacco stem pixels in a single image:

[0074] s n =Sum(Img n (x,y)==255) (6)

[0075] Where: S n is the number of pixels of the tobacco stem in the nth image, n is the nth image, and Img(x, y) is the value of the coordinate (x, y) in the image;

[0076] Calculate the pixel ratio of tobacco stems in a single image:

[0077]

[0078] Calculate the weight percentage of tobacco stems in a single image:

[0079]

[0080] Where: ρ1 is the average density of tobacco stems, ρ1 is the average density of tobacco leaves, V1 is the volume of tobacco stems, and V2 is the volume of tobacco leaves;

[0081] Calculate the minimum cumulative number of images:

[0082]

[0083] Where: m true is the actual weight proportion of tobacco stems during the cumulative time measured offline;

[0084] According to formula (4), the minimum n is obtained, and the value is N, where N is the minimum number of accumulated images in a steady-state output state. The steady-state output state is that the average value of the weight percentage of tobacco stems in a single image calculated cumulatively is very close to the average value of the weight percentage of real tobacco stems measured cumulatively offline, and the very close means that the error between the above two is less than or equal to 5%;

[0085] That is: when When , it is considered that formula (4) holds;

[0086] Then find the minimum cumulative time:

[0087] T=N / f (10)

[0088] Where: f is the camera shooting frequency;

[0089] The minimum number of accumulated images N and the minimum accumulated time T are the minimum number of images required for the detection result to be representative or the time required to capture the minimum number of images at a certain shooting frequency of the camera. The representativeness of the detection result is the steady-state output state of the detection result. When the detection result is in the steady-state output state, the actual weight proportion of tobacco stems within any minimum accumulated time will be calculated by combining the formulas (8) and (9).

[0090] Depend on Figure 1-6 It can be seen that the present invention creatively associates the area ratio of the tobacco stems in the surface photo of a pile of tobacco leaf and stem mixture with the mass ratio of the tobacco stems in the pile of tobacco leaf and stem mixture, and uses artificial intelligence methods to quantify this association in specific scenarios, achieving the effect of predicting the mass ratio by calculating the area ratio, and solving the various disadvantages of traditional direct or indirect measurement of mass ratio.

[0091] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0092] Those skilled in the art can understand that the components in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The components in the above embodiment can be combined into one component, or can be further divided into multiple sub-components.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate rather than limit the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the present invention can still be modified or replaced by equivalents. Any modification or partial replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for detecting tobacco stems online after threshing during threshing and re-roasting, comprising setting up an image acquisition and analysis device on a production line, characterized in that: Also includes: Collect data in a preset way, pre-process the data, and divide it into training set and test set; Using the training set to train a preset tobacco stem segmentation technology model, and using the test set to test the preset tobacco stem segmentation technology model; The preset tobacco stem segmentation technology model is used for actual testing, and the test results are processed by the area ratio and mass ratio correlation method to predict the tobacco stem mass ratio.

2. The method for online detection of tobacco stems after threshing in the threshing and re-roasting process according to claim 1, characterized in that: The image acquisition and analysis device includes an industrial camera, a light source system, structural parts, a data processing device, an acquisition card, a data transmission line, and a high-performance processor.

3. The method for online detection of tobacco stems after threshing in the threshing and re-roasting process according to claim 1, characterized in that: The data is collected in a preset manner, preprocessed, and divided into a training set and a test set, including: Obtaining a surface image of a mixture of tobacco leaves and tobacco stems on a production line and an artificially annotated tobacco stem mask image corresponding to the surface image; Screening whether the surface image and the mask image are representative; Perform data augmentation on the filtered images; The data-augmented images are divided into training sets and test sets according to a preset ratio.

4. The method for online detection of tobacco stems after threshing in the threshing and re-roasting process according to claim 1, characterized in that: The preset tobacco stem segmentation technology model is the U-Net++ model.

5. The method for online detection of tobacco stems after threshing in the threshing and re-roasting process according to claim 1, characterized in that: The processing of the detection results by the method of associating area ratio with mass ratio to predict the tobacco stem ratio comprises: Count the number of tobacco stem pixels in a single image: s n =Sum(Img n (x,y)==255) (1) Where: S n is the number of pixels of the tobacco stem in the nth image, n is the nth image, and Img(x, y) is the value of the coordinate (x, y) in the image; Calculate the pixel ratio of tobacco stems in a single image: Calculate the weight percentage of tobacco stems in a single image: Where: ρ1 is the average density of tobacco stems, ρ1 is the average density of tobacco leaves, V1 is the volume of tobacco stems, and V2 is the volume of tobacco leaves; Calculate the minimum number of cumulative images: Where: m true is the actual weight proportion of tobacco stems during the cumulative time measured offline; According to formula (4), the minimum n is obtained, and the value is N, where N is the minimum number of accumulated images in a steady-state output state. The steady-state output state is that the average value of the weight percentage of tobacco stems in a single image calculated cumulatively is very close to the average value of the weight percentage of real tobacco stems measured cumulatively offline, and the very close means that the error between the above two is less than or equal to 5%; That is: when When , it is considered that formula (4) holds; Then find the minimum cumulative time: T=N / f (5) Where: f is the camera shooting frequency.

Images