A tobacco grading method based on upper layer tobacco segmentation
By using a tobacco leaf grading method based on a 3D camera and YOLOv8-PiCO, the problem of uncertainty in identifying the top layer of tobacco leaves in the automatic grasping scheme is solved, achieving efficient and accurate tobacco leaf grading that is suitable for actual production environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHU TUS VISION INFORMATION TECH CO LTD
- Filing Date
- 2023-03-13
- Publication Date
- 2026-05-15
AI Technical Summary
Existing tobacco grading methods have difficulty accurately identifying the top layer of tobacco leaves in automated grasping schemes, resulting in uncertainty and low accuracy in identification and grading.
The method employs 3D camera-based tobacco leaf image acquisition, uses YOLOv8 for top layer tobacco leaf segmentation, combines PiCO for multi-label annotation and 3D structured light technology for top layer tobacco leaf screening, and combines national tobacco leaf grading standards for partial label learning to achieve efficient and accurate tobacco leaf grading.
This system enables efficient and accurate grading of the top layer of tobacco leaves in an automated grasping scheme, improving the automation and intelligence level of tobacco leaf grading and making it suitable for actual production environments.
Smart Images
Figure CN116309439B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of machine vision technology, specifically relating to a tobacco leaf grading method based on upper layer tobacco leaf segmentation. Background Technology
[0002] Tobacco leaf grading technology plays a crucial role in the automated tobacco industry. Existing tobacco leaf grading methods include two approaches: (1) extracting tobacco leaf features from images acquired using a 2D camera and then grading the tobacco leaves; (2) collecting a certain number of training samples and determining labels, then training based on a traditional classifier or deep learning. However, both methods have their drawbacks: Method 1, using a 2D camera, can extract features from a single tobacco leaf on a two-dimensional plane, but it cannot robustly extract features from stacked tobacco leaves. It is difficult to accurately grade tobacco leaves at different heights on top of a pile of stacked tobacco leaves. Method 2 is only suitable for ideal tobacco collection environments, such as manually unfolding the tobacco leaves, placing them face up, flattening them, and positioning them at a fixed angle on a conveyor belt. In actual tobacco collection scenarios, this is prone to slow efficiency and low accuracy due to long-term repetitive actions, making it difficult to popularize.
[0003] In actual production, to avoid manual tobacco handling, automated grasping solutions (based on robotic arms adsorbing, stacking, and grading tobacco leaves) are more intelligent and efficient. However, such tobacco piles can only identify and grade the top layer of leaves, which are not necessarily completely flat and facing upwards. Therefore, when identifying and grading the top layer of leaves, the tobacco leaf labels themselves have significant uncertainty. Experts with extensive domain knowledge often assign different labels to the same sample of front, back, and folded tobacco leaves. Therefore, learning from training samples with defined labels, whether using traditional classifiers or existing conventional deep learning classification models, is difficult to apply to automated grasping solutions, resulting in low identification and grading accuracy. Summary of the Invention
[0004] The purpose of this invention is to provide a tobacco grading method based on the segmentation of the upper layer of tobacco leaves, which solves the technical problem in the prior art that it is difficult to accurately identify and grade the uppermost layer of tobacco leaves that form a tobacco pile obtained by automatic grasping scheme.
[0005] The method for grading tobacco leaves based on the segmentation of the upper layer of tobacco leaves includes the following steps:
[0006] S1. Acquire tobacco leaf images based on a 3D camera and generate training samples of tobacco leaf images. The training samples are labeled with label 1 for positive samples and label 2 for strong negative samples.
[0007] S2. Perform the top-level tobacco leaf segmentation task based on YOLOv8, and train instance segmentation based on YOLOv8, which has the best performance in the instance segmentation task.
[0008] S3. Based on the previous steps, obtain and save the top layer of tobacco leaves after removing the background from the training samples. Perform multi-label annotation on the image data of the top layer of tobacco leaves, so that each sample is associated with a set of candidate labels.
[0009] S4. Training for tobacco leaf grading based on PiCO using image data of associated label sets as samples;
[0010] S5: The 3D camera acquires images of tobacco leaves, and the top layer of tobacco leaves is segmented and identified based on the yolov8 model trained in step S2.
[0011] S6. Based on the PiCO model trained in step S4, classify the tobacco leaves using the image data of the top layer of tobacco leaves identified in step S5.
[0012] Preferably, in step S5, for multiple labels l masks that appear in the actual prediction, the depth values of the centroids of different masks are compared based on 3D structured light technology, and the mask of the top layer of tobacco leaves is selected. Based on this mask, the foreground in the original image, i.e., the image of the top layer of tobacco leaves, is extracted.
[0013] Preferably, step S5 specifically includes: the instance segmentation algorithm segments the complete tobacco leaf region, and there may be one or more masks labeled 1; when there is only one mask labeled 1, the mask is directly returned as the tobacco leaf region to be analyzed; when there are multiple masks labeled 1, the depth values of the centroid positions of the masks of different labels 1 are derived based on 3D structured light, and the 3D camera transmits the three-dimensional coordinates (x, y, z) of the position, where x and y are the centroid coordinates of the mask in the two-dimensional plane, and z is the depth value of the corresponding point acquired by the 3D camera; the mask with the smallest z value is selected and the corresponding mask is identified as the complete tobacco leaf segmented at the top layer for analysis.
[0014] Preferably, step S1 specifically includes:
[0015] S1.1 Collect tobacco leaves from different parts of the plant;
[0016] S1.2 Adjust the tobacco leaves to various states to fully simulate tobacco leaf production;
[0017] S1.3 Classify and label tobacco leaf images into two types of labels: Label 1 is for whole tobacco leaves or tobacco leaves that are slightly obscured. Here, slight obscuration means that more than 90% of the tobacco leaf area is exposed, the tobacco leaf is basically unbroken, and the length of the tobacco leaf is basically not curved; Label 2 is for non-whole tobacco leaves or various parts that are obscured.
[0018] S1.4 After labeling in the above manner, an image is labeled with the outlines of label 1 for n positive samples and the outlines of label 2 for m strong negative samples, where n≥1 and m≥0.
[0019] Preferably, in step 2, yolov8-seg is selected for instance segmentation training. Since the tobacco leaf edge and folded tobacco leaf features in the image have a certain degree of coupling, the attention mechanism CBAM is added to yolov8-seg, making it easier for the model to focus on the key information of the whole tobacco leaf.
[0020] Preferably, in step S3, the candidate label set should include not only the real labels corresponding to the samples, but also tobacco leaf labels of adjacent grades belonging to the same part of the image, and the length of each candidate label set should not exceed 3.
[0021] Preferably, in step S4, a Focus module is added before the input to the PiCO network to improve the convergence speed; label learning is performed based on PiCO, and the label representation of the training data is optimized by combining the tobacco leaf state with the grade classification standard between tobacco leaf parts.
[0022] Preferably, in step S5, the tobacco leaf image captured by the 3D camera is an image of a tobacco leaf pile formed by a robotic arm grasping or adsorbing it.
[0023] The advantages of this invention are as follows: This method is applicable to scenarios involving stacked tobacco leaves, such as tobacco leaf piles formed by robotic arms grasping or adsorbing in automated grasping schemes. This method segments the top layer of tobacco leaves and performs tobacco leaf grading, making it more automated, efficient, and accurate. Furthermore, this method uses 3D structured light technology to design a segmentation scheme for the top layer of tobacco leaves and combines it with national tobacco leaf grading standards to design a more targeted biased label learning scheme, which helps improve the accuracy of tobacco leaf grading, expands hardware design ideas, and contributes to the intelligent and efficient realization of the tobacco leaf grading system. Attached Figure Description
[0024] Figure 1 This is a flowchart illustrating the tobacco leaf grading method based on upper layer tobacco leaf segmentation according to the present invention. Detailed Implementation
[0025] The following detailed description of the embodiments, with reference to the accompanying drawings, will further illustrate the specific implementation of the present invention, in order to help those skilled in the art to have a more complete, accurate, and thorough understanding of the inventive concept and technical solutions of the present invention.
[0026] like Figure 1 As shown, the present invention provides a tobacco leaf grading method based on the segmentation of the upper layer of tobacco leaves, which includes the following steps.
[0027] S1. Acquire tobacco leaf images using a 3D camera to generate training samples of tobacco leaf images. The training samples are labeled with label 1 for positive samples and label 2 for strong negative samples.
[0028] This step specifically includes...
[0029] S1.1 Collect tobacco leaves from the bottom leaves, lower second shed, middle leaves, upper second shed, and top leaves respectively, and the final data sample for each part shall not be less than 1,000 leaves.
[0030] S1.2 Adjust the tobacco leaves to multiple states, including the front, back, and folded sides, to fully simulate tobacco leaf production.
[0031] S1.3 Classify and label the tobacco leaf images into two types of labels: Label 1 is for whole tobacco leaves or slightly occluded tobacco leaves. Here, "slightly occluded" means that more than 90% of the tobacco leaf area is exposed, the tobacco leaf is basically unbroken, and the length of the tobacco leaf is basically not curved; Label 2 is for non-whole tobacco leaves or various occluded parts. Note that small fragments of tobacco leaves in the image are not labeled. Small fragments of tobacco leaves refer to the exposed area of less than 5×5 pixels.
[0032] S1.4 After labeling in the above manner, an image is labeled with the outlines of label 1 for n positive samples and the outlines of label 2 for m strong negative samples, where n≥1 and m≥0.
[0033] Whether it's the design of a solution for rapid grading of whole frames of tobacco leaves or the design of a solution based on intelligent automatic grasping by a robotic arm, 3D cameras make up for the shortcomings of 2D cameras in terms of depth features.
[0034] S2. Perform the top-level tobacco leaf segmentation task based on YOLOv8, and train the instance segmentation based on YOLOv8, which has the best performance in the instance segmentation task.
[0035] Considering the speed and accuracy of tobacco leaf segmentation, we selected yolov8-seg, which currently has the best performance in instance segmentation tasks, for instance segmentation training. yolov8-seg is a single-stage end-to-end segmentation network that meets real-time requirements and outperforms other YOLO series networks of the same level.
[0036] Since the edges of tobacco leaves and the features of folded tobacco leaves in the image have a certain degree of coupling, the attention mechanism CBAM (Convolutional Block Attention Module) is added to yolov8-seg, making it easier for the model to focus on the key information of the whole tobacco leaf and achieving higher accuracy.
[0037] S3. Based on the previous steps, obtain and save the top layer of tobacco leaves after removing the background from the training samples. Ask experts to perform multi-label annotation on the image data of the top layer of tobacco leaves, so that each sample is associated with a set of candidate labels.
[0038] In addition to the real labels corresponding to the samples, the candidate label set should also include tobacco leaf labels of adjacent grades belonging to the same part of the image, and the length of each candidate label set should not exceed 3.
[0039] S4. Training for tobacco leaf grading is performed using image data of associated label sets as samples based on PiCO (Contrastive Label Disambiguation for Partial Label Learning).
[0040] Since PiCO can be viewed as an EM (Expectation Maximization) process, its drawback is a slow convergence speed. Therefore, a Focus module is added before the network input to improve the convergence speed while ensuring approximate accuracy.
[0041] This step provides a candidate label set based on national tobacco leaf grading standards and performs label learning based on PiCO. The actual state of tobacco leaves in production environments is complex, exhibiting characteristics of label ambiguity / fuzziness. Combining tobacco leaf state with grading standards between leaf parts optimizes the label representation of the training data. The PiCO framework utilizes contrastive learning techniques to effectively solve the label disambiguation problem in partial label learning, making it suitable for tobacco leaf grading tasks.
[0042] S5. The 3D camera acquires images of the piled tobacco leaves picked up by the robotic arm in the automatic grasping scheme, and the top layer of tobacco leaves is segmented and identified based on the yolov8 model trained in step S2.
[0043] For multiple label 1 masks (contour regions) that appear in the actual prediction, the depth values of the centroids of different masks are compared based on 3D structured light technology, and the mask of the top layer of tobacco leaves is selected. Based on this mask, the foreground in the original image is extracted, that is, the image of the top layer of tobacco leaves.
[0044] Specifically: The instance segmentation algorithm segments the complete tobacco leaf region, and there may be one or more masks (contour regions) labeled 1. When there is only one mask labeled 1, the mask is directly returned as the tobacco leaf region to be analyzed. When there are multiple masks labeled 1, the depth values of the centroid positions of the masks of different labels 1 need to be derived based on 3D structured light. The 3D camera transmits the three-dimensional coordinates (x, y, z) of this position, where x and y are the centroid coordinates of the mask in the two-dimensional plane, and z is the depth value of the corresponding point acquired by the 3D camera. The mask with the smallest z value is selected and identified as the topmost segmented complete tobacco leaf for analysis, which can effectively improve the accuracy of tobacco leaf grading.
[0045] 3D structured light technology can achieve very high spatial resolution and very high accuracy at close range, which is more conducive to identifying the top layer of tobacco leaves and plays a vital role in tobacco leaf grading.
[0046] S6. Based on the PiCO model trained in step S4, classify the tobacco leaves using the image data of the top layer of tobacco leaves identified in step S5.
[0047] This method is applicable to scenarios involving stacked tobacco leaves, segmenting the top layer of leaves and grading them in a more automated, efficient, and accurate manner. Furthermore, this method utilizes 3D structured light technology to design a segmentation scheme for the top layer of tobacco leaves and incorporates a more targeted partial label learning scheme based on national tobacco grading standards. This helps improve the accuracy of tobacco grading, expands hardware design ideas, and contributes to the intelligent and efficient implementation of tobacco grading systems.
[0048] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.
Claims
1. A method for grading tobacco leaves based on the segmentation of the upper layer of tobacco leaves, characterized in that: Includes the following steps: S1. Acquire tobacco leaf images based on a 3D camera and generate training samples of tobacco leaf images. The training samples are labeled with label 1 for positive samples and label 2 for strong negative samples. S2. Perform the top-level tobacco leaf segmentation task based on YOLOv8, and train instance segmentation based on YOLOv8, which has the best performance in the instance segmentation task. S3. Based on steps S1 and S2, obtain and save the top layer of tobacco leaves after background removal from the training samples, and perform multi-label annotation on the image data of the top layer of tobacco leaves, so that each sample is associated with a set of candidate labels. S4. Based on PiCO, use image data of associated label sets as samples to train tobacco leaf grading; S5: The 3D camera acquires images of tobacco leaves, and the top layer of tobacco leaves is segmented and identified based on the yolov8 model trained in step S2. S6. Based on the PiCO model trained in step S4, tobacco leaves are graded using the image data of the top layer of tobacco leaves identified in step S5. Step S1 specifically includes: S1.1 Collect tobacco leaves from different parts of the plant; S1.2 Adjust the tobacco leaves to various states to fully simulate tobacco leaf production; S1.3 Classify and label tobacco leaf images into two types of labels: Label 1 is for whole tobacco leaves or tobacco leaves that are slightly obscured. Here, slight obscuration means that more than 90% of the tobacco leaf area is exposed, the tobacco leaf is basically unbroken, and the length of the tobacco leaf is basically not curved; Label 2 is for non-whole tobacco leaves or various parts that are obscured. S1.4 After labeling in the above manner, an image is labeled with the outlines of label 1 for n positive samples and the outlines of label 2 for m strong negative samples, where n≥1, m≥0, and are all integers.
2. The tobacco grading method based on upper layer tobacco leaf segmentation according to claim 1, characterized in that: In step S5, for multiple labels 1 masks that appear in the actual prediction, the depth values of the centroids of different masks are compared based on 3D structured light technology, and the mask of the top layer of tobacco leaves is selected. Based on this mask, the foreground in the original image is extracted, that is, the image of the top layer of tobacco leaves.
3. The tobacco leaf grading method based on upper layer tobacco leaf segmentation according to claim 2, characterized in that: Step S5 specifically includes: the instance segmentation algorithm segments the complete tobacco leaf region, and there may be one or more masks labeled 1; when there is only one mask labeled 1, the mask is directly returned as the tobacco leaf region to be analyzed; when there are multiple masks labeled 1, the depth values of the centroid positions of the masks of different labels 1 are derived based on 3D structured light, and the 3D camera transmits the three-dimensional coordinates (x, y, z) of the position, where x and y are the centroid coordinates of the mask in the two-dimensional plane, and z is the depth value of the corresponding point acquired by the 3D camera; the mask with the smallest z value is identified as the complete tobacco leaf segmented at the top layer, and the mask with the smallest z value is selected for analysis.
4. The tobacco leaf grading method based on upper layer tobacco leaf segmentation according to claim 1, characterized in that: In step 2, yolov8-seg is selected for instance segmentation training. Since the tobacco leaf edge and folded tobacco leaf features in the image have a certain degree of coupling, the attention mechanism CBAM is added to yolov8-seg to make the model more likely to focus on the key information of the whole tobacco leaf.
5. The tobacco leaf grading method based on upper layer tobacco leaf segmentation according to claim 1, characterized in that: In step S3, in addition to the real labels corresponding to the samples, the candidate label set should also include tobacco leaf labels of adjacent grades belonging to the same part of the image, and the length of each candidate label set shall not exceed 3.
6. The tobacco leaf grading method based on upper layer tobacco leaf segmentation according to claim 1, characterized in that: In step S4, a Focus module is added before the input to the PiCO network to improve the convergence speed; label learning is performed based on PiCO, and the label representation of the training data is optimized by combining the tobacco leaf state with the grade classification standard between tobacco leaf parts.
7. The tobacco leaf grading method based on upper layer tobacco leaf segmentation according to claim 1, characterized in that: In step S5, the tobacco leaf images captured by the 3D camera are images of tobacco leaf piles formed by a robotic arm grasping or adsorbing them.