A vision-assisted automatic tobacco feeding device and method

Abstract

An automatic tobacco leaf feeding device based on vision assistance includes a positioning column, a horizontal moving device, a vertical moving device, a controller, a three-axis robotic arm, and a robotic gripper. Its key feature is that: binocular cameras are symmetrically arranged on both sides of the horizontal support of the horizontal moving device, with the distance between the two horizontal supports being the width of two tobacco baskets; a laser rangefinder is located in the middle of the vertical support of the vertical moving device; the binocular cameras and the laser rangefinder are connected to the controller via data cables; and the robotic gripper can rotate in a horizontal position. This invention improves feeding efficiency and accuracy through the visual assistance of the binocular cameras and the laser rangefinder, and can accurately grasp tobacco leaves when there is little remaining material. It can also grasp two tobacco baskets simultaneously, allowing both baskets to enter and exit at the same time. The most significant feature of this invention is that it achieves uninterrupted automatic feeding of tobacco leaves into two baskets, reducing the labor intensity of workers and improving work efficiency.

Description

Technical Field

[0001] This invention belongs to the field of tobacco processing and mainly relates to a vision-assisted automatic tobacco feeding device and method. Background Technology

[0002] Feeding raw tobacco leaves is a crucial step in tobacco production. Currently, manual feeding is generally used, with workers sequentially removing tobacco leaves from the baskets, spreading them out, and placing them on a leaf-laying platform. This method is labor-intensive, lacks precise control over the feeding flow, and suffers from low efficiency due to variations in worker skill levels. Existing robotic feeding devices, lacking effective visual assistance, suffer from low feeding efficiency, long working times, and inability to accurately pick up remaining tobacco leaves when there is little material left, directly impacting the stability of feeding in subsequent processes. Chinese patent CN214802252U discloses an automatic material feeding device based on a gantry robotic arm. This device uses a robotic arm to pick up tobacco leaves from a single basket, but the gripper only picks up tobacco leaves at equal intervals in a step-by-step manner, resulting in low picking efficiency and accuracy, and the problem of missing remaining material.

[0003] Visual positioning technology refers to the technology of acquiring environmental image information through binocular or multi-view vision devices and using stereo vision positioning algorithms to analyze and reconstruct the three-dimensional information of the scene to obtain the position information of target objects. This technology has the advantages of low cost, high accuracy, and high flexibility. A laser rangefinder is an instrument that uses lasers to measure the distance to a target. During operation, the laser rangefinder emits a laser and calculates the time from emission to retraction to measure the distance. It includes one-dimensional laser rangefinders for distance measurement and positioning; two-dimensional laser rangefinders for contour measurement and positioning, area monitoring, etc.; and three-dimensional laser rangefinders for three-dimensional contour measurement, three-dimensional spatial positioning, etc. Currently, both technologies have applications in industrial measurement and control, construction, and drones, but there are no reports on the integrated and innovative application of these two technologies. Summary of the Invention

[0004] To overcome the problems existing in the prior art, the present invention provides a vision-assisted automatic tobacco feeding device and method. This invention further addresses practical application problems based on the aforementioned device, primarily through vision assistance to achieve uninterrupted automatic feeding of tobacco leaves into two baskets within the tobacco crate. It allows for rapid feeding when there is a large amount of remaining material and precise grasping when there is a small amount, thus improving grasping efficiency and avoiding the safety hazard of the grippers grabbing the bottom of the tobacco crate when there is little remaining material.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] An automatic tobacco feeding device based on vision assistance includes a positioning column, a horizontal moving device and a horizontal support, a vertical moving device and a vertical support, a controller, a three-axis robotic arm, and a mechanical gripper located at the lower end of the three-axis robotic arm. This device is an improvement on the main structure of existing inventions, mainly by adding a vision assistance system and corresponding control, and changing the single tobacco basket gripping to a double tobacco basket parallel arrangement. Specifically, binocular cameras are symmetrically arranged on both sides of the horizontal support, and the distance between the two horizontal supports is the width to accommodate two tobacco baskets. A laser rangefinder is located at the lower end of the middle of the vertical support. The binocular cameras and the laser rangefinder are connected to the controller via a data cable. The mechanical gripper is controlled to rotate in a horizontal position by a rotary drive motor. With the vision assistance of the binocular cameras and the laser rangefinder, the feeding efficiency and accuracy are improved, and it can accurately grasp when there is little remaining material. It can also grasp two tobacco baskets simultaneously, allowing both baskets to enter and exit at the same time.

[0007] A vision-assisted automatic tobacco feeding method, utilizing the aforementioned device, mainly includes the following steps:

[0008] 1) Perform initial calibration of the laser rangefinder and binocular camera, set the scanning range of the laser rangefinder and obtain the unloaded scanning height, and correct the original image of the binocular camera;

[0009] 2) The tobacco baskets filled with tobacco leaves after the initial selection are sent to a fixed position, and the vision assistance and robotic arm gripping device is activated;

[0010] 3) The binocular camera starts shooting and transmits the data to the controller. The controller analyzes and determines whether there are tobacco leaves in the tobacco basket and the initial height of the tobacco leaves, etc.

[0011] 4) When there is a lot of leftover material, the binocular camera switches from working mode to silent mode, the laser rangefinder starts scanning, measures the real-time height information of the tobacco leaves in the tobacco basket and transmits it to the controller. When there is a lot of leftover material, the controller controls the descent height and running speed of the mechanical gripper, and moves its position in sequence to grab the tobacco leaves and release them onto the conveyor belt.

[0012] 5) When there is little remaining material, the laser instrument switches from working to stopped, the binocular camera starts to take pictures and transmit data, the mechanical gripper runs at a slower speed, and the controller analyzes the camera data and controls the mechanical gripper to accurately grab the tail material based on the location distribution information of the remaining tobacco leaves.

[0013] Furthermore, the automatic tobacco feeding device mainly consists of a three-way moving device, a three-axis robotic arm, a mechanical gripper, a controller, a binocular camera, and a laser rangefinder. The gripper on the mechanical gripper can reach any three-dimensional point within the material feeding area to perform a gripping and releasing action.

[0014] Furthermore, before using the binocular camera, initial calibration using a checkerboard pattern is required. First, the left camera is used to capture the checkerboard pattern to obtain the original image, and the intrinsic parameter matrix K1 and distortion vector D1 are calibrated. The same operation is repeated for the right camera to obtain the intrinsic parameter matrix K2 and distortion vector D2. Using the four parameter matrices obtained, the Stereo Calibrate function is used to calculate the rotation matrix R and translation vector T of the left and right cameras. Finally, the original image is corrected using the obtained series of matrices so that the two corrected images are located on the same plane and are parallel to each other. Finally, pixel matching is performed, and the depth of each pixel is calculated based on the matching results to obtain a depth map.

[0015] Furthermore, the laser rangefinder needs to be initially calibrated before use. First, the scanning angle range is calibrated, and the start and end angles are set to ensure that they cover at least the outermost edge of each of the two tobacco baskets. Second, the bottom surface of the tobacco basket when it is unloaded is used as the zero point of the laser scanning height.

[0016] Furthermore, the visual assistance system operates as follows: after feeding begins, the binocular camera works first and transmits data to the controller. The controller analyzes and determines whether there are tobacco leaves in the tobacco basket and their initial height. If there is a large amount of remaining material, the binocular camera goes into silent mode, and the laser rangefinder begins scanning, measuring the real-time height of the tobacco leaves in the basket and transmitting it to the controller. The controller then controls the mechanical gripper to move sequentially and grab the tobacco leaves. When there is less remaining material, the laser rangefinder goes into silent mode, the binocular camera intervenes, and the controller analyzes the images it captures. Based on the depth and position information, the controller controls the mechanical gripper to precisely grab the remaining tobacco leaves at the bottom of the basket.

[0017] Furthermore, the controller determines the presence of tobacco leaves based on images captured by the binocular camera by pre-capturing images of empty tobacco baskets during the calibration phase and transmitting them to the system controller. Upon commencement of feeding, the binocular camera first captures an image and transmits the data to the controller. The controller's embedded image processing module preprocesses the images, employing image filtering to remove noise interference and image segmentation and edge detection for feature extraction. The image processing module uses the SIFT algorithm to perform feature matching between empty and normal operation photos. Features are obtained by calculating feature points and their related scale and orientation descriptors in the two images, and image feature point matching is performed. A low feature point matching ratio indicates the presence of tobacco leaves in the basket. Specifically, median filtering is used for image filtering; local binarization is used for image segmentation; and the Canny operator is used to analyze and calculate edge values ​​for image edge detection.

[0018] Furthermore, the controller calculates the initial height of the tobacco leaf based on the images captured by the binocular camera by calibrating the binocular stereo vision camera as described above to eliminate camera distortion, and then projecting the two cameras onto the same plane through correction. Thus, the distance d between the points projected onto the left and right imaging planes of the same point in the same coordinate system is obtained. The distance T between the two cameras (i.e., the baseline) can be obtained through actual measurement. The focal length f is a known parameter. Finally, the distance z from the feature point to the two cameras is calculated according to the principle of geometric triangles. The calculation formula is d / T = (zf) / z. Through equation transformation, the vertical distance z from the feature point to the baseline can be obtained. The difference between this vertical distance and the vertical height of the camera is the actual height of the feature point.

[0019] Furthermore, the controller calculates the position information of the remaining tobacco leaves based on the images captured by the binocular camera. The method involves inputting a pre-processed image, and the controller's embedded position calculation module uses the YOLOv5 target detection algorithm to train a localization model on the photo to obtain the accurate position of the tobacco leaves in the basket. The implementation principle is as follows: assuming an image is divided into S×S grids, if the center of the detected target falls within a grid, the specific position of the detected target needs to be estimated. Firstly, the planar position is determined according to the plane center of the detected target within that grid; secondly, the height position is determined according to the image depth information.

[0020] Furthermore, the controller calculates the height of the tobacco leaves in the basket based on the laser scanning data by fixing the laser range sensor in the center of the bottom surface of the longitudinal support and emitting laser at a certain frequency. During the movement of the robotic arm, it scans and acquires the feature point information of several material cross sections, that is, the distance between the laser light source and a point on the material surface at each angular interval from the starting angle to the ending angle. The projection of the feature point in the height direction is calculated by trigonometric method. The difference between this projection and the empty height measured during the initial calibration is used to obtain the actual height value of each feature point.

[0021] Furthermore, the switching threshold between the binocular camera and the laser is determined based on the height of the tobacco leaves. It needs to be preset according to the actual situation before feeding. Assuming the height of the tobacco basket is h and the switching ratio of the visual aid device is p, the switching threshold w = h * p. During normal feeding, when the average height of the tobacco leaves in the basket is greater than w, it is considered that there is a lot of material left, the laser switches to working mode, and the binocular camera is in silent mode; when the average height of the tobacco leaves in the basket is less than or equal to w, it is considered that there is little material left, the binocular camera switches to working mode, and the laser switches to silent mode.

[0022] Furthermore, the gripping speed of the mechanical gripper is not constant. When the laser rangefinder is working alone, there is more residual material, and the mechanical gripper grips the tobacco leaves at a faster speed to improve the working efficiency of the device. After the binocular camera is involved, there is less residual material, and in order to accurately grip the remaining tobacco leaves and prevent the gripper from pinching the bottom of the tobacco basket and damaging the device, the mechanical gripper moves at a slower speed.

[0023] Furthermore, the gripper of the mechanical gripper picks up a certain weight of tobacco leaves from the tobacco basket and places them on the conveyor belt. A weighing sensor is installed between the end of the three-axis robotic arm and the mechanical gripper. The controller controls the feeding speed of the mechanical gripper based on parameters such as the real-time weight signal and the target feeding flow rate, so as to achieve stable control of the feeding flow rate.

[0024] Furthermore, the device operates by continuously feeding material into two baskets, with two feeding modes: indirect feeding (total quantity) and continuous feeding (feeding based on flow rate).

[0025] In summary, the beneficial effects of the above-mentioned technical solution of the present invention are as follows: It achieves uninterrupted automatic feeding of tobacco leaves into two baskets, reducing the labor intensity of workers and improving work efficiency. Simultaneously, the application of a vision-assisted system (including a binocular camera and a laser rangefinder) enables rapid feeding when there is a large amount of remaining material and precise grasping when there is a small amount, thus improving the device's grasping efficiency and avoiding the safety hazard of the grippers grabbing the bottom of the tobacco basket when there is little remaining material. Furthermore, the controller calculates the set weight for the next feeding in real time, thereby achieving stable control of the feeding flow rate.

[0026] The main difference between this invention and prior art lies in the significant change in application scenario. This invention is geared towards single-line, box-by-box feeding. However, in actual cigarette factory applications, it has been found that some factories require parallel, dual-line, box-by-box feeding. In such cases, it is impossible to set up feeding machines on two separate lines. Firstly, the spacing between the lines is too small, resulting in insufficient space. Secondly, it would increase equipment costs. Thirdly, the asynchronous operation of the two feeding machines would affect feeding efficiency. Therefore, the space in the cigarette basket is widened to accommodate two cigarette boxes simultaneously for feeding. Furthermore, a prominent problem with the previous invention is that the gripper only feeds the tobacco leaves at equal intervals in a step-by-step manner, resulting in low feeding efficiency and accuracy, and the problem of not being able to pick up leftover material. This invention adds visual assistance to improve feeding efficiency and accuracy, and solves the problem of picking up leftover material. Attached Figure Description

[0027] Figure 1 This is a front view of the automatic tobacco feeding device.

[0028] Figure 2 This is a side view of the structure of an automatic tobacco feeding device.

[0029] Figure 3 This is a schematic diagram of the mechanical gripper.

[0030] Figure 4 This is a control flowchart for an automatic tobacco feeding device.

[0031] In the diagram: 1. Positioning column; 2. Mechanical gripper; 21. Gripper; 22. Rotary shaft; 23. Rotary drive motor; 3. Lateral movement device; 31. Lateral base; 32. First motor; 33. Lateral support; 4. Longitudinal movement device; 41. Longitudinal support; 42. Longitudinal base; 43. Second motor; 5. Vertical movement device; 51. Vertical base; 52. Vertical support; 53. Third motor; 6. Tobacco basket; 7. Binocular camera; 8. Laser rangefinder; 9. Controller; 10. Conveyor belt. Detailed Implementation

[0032] The features and principles of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] like Figure 1-3 As shown: The automatic tobacco feeding device based on vision assistance of the present invention includes a positioning column 1, a horizontal moving device 3 and a horizontal support 33, a vertical moving device 4 and a vertical support 41, a vertical moving device 5 and a vertical support 52, a controller 9, a three-axis robotic arm, and a mechanical gripper 2 set at the lower end of the three-axis robotic arm. A binocular camera 7 is symmetrically arranged on both sides of the horizontal support 33, and the distance between the two horizontal supports is the width to accommodate two tobacco baskets 6. A laser rangefinder 8 is located at the lower end of the middle of the vertical support 41. The binocular camera 7 and the laser rangefinder 8 are connected to the controller 9 via a data cable. The mechanical gripper 2 is controlled to rotate in a horizontal position by a rotary drive motor 23. With the visual assistance of the binocular camera 7 and the laser rangefinder 8, the feeding efficiency and accuracy are improved, and it can accurately grasp when there is little remaining material. It can also grasp two tobacco baskets 6 simultaneously, allowing both baskets to enter and exit at the same time.

[0034] A vision-assisted automatic tobacco feeding method, utilizing the aforementioned device, mainly includes the following steps (see...). Figure 4 ):

[0035] 1) Perform initial calibration on the laser rangefinder 8 and the binocular camera 7 (stereo vision depth camera), set the scanning range of the laser rangefinder and obtain the idle scanning height, and correct the original image of the binocular camera;

[0036] 2) The tobacco basket 6, filled with tobacco leaves after the initial selection, is sent to a fixed position, and the vision assistance and robotic arm gripping device is activated;

[0037] 3) The binocular camera 7 starts shooting and transmits the data to the controller 9. The controller analyzes and determines whether there are tobacco leaves in the tobacco basket and the initial height of the tobacco leaves, etc.

[0038] 4) When there is a lot of leftover material, the binocular camera 7 switches from working state to silent state, and the laser rangefinder 8 starts scanning to measure the real-time height information of the tobacco leaves in the tobacco basket and transmit it to the controller. When there is a lot of leftover material, the controller controls the descent height and running speed of the mechanical gripper, and moves its position in sequence to grab the tobacco leaves and release them onto the conveyor belt.

[0039] 5) When there is little remaining material, the laser rangefinder 8 switches from working to stopped, the binocular camera 7 starts to take pictures and transmit data, the mechanical gripper 2 reduces its running speed, and the controller 9 analyzes the camera data and controls the mechanical gripper 2 to accurately grab the tail material according to the location distribution information of the remaining tobacco leaves.

[0040] The present invention is described in more detail below:

[0041] like Figure 1 As shown, the invention mainly includes a positioning column 1, a three-axis robotic arm, a mechanical gripper 2, a controller 9, a horizontal moving device 3, a vertical moving device 4, a vertical moving device 5, a tobacco basket 6, a binocular camera 7, and a laser rangefinder 8.

[0042] Four positioning posts 1 support the three-axis robotic arm and gripper 2. The positioning posts 1 can be fixed to the ground with anchor bolts to improve stability. The distance between the two horizontal supports 33 is equal to the width of the two tobacco baskets 6. The three-axis robotic arm is positioned above the longitudinal support 41, and the mechanical gripper 2 is positioned at the end of the three-axis robotic arm. The three-axis robotic arm is used to drive the mechanical gripper 2 to move horizontally between the tobacco baskets 6 and the belt 10, as well as to move vertically. The gripper portion of the mechanical gripper 2 can rotate along the rotation axis 22 on the horizontal plane to adjust its posture. Its rotation is driven by a rotary drive motor 23 for precise gripping and releasing of tobacco leaves.

[0043] The binocular camera 7 is mounted on the horizontal support 33, located above the left and right sides of the tobacco basket 6; the laser rangefinder 8 is mounted at the midpoint of the lower end face of the vertical support 41, and scans the tobacco basket 6 below during the longitudinal translation process to obtain surface feature information of the tobacco leaf accumulation layer.

[0044] The three-axis robotic arm includes a horizontal moving device 3, a vertical moving device 4, and a vertical moving device 5. The horizontal moving device 3 includes a horizontal base 31 slidably connected to a horizontal support 33 and a first motor 32 mounted on the horizontal base 31. The vertical moving device 4 includes a vertical support 41 mounted between two horizontal bases 31, a vertical base 42 slidably connected to the vertical support 41, and a second motor 43 mounted on the vertical base 42. The vertical moving device 5 includes a vertical base 51 mounted on the vertical base 42, a vertical support 52 slidably connected to the vertical base 51, and a third motor 53 mounted on the vertical base 51. A mechanical gripper 2 is mounted at the bottom of the vertical support 52.

[0045] Photoelectric sensors are installed at both ends of the horizontal support 33, the longitudinal support 41, and the vertical support 52 to detect the distances between the ends of the longitudinal support 41 and the horizontal support 33, the distances between the ends of the vertical base 51 and the longitudinal support 41, and the distances between the ends of the vertical support 52 and the vertical base 51. The photoelectric sensors are connected to the controller to provide data for the controller to determine the position of the gripper 2.

[0046] In use, the laser rangefinder 8 and binocular camera 7 are first initially calibrated. Then, two tobacco baskets 6 filled with tobacco leaves are placed side by side on the ground between two horizontal supports 33. In this embodiment, the height of the tobacco basket is 1.5 meters, the working switching ratio of the visual aid device is 20%, and the working switching threshold w is 0.3 meters. During normal feeding, when the average height of the tobacco leaves in the basket is greater than 0.3 meters, the laser rangefinder is in working mode, and the binocular camera is in silent mode; when the average height of the tobacco leaves in the basket is less than or equal to 0.3 meters, the binocular camera is in working mode, and the laser rangefinder is in silent mode. The binocular camera 7 starts working and transmits data to the controller. After the controller determines that the conditions of having tobacco leaves between the two horizontal supports 33 and the initial height of the tobacco leaves being greater than 0.3 meters are met, the binocular camera 7 switches to silent mode, the laser rangefinder 8 starts, scans the tobacco basket 6 below in real time, and transmits the data to the controller 9. The controller 9 controls the three-axis robotic arm to drive the mechanical gripper 2 to move at a relatively fast speed above the tobacco basket 6, then moves vertically downwards. After contacting the tobacco leaves, the controller 9 controls the mechanical gripper 2 to engage and grab the tobacco leaves. Then, the mechanical gripper 2 moves upwards and then horizontally above the conveyor belt 10 to release the tobacco leaves. The grabbing and moving trajectory of the mechanical gripper 2 on the two tobacco baskets 6 is S-shaped on the plane (grabbing tobacco leaves from both baskets simultaneously in an S-shape results in a shorter total moving distance for the robotic arm, higher grabbing efficiency, and allows both baskets to enter and exit simultaneously). This cycle repeats until there is less remaining material. At this point, the operating speed of the three-axis robotic arm decreases, the laser rangefinder 8 switches to a silent state, and the binocular camera 7 switches to a working state. The mechanical gripper 2 then grabs the tobacco leaves point by point according to the position information given by the position calculation module, completing the precise grabbing of the remaining material.

[0047] When there is a lot of residual material, the mechanical gripper 2 cyclically grabs the tobacco leaves in an S-shaped trajectory. When there is less residual material, the mechanical gripper 2 no longer moves in an S-shape, but instead completes the precise grabbing of the remaining tail material in a fixed-point grabbing manner according to the grabbing position information given by the controller.

[0048] This invention enables continuous automatic feeding of tobacco leaves into two baskets of tobacco, reducing the labor intensity of workers and improving work efficiency. Simultaneously, it utilizes a vision-assisted system (including a binocular camera and laser rangefinder) for rapid feeding when there is a large amount of material remaining and precise grasping when there is a small amount, thus improving the device's grasping efficiency and avoiding the safety hazard of the grippers grabbing the bottom of the tobacco basket when there is little material remaining. Furthermore, the controller can calculate the set weight for the next feeding in real time, thereby achieving stable control of the feeding flow rate.

Claims

1. A vision-assisted automatic tobacco feeding method, characterized in that: This system operates based on an automatic tobacco feeding device, which includes a positioning column, a horizontal moving device and a horizontal support, a vertical moving device and a vertical support, a controller, a three-axis robotic arm, and a mechanical gripper located at the lower end of the three-axis robotic arm. Its key feature is that binocular cameras are symmetrically arranged on both sides of the horizontal support, with the distance between the two horizontal supports being the width to accommodate two tobacco baskets. A laser rangefinder is located at the lower end of the middle of the vertical support. The binocular cameras and the laser rangefinder are connected to the controller via a data cable. The mechanical gripper is controlled to rotate in a horizontal position by a rotary drive motor. The visual assistance of the binocular cameras and the laser rangefinder improves feeding efficiency and accuracy, enabling precise gripping when there is little remaining material, and simultaneously gripping both tobacco baskets to ensure simultaneous entry and exit. The system includes the following steps: 1) Perform initial calibration of the laser rangefinder and binocular camera, set the scanning range of the laser rangefinder and obtain the unloaded scanning height, and correct the original image of the binocular camera; 2) The tobacco baskets filled with tobacco leaves after the initial selection are sent to a fixed position, and the vision assistance and robotic arm gripping device is activated; 3) The binocular camera starts shooting and transmits the data to the controller. The controller analyzes and determines whether there are tobacco leaves in the tobacco basket and the initial height information of the tobacco leaves; 4) When there is a lot of leftover material, the binocular camera switches from working mode to silent mode, the laser rangefinder starts scanning, measures the real-time height information of the tobacco leaves in the tobacco basket and transmits it to the controller. When there is a lot of leftover material, the controller controls the descent height and running speed of the mechanical gripper, and moves its position in sequence to grab the tobacco leaves and release them onto the conveyor belt. 5) When there is little remaining material, the laser instrument switches from working to stopped, the binocular camera starts to take pictures and transmit data, the mechanical gripper runs at a slower speed, and the controller analyzes the camera data and controls the mechanical gripper to accurately grab the tail material based on the location distribution information of the remaining tobacco leaves. The switching threshold between the binocular camera and the laser is determined based on the height of the tobacco leaves. It needs to be preset according to the actual situation before feeding. Assuming the height of the tobacco basket is h and the switching ratio of the vision aid is p, then the switching threshold w = h * p. During normal feeding, when the average height of the tobacco leaves in the basket is greater than w, it is considered that there is a lot of material left, the laser switches to working mode, and the binocular camera is in silent mode; when the average height of the tobacco leaves in the basket is less than or equal to w, it is considered that there is little material left, the binocular camera switches to working mode, and the laser switches to silent mode.

2. The automatic tobacco feeding method based on vision assistance according to claim 1, characterized in that: Before using a binocular camera, initial calibration using a checkerboard pattern is required. First, the left camera is used to capture the checkerboard pattern to obtain the original image, and the intrinsic parameter matrix K1 and distortion vector D1 are obtained through calibration. The same operation is repeated for the right camera to obtain the intrinsic parameter matrix K2 and distortion vector D2. Using the four parameter matrices obtained, the Stereo Calibrate function is used to calculate the rotation matrix R and translation vector T of the left and right cameras. Finally, the original image is corrected using a series of matrices to make the two corrected images lie on the same plane and are parallel to each other. Finally, pixel matching is performed, and the depth of each pixel is calculated based on the matching results to obtain a depth map.

3. The automatic tobacco feeding method based on vision assistance according to claim 1, characterized in that: Before using a laser rangefinder, initial calibration is required. First, the scanning angle range is calibrated, and the start and end angles must be set to ensure that at least the outermost edge of each of the two tobacco baskets is covered. Second, the bottom surface of the tobacco basket when it is empty is used as the zero point of the laser scanning height.

4. The automatic tobacco feeding method based on vision assistance according to claim 1, characterized in that: The visual assistance system operates as follows: after feeding begins, the binocular camera works first and transmits data to the controller. The controller analyzes and determines whether there are tobacco leaves in the tobacco basket and their initial height. If there is a lot of remaining material, the binocular camera goes into silent mode, and the laser rangefinder starts scanning, measuring the real-time height of the tobacco leaves in the basket and transmitting it to the controller. The controller then controls the mechanical gripper to move sequentially and grab the tobacco leaves. When there is less remaining material, the laser rangefinder goes into silent mode, the binocular camera intervenes, and the controller analyzes the images it captures. Based on the depth and position information, the controller controls the mechanical gripper to precisely grab the remaining tobacco leaves at the bottom of the basket.

5. The automatic tobacco feeding method based on vision assistance according to claim 2, characterized in that: The controller determines the presence of tobacco leaves based on images captured by the binocular camera as follows: During the calibration phase, the binocular camera pre-captures images of empty tobacco baskets and transmits them to the system controller. Upon commencement of feeding, the binocular camera first captures an image and transmits the data to the controller. The controller's embedded image processing module preprocesses the images, employing image filtering to remove noise interference. Image segmentation and edge detection are used to extract features from the images. The image processing module uses the SIFT algorithm to perform feature matching between empty and normal operating photos. By calculating feature points and their related scale and orientation descriptors in the two images, features are obtained and image feature point matching is performed. A low feature point matching ratio indicates the presence of tobacco leaves in the tobacco basket. Median filtering is used for image filtering; local binarization is used for image segmentation; and the Canny operator is used to analyze and calculate edge values ​​for image edge detection.

6. The automatic tobacco feeding method based on vision assistance according to claim 5, characterized in that: The controller calculates the initial height of the tobacco leaf based on the images captured by the binocular cameras by projecting the two cameras onto the same plane through calibration. This yields the distance d between the points projected onto the left and right imaging planes of the same point in the same coordinate system. The distance T between the two cameras is obtained through actual measurement, and the line connecting the two cameras is the baseline. The focal length f is a known parameter. Finally, the distance z between the feature point and the baseline is calculated based on the principle of geometric triangles. The calculation formula is d / T = (zf) / z. The vertical distance z from the feature point to the baseline is obtained through equation transformation. The difference between this vertical distance and the vertical height of the camera is used to obtain the actual height of the feature point.

7. The automatic tobacco feeding method based on vision assistance according to claim 1, characterized in that: The controller calculates the position information of the remaining material based on the image captured by the binocular camera. The input is a pre-processed image. The controller's embedded position calculation module uses the YOLOv5 target detection algorithm to train a positioning model on the photo to obtain the accurate position of the tobacco leaves in the basket. The principle is as follows: Assuming an image is divided into S×S grids, if the center of the detected target falls in the grid, the specific position of the detected target needs to be estimated. One is the planar position, which is determined according to the plane center of the detected target in the grid. The other is the height position, which is determined according to the image depth information.

8. The automatic tobacco feeding method based on vision assistance according to claim 1, characterized in that: The controller calculates the height of the tobacco leaves in the basket based on the laser scanning data by fixing the laser range sensor in the center of the bottom surface of the longitudinal support and emitting laser at a certain frequency. During the movement of the robotic arm, it scans and acquires the feature point information of several material cross sections, that is, the distance between the laser light source and a point on the material surface at each angular interval from the starting angle to the ending angle. The projection of the feature point in the height direction is calculated according to the trigonometric method. The difference between this projection and the empty height measured during the initial calibration is used to obtain the actual height value of each feature point.

Images