Visual guidance redried tobacco leaf automatic quantitative feeding system and implementation method

Abstract

According to the vision-guided redried tobacco leaf automatic quantitative feeding system and the implementation method, the tobacco leaf placement posture is estimated through the three-dimensional measurement technology to obtain three-dimensional information of a tobacco leaf scene, so that the accurate grabbing posture and the opening and closing size of the claw are calculated, the mechanical arm and the gripper are cooperatively controlled to conduct grabbing operation, and the grabbing precision is improved. And the tobacco leaves are quantified through a weight sensor. Through the integrated robot technology and the advanced computer vision system, automatic grabbing and placing operation on tobacco leaves on the operation platform is achieved, so that the tobacco leaf feeding and grabbing efficiency and accuracy are improved, the grabbing effectiveness and stability are ensured, and the complexity of manual operation is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction and target detection, and in particular to a vision-guided automatic quantitative feeding system for redried tobacco leaves and an implementation method thereof. Background Art

[0002] Threshing and redrying, as a key link in the tobacco industry chain, plays a pivotal role, directly impacting the overall quality and market competitiveness of the product. In recent years, as consumers' expectations for cigarette quality have increased, tobacco industry companies have placed increasing emphasis on product quality control, particularly by setting higher standards and requirements for the homogenization of redrying tobacco leaves.

[0003] In the traditional loading and spreading process, workers manually place tobacco leaves of varying grades and weights onto the production line's conveyor belt according to the recipe. After loading and spreading, the leaves are transported via conveyor belt to the next process. Currently, most tobacco redrying plants rely on manual labor for measuring, transferring, placing, and spreading the leaves. This process not only results in low production efficiency but also lacks precision, significantly compromising recipe accuracy. Furthermore, manual operations are labor-intensive and pose a poor working environment. Workers work in an environment exposed to high dust and dense smoke, severely impacting their physical and mental health and increasing the risk of occupational diseases and occupational health risks.

[0004] In recent years, with the deep integration of intelligent manufacturing technologies in tobacco processing, automated tobacco leaf loading methods based on sensor technology and robotic arm systems have gradually become a research hotspot. Through the coordinated optimization of multi-sensor fusion perception and robotic arm control, this technology can identify and estimate tobacco leaf stacking scenarios, thereby completing high-speed automated grasping and loading of incoming tobacco leaves, providing an intelligent upgrade path for traditional manual operation.

[0005] However, existing methods are often limited to single ultrasonic sensing or simple threshold-based region segmentation methods to obtain grasping features. These methods lack the necessary sensory dimensionality and information richness, making grasping location calculations inaccurate and inadequate for complex stacking scenarios. Perception errors often arise from actual working conditions, such as occlusion between tobacco leaves and wrinkling and deformation, leading to mechanical damage (such as wrinkling and breakage) or grasping failure during the grasping process. This not only reduces equipment utilization but also increases overall operating costs, severely restricting the widespread application of this technology in large-scale production scenarios.

[0006] Patent application CN114986497A discloses a tobacco leaf grasping method based on centroid detection. This method aims to address the inaccurate grasping and leaf slipping problems caused by unstable leaf positioning and folding in traditional tobacco leaf grasping techniques. The method involves three main steps: First, an image of the tobacco leaf is captured and the veins are extracted to generate a vein image. An image sensor is used to capture the tobacco leaf from above, distortion correction is performed, and a neural network model is used to segment the veins. This results in a clear vein image, and the centroid coordinates of the veins in each grasping area are calculated. The vein image is segmented to generate a partitioned image, and a minimum bounding rectangle is generated for each grasping area. The centroid coordinates are calculated from this image, providing a precise location for subsequent grasping. Finally, an array of suction cups is used to grasp the tobacco leaf. Based on the centroid coordinates, the motion vector of each suction cup is determined, controlling the suction cup's motion to achieve grasping of the tobacco leaf.

[0007] However, this method still has significant limitations in actual engineering applications. First, the effectiveness of the vein segmentation and detection strategy it proposes is highly dependent on the complete flat state of the tobacco leaves: when tobacco leaves are folded or staggered in a natural stacking scenario, the vein structure may be partially or completely blocked, causing the segmentation and detection algorithm to fail, which in turn directly affects the accuracy and robustness of subsequent grasping operations. Secondly, although the control strategy of planning the suction cup movement vector based solely on the center of mass coordinates can improve the stability of the grasping process, it is difficult to ensure the consistency of the orientation of the tobacco leaves on the conveyor belt after each grasping - this orientation deviation will not only reduce the transmission efficiency, but may also cause connection obstacles or misoperation in subsequent processing steps. In addition, the technology has insufficient ability to accurately control the number of grasping. In actual operations, it is easy to deviate from the preset standard due to fluctuations in the stacking shape of tobacco leaves or limitations in sensor accuracy, thereby affecting the quantitative control requirements of the production process and the stability of product quality. Summary of the Invention

[0008] In view of the defects in the prior art, the purpose of the present invention is to provide a vision-guided automatic quantitative feeding system for redried tobacco leaves and an implementation method.

[0009] The vision-guided automatic quantitative feeding system for redried tobacco leaves provided by the present invention comprises: a six-degree-of-freedom robotic arm, a binocular industrial camera, a two-finger translation gripper, a weighing sensor, and an industrial control computer;

[0010] The six-degree-of-freedom robotic arm, binocular industrial camera, two-finger translation gripper and weighing sensor are respectively connected to the industrial control computer;

[0011] The binocular industrial camera is connected to a connector via a fixed base, and the connector is connected to the upper end of the two-finger translation gripper;

[0012] The load cell is connected to a standard flange at the end of the six-degree-of-freedom robotic arm;

[0013] The two-finger translation gripper is connected to the end of the six-degree-of-freedom robotic arm through a weight sensor, and the weight of the tobacco leaves grasped by the gripper is measured by the tension between the end of the robotic arm and the gripper.

[0014] Two industrial cameras are connected via a base to form a stereoscopic vision system, which is then connected to a two-finger translational gripper via a connector. The stereoscopic vision system uses concentric circle coded targets to extract feature points, thereby calibrating the internal and external parameters and distortion coefficients of the two industrial cameras. The stereoscopic vision system serves as the image data acquisition device of the feeding system, capturing images of tobacco leaves.

[0015] The tobacco leaves delivered from the warehouse are temporarily stored on the operating platform. The robotic arm moves to the photo-taking position and uses a stereo vision system to capture tobacco leaf images for target detection and three-dimensional measurement. The optimal grasping posture and gripper opening degree are calculated, and the posture is transferred to the robotic arm. The robotic arm moves to the target position and uses a weighing sensor to measure the weight to achieve fixed-weight tobacco leaf grasping.

[0016] Preferably, the stereo vision system adopts an eye-on-hand calibration mode, pre-plans a plurality of spatially evenly distributed robotic arm postures, and during the feature point extraction process, utilizes the autopolar triangle property of concentric circles to locate the center point of each concentric circle target;

[0017] Suppose there are a pair of concentric circles C1 and C2, and suppose point x and line a are the common pole and common polar line of C1 and C2, then:

[0018]

[0019] Where λ is a non-zero parameter, I is the identity matrix; the common poles of C1 and C2 are expressed by Solve the eigenvalues ​​and eigenvectors of ;

[0020] On the projected image plane, the common autopolar triangle property of the two ellipses still holds:

[0021]

[0022] Where H represents the homography transformation matrix between the calibration template plane and the camera imaging plane; E1 and E2 are the elliptical contours obtained by projecting C1 and C2 through the homography matrix transformation;

[0023] Therefore, it can be solved by The eigenvalues ​​and eigenvectors are used to calculate the center projection coordinates of the concentric circle target, that is, the feature point coordinates of the coded target, for subsequent camera calibration;

[0024]

[0025] Among them, P is the coordinate of the feature point to be sought, for The characteristic value of

[0026] Before using the camera for measurement, it is necessary to complete the camera calibration, that is, obtain the camera parameters;

[0027] The calibration plate is freely moved within the camera field of view, and a calibration plate image is obtained from different angles. The coordinates of each feature corner point in the calibration plate image are detected, and multiple feature corner points of each image are extracted with sub-pixel accuracy. The correspondence between the feature points is used to solve the world coordinate system OX in each perspective. W Y W Z W With pixel coordinate system o u -The homography matrix between uv, taking the plane where the calibration plate is located as the world coordinate system Z W = 0, then for the points on the calibration plate plane:

[0028]

[0029] Among them, r1, r2, r3 represent the column vectors of the rotation matrix R between the camera imaging plane and the calibration plate; t represents the translation vector between the camera imaging plane and the calibration plate, that is, the world coordinates of the camera optical center; K is the camera internal parameter matrix, s is the scale factor; let Then we have:

[0030]

[0031] Given the spatial coordinates and image coordinates of the calibration points, the camera's intrinsic and extrinsic parameters and distortion coefficients are obtained by solving the least squares equation and further optimizing it with the Levenberg-Marquarat algorithm, and converging to the global optimal solution through multiple iterations.

[0032] Preferably, the camera coordinate system and the robot arm end coordinate system satisfy the hand-eye equation:

[0033] AX=XB

[0034] Among them, the A matrix is ​​the transformation between the end coordinate system of the manipulator and the base coordinate system; the B matrix is ​​the transformation between the camera coordinate system and the world coordinate system, that is, the calibration plate coordinate system; X is the transformation relationship between the end coordinate system of the manipulator and the camera coordinate system;

[0035] Expand the matrix in the equation and express it as:

[0036]

[0037] R a ,t a ,R b ,t bare known values ​​and are all orthogonal unit matrices. They are solved by the TASI two-step method to obtain the rotation matrix R in the hand-eye matrix gc and the translation vector t gc .

[0038] Preferably, in a stereo vision system, after the vision system is calibrated, epipolar correction is first performed on the images of the left and right cameras to ensure that the matching points of the two camera images are at the same vertical coordinate. The obtained epipolar correction image is input into the stereo matching network to extract depth information and obtain disparity values;

[0039] According to the obtained disparity value, the three-dimensional coordinates are calculated using the internal and external parameters of the camera. The two-dimensional coordinates of each image point are converted into three-dimensional coordinates through the known camera parameters and the disparity value to reconstruct a complete three-dimensional point cloud.

[0040] Preferably, a six-degree-of-freedom robotic arm guides a stereo vision system to collect tobacco leaf images in different positions and postures, construct a rotational bounding box annotation dataset, and train a lightweight object detection network to achieve accurate detection of the petiole and leaf tip areas. Specifically:

[0041] Perform data cleaning on all image data and retain a preset number of images through dataset expansion;

[0042] Use the rotating bounding box annotation tool rolabelimg to perform rotating bounding box annotation on the collected tobacco leaf images, and annotate the petiole area vein and leaf tip area tac of each image respectively. The corresponding annotation information is as follows:

[0043] B=(x c ,y c ,w,h,θ)

[0044] Among them, (x c ,y c ) is the coordinate of the center of the bounding box, w is the width of the bounding box, h is the height of the bounding box; θ is the rotation angle, ranging from [-90°, 90°];

[0045] Generate a json file corresponding to each image, convert the json file into a txt file required for deep learning network training through dataset conversion, and finally divide the dataset into training set, validation set and test set in a ratio of 8:1:1;

[0046] The YOLO algorithm was used as the baseline model, and the YOLO11-OBB model was used to train the network on the constructed tobacco leaf rotated bounding box dataset. During the training process, the number of epochs was set to 400, and the model with the best evaluation index was saved. The network weight file obtained by training on the Python side was converted to the ONNX format, and the network model was called on the C++ side using the ONNX link dependency library.

[0047] Preferably, quantitative grasping is performed using three-dimensional measurement and target detection technology and gravity sensor technology, and the specific steps are as follows:

[0048] Step 1: Use target detection technology to detect tobacco leaves and obtain the rotation bounding box B of its petiole area and leaf tip area vein and B tac ;

[0049] Step 2: Connect the center points of the two rotated bounding boxes to generate a vector D = C indicating the overall placement direction of the tobacco leaves tac -C vein , where C tac and C vein are the center coordinates of the two rotating bounding boxes; the direction of the end effector's connecting rod is set to be parallel to the obtained overall placement direction vector of the tobacco leaf, ensuring that the gripper's posture approaching the tobacco leaf is consistent with the natural placement direction of the tobacco leaf;

[0050] Step 3: Based on the detected petiole region bounding box, select a fixed width W1 from the leftmost end and compare the width with the petiole region bounding box height h vein The jointly defined area is used as the initial grasping target area B1, and this fixed width is set as the predetermined distance for the initial gripper opening;

[0051] Step 4: Calculate the coordinates P of the three-dimensional center point of the initial capture target area B1 defined in step 3 in the camera coordinate system based on the point cloud data obtained by stereo matching. c , this point is the grasping target point that the robot arm needs to locate; at the same time, the actual distance W of the width W1 in three-dimensional space is calculated c1 , this distance is the actual width of the initial gripper opening;

[0052] Step 5: Use the pre-calibrated hand-eye transformation matrix R camera_to_end and T camera_to_end , convert the three-dimensional coordinates of the grasping target point obtained in step 4 from the camera coordinate system to the robot arm base coordinate system:

[0053] P end =R camera_to_end P c +T camera_to_end

[0054] P base =R end_to_base P end +T end_to_base

[0055] Among them, P end 、P base Respectively represent the three-dimensional coordinates of the point in the robot arm end coordinate system and the robot arm base coordinate system; send the converted target point coordinates, tobacco leaf direction vector, and gripper opening width parameters to the robot arm and gripper controller respectively, control the robot arm to move to the target point, and drive the gripper to grab the tobacco leaf with a preset posture and opening;

[0056] Step 6: The weighing sensor records the weight of each grab and accumulates it. When the remaining weight required for grabbing is less than the value for one grab, the volume v required for the remaining weight is estimated using the tobacco leaf density model, and then the final gripper opening is calculated;

[0057] Step 7: Using the 3D point cloud data obtained using binocular stereo matching technology, calculate the average tobacco leaf thickness h on the vertical cross section of the petiole area. Calculate the total tobacco leaf length l based on the center point distance and frame height of the rotated bounding box between the petiole and the tip area:

[0058] l=(h tac +h vein ) / 2+||C tac -C vein ||

[0059] h tac is the height of the bounding box of the tip area;

[0060] Based on the required volume of the remaining weight, the length and average thickness of the tobacco leaves, calculate the width W2 required for the final grab:

[0061] W2=v / (hl)

[0062] And set it as the last gripper opening.

[0063] The method for implementing the vision-guided automatic quantitative feeding system for redried tobacco leaves provided by the present invention comprises the following steps:

[0064] Use industrial control computers to control binocular industrial cameras to collect image data;

[0065] The camera's internal and external parameters are obtained through stereo vision system calibration, and image dedistortion processing is performed on the tobacco leaf placement image data;

[0066] Perform hand-eye calibration to obtain the coordinate system transformation relationship between the binocular industrial camera and the six-degree-of-freedom robotic arm;

[0067] Stereo matching is performed on tobacco leaf image data collected by two binocular industrial cameras to generate a 3D point cloud and reconstruct the 3D scene;

[0068] Perform grasping area detection on tobacco leaves in different postures and obtain the rotating bounding boxes of the petiole and tip of the tobacco leaves;

[0069] Combining the 3D point cloud with the regional detection results, the position and posture of the grasping point and the degree of gripper opening and closing are calculated, and quantitative grasping is achieved with the help of weight sensors.

[0070] The three-dimensional point cloud and target detection results are displayed on the interface through the human-computer interaction module, and user operation buttons are provided to set the six-degree-of-freedom robotic arm position and target weight.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] (1) The present invention proposes a vision-guided automatic quantitative feeding system for redried tobacco leaves. The system combines advanced three-dimensional visual measurement technology with robotic arm technology to achieve visual perception of tobacco leaves in different placement positions, directions and states. By utilizing multi-source information fusion calculation, the system can effectively complete the quantitative automatic grasping operation of tobacco leaves, thereby realizing high-speed automatic grasping and feeding of incoming tobacco leaves. It not only effectively eliminates the high cost, low efficiency and high intensity production operations of traditional placement and feeding links, but also basically realizes the functions of automatic feeding, intelligent proportioning and standardized management and control. Through the implementation of the automation system, the manpower input in the production process will be significantly reduced, and the labor intensity of workers will also be greatly reduced, thereby improving overall work efficiency. In addition, the intelligent characteristics of the system will ensure the accuracy of the formula and the stability of tobacco leaf quality. Ultimately, the company will promote digital transformation and enhance market competitiveness while improving product quality.

[0073] (2) The present invention realizes the accurate identification of the placement direction of tobacco leaves through the coordinated application of three-dimensional reconstruction and target detection technology, which enables the robot arm to accurately control the grasping direction when performing the grasping operation, and minimizes the damage to the integrity of the tobacco leaves caused by directional deviation; at the same time, the system dynamically adjusts the grasping width and depth parameters based on the real-time weight value obtained by the weight sensor and the three-dimensional information of the scene, thereby effectively realizing the quantitative grasping function; this solution effectively solves the problems of insufficient accuracy of manual leaf laying and feeding and difficulty in controlling the formula ratio in the traditional leaf threshing and re-roasting process, and meets the tobacco industry's demand for accurate feeding and homogenization processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0075] Figure 1This is the overall schematic diagram of the automated quantitative feeding system designed by the present invention;

[0076] Figure 2 This is a flow chart of the automated quantitative feeding system designed for the present invention;

[0077] Figure 3 Flowchart for visual system calibration;

[0078] Figure 4 It is a schematic diagram of the properties of concentric self-polar triangles;

[0079] Figure 5 Schematic diagram of visual system calibration;

[0080] Figure 6 It is the visual system calibration diagram and calibration result diagram;

[0081] Figure 7 This is a schematic diagram of the calibration of the robotic arm system;

[0082] Figure 8 Schematic diagram of stereo matching and 3D reconstruction;

[0083] Figure 9 This is a three-dimensional reconstruction rendering;

[0084] Figure 10 A diagram showing the format of the dataset used for training;

[0085] Figure 11 This is a diagram of the rotated bounding box format;

[0086] Figure 12 Schematic diagram of loss function and evaluation index during training;

[0087] Figure 13 This is a graph showing the effect of testing the trained model on the test set;

[0088] Figure 14 This is a flow chart of the quantitative grabbing method designed by the present invention;

[0089] Figure 15 Schematic diagram of the overall design of the human-computer interaction interface. DETAILED DESCRIPTION

[0090] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0091] Example

[0092] The purpose of the present invention is to design a system that can automatically and quantitatively load redried tobacco leaves by visually guiding a robotic arm in a complex and harsh redried tobacco production environment, aiming to solve the problems of low accuracy of manual leaf laying and feeding and difficulty in controlling the proportion of the formula in the traditional leaf threshing and redrying process, and meet the needs of the cigarette industry for precise feeding and homogenization processing. The three-dimensional measurement technology is used to estimate the placement of tobacco leaves to obtain three-dimensional information of the tobacco scene, thereby calculating the precise grasping posture and the size of the claw opening and closing, and collaboratively controlling the robotic arm and gripper to perform the grasping operation, and quantifying the tobacco leaves through the weight sensor. By integrating robotic technology and advanced computer vision systems, the tobacco leaves on the operating platform can be automatically grasped and placed, thereby improving the efficiency and accuracy of tobacco leaf feeding and grasping, ensuring the effectiveness and stability of grasping, and reducing the complexity of manual operation.

[0093] This invention primarily uses a binocular industrial camera as a visual sensor and employs visual measurement technology to acquire the three-dimensional coordinates of spatial objects, thereby accurately measuring parameters such as the object's geometric dimensions and motion state. Furthermore, an industrial robot and gripper are combined as actuators, controlling the robot's movement to the target position via commands. Furthermore, a weighing sensor provides auxiliary information, enabling automated quantitative grasping of tobacco leaves.

[0094] The main construction steps of the automated loading system designed by the invention are as follows:

[0095] 1. Industrial on-site hardware system construction: The hardware component of the system mainly consists of four parts: a six-degree-of-freedom robotic arm, a two-finger translational gripper, a binocular stereo vision system, and a weighing sensor. The two-finger translational gripper is connected to the six-degree-of-freedom robotic arm via a circular tension sensor. A stereo vision system is installed at the end of the robotic arm. The stereo vision system consists of two industrial cameras and serves as the image data acquisition device for the automatic feeding system to capture images of tobacco leaves. The tobacco leaves delivered from the warehouse are temporarily stored on the operating platform. The robotic arm moves to the photo-taking position. The stereo vision system uses the image of the tobacco leaves for target detection and measurement. The optimal grasping posture and the degree of gripper opening are calculated. The posture is transferred to the robotic arm, which automatically moves to the target position to complete the grasping. After the grasping is completed, the weight is recorded and the robot automatically moves to the conveyor belt and releases the tobacco leaves for subsequent processing.

[0096] 2. Vision System Parameter Calibration: A high-precision binocular stereo vision system calibration algorithm based on a coded target calibration plate is proposed. Using concentric circular coded targets, feature points are extracted, which are then calibrated to the intrinsic parameters and distortion coefficients of two industrial cameras. Furthermore, to avoid missed feature point detection in uneven ambient lighting, a feature point extraction method suitable for complex industrial environments is designed. This method extracts the coded target's ROI region through object detection and extracts feature points based on the properties of concentric circular autopolar triangles, thereby improving the robustness of vision system calibration.

[0097] 3. Robotic arm system calibration: To ensure accurate coordinate conversion between the robotic arm and the vision system, high-precision hand-eye calibration is required. The calibration plate is fixed in place, and the robotic arm is moved along six axes to capture 30 sets of calibration images. The corresponding position of the robotic arm end in the robotic arm base coordinate system at the time of each set of images is recorded for hand-eye calibration. By solving the hand-eye calibration equations, the rotation and translation matrices between the camera coordinate system and the robotic arm end coordinate system are calculated to ensure accurate calculation of subsequent grasping point position conversions, allowing the robotic arm to accurately move to the corresponding position.

[0098] 4. 3D reconstruction of the tobacco scene: To accurately execute subsequent capture operations, the left and right images are corrected using the calibration parameters of the calibrated binocular camera. A stereo matching network is then used to calculate the disparity using binocular stereo matching. Finally, the 3D coordinates of each point in the image are calculated based on the disparity information, generating a 3D point cloud and reconstructing the 3D scene.

[0099] 5. Deep Learning Object Detection Model Training and Deployment: A robotic arm guides a stereoscopic vision system to capture images of tobacco leaves in various positions and postures. After collecting a sufficient number of leaf images, the images are preprocessed and labeled with petioles and tips using dataset annotation software to determine their orientation and grasping area. A neural network is trained based on the dataset generated for tobacco leaf grasping scenarios to obtain the optimal detection model. This neural network model accurately determines the directional grasping area for tobacco leaves.

[0100] 6. Three-dimensional measurement to achieve quantitative grasping strategy: Use a deep learning model to detect the bounding boxes of the petiole and leaf tip, connect the center points of the two bounding boxes to generate a tobacco leaf direction vector, and set the direction of the gripper connecting rod to be parallel to the vector. Define a fixed width of the petiole area as the initial gripper opening distance, calculate the three-dimensional coordinates of the grasping point and convert it from the camera coordinate system to the robot arm base coordinate system, and send instructions for grasping, transmitting and releasing. Record the weight of each grasp. When the remaining weight is lower than a certain value, estimate the required volume through the density model. According to the distance between the center points of the petiole and the leaf tip bounding box and the frame height, the total length of the tobacco leaf is approximately calculated, and the average thickness of the petiole area is calculated using binocular vision data. Finally, the gripper opening degree is obtained to achieve quantitative grasping.

[0101] 7. Human-Computer Interaction and Data Display: A human-computer interaction interface was designed, integrating a stereo matching network and a target detection network. Through algorithmic design, the interface can display the scene structure point cloud generated by the stereo matching network in real time. Users can view the scene point cloud by zooming, rotating, and other methods, and can choose to save the current point cloud. In addition, the interface also displays a real-time detection rendering of the tobacco leaf placement in the image captured by the binocular camera, allowing users to intuitively observe the detection results. Using buttons on the interface, users can set the robot arm's initial position, release position, and other posture parameters, as well as the target grasping weight. This allows the system to perform effective automated grasping operations based on this information, thereby improving grasping accuracy and efficiency.

[0102] like Figure 1 As shown, the overall design scheme of the design and implementation method of a visually guided automatic quantitative feeding system for redried tobacco leaves designed by the present invention is as follows:

[0103] The system consists of a six-degree-of-freedom (DOF) robotic arm, a binocular industrial camera, a two-finger translation gripper, a load cell, and an industrial control computer for integrated control. Specifically, the binocular industrial camera is connected to a long connector via a fixed base, which in turn is connected to the upper end of the two-finger translation gripper. This integrated component is connected to a standard flange at the end of the six-degree-of-freedom (DOF) robotic arm via a disc-shaped load cell, forming an integrated gravity-vision-grasping execution terminal system. Regarding software communication, the six-degree-of-freedom robotic arm connects to the network port of the industrial control computer via the Ethernet communication protocol (TCP) through its controller, enabling motion command issuance and status feedback. The binocular industrial camera is connected to the industrial control computer via a shielded dual USB 3.0 port. The two-finger translation gripper and the load cell are each connected to a driver via a data cable. Both drivers are then connected to the industrial control computer via a USB to RS-485 converter.

[0104] The two-finger translational gripper is connected to the disc at the end of the six-degree-of-freedom robotic arm via a circular weight sensor. The weight of the tobacco leaves grasped by the gripper is measured by the tension between the end of the robotic arm and the gripper. The stereo vision system, which provides visual perception and three-dimensional measurement, consists of two industrial cameras connected by a base and connected to the gripper via connectors. The stereo vision system serves as the image data acquisition device for the automatic loading system, capturing images of the tobacco leaves. Tobacco leaves delivered from the warehouse are temporarily stored on the operating platform. The robotic arm then moves to the image capture position. The stereo vision system captures images of the tobacco leaves for target detection and three-dimensional measurement. The optimal grasping posture and gripper opening degree are calculated and transferred to the robotic arm. The robotic arm then automatically moves to the target position and uses a load cell to measure the weight, enabling fixed-weight tobacco leaf grasping. The six-degree-of-freedom robotic arm, two-finger translational gripper, binocular stereo vision system, and load cell are each connected to an integrated industrial control computer via wiring, enabling coordinated control of each component by the industrial control computer.

[0105] The overall process of the tobacco leaf automatic quantitative feeding system designed by the present invention is as follows: Figure 2 As shown in the figure, it is mainly divided into hardware modules and software modules. The hardware module mainly includes the following steps: using the industrial control computer to control the industrial camera module to collect image data; using the robotic arm control module to control the posture of the six-degree-of-freedom robotic arm; using the gripper control module to control the opening and closing degree of the gripper to achieve the grasping and releasing of tobacco leaves; and using the weight sensor module to obtain weight information. The software module mainly includes the following steps: obtaining the internal and external parameters of the camera through the visual system calibration module and performing image dedistortion processing on the tobacco leaf placement image data; performing hand-eye calibration through the robotic arm visual system calibration module to obtain the coordinate system conversion relationship between the camera and robotic arm; stereo matching of the tobacco leaf image data collected by the two industrial cameras through the scene 3D reconstruction module to generate a 3D point cloud and reconstruct the 3D scene; the tobacco leaf grasping area detection module detects the grasping area of ​​tobacco leaves in different postures and obtains the rotation bounding box of the tobacco leaf petiole and tip area for subsequent module calculations; the quantitative grasping module combines the 3D point cloud with the target detection results to calculate the grasping point posture and gripper opening and closing degree, and realizes quantitative grasping based on the weight sensor; the 3D point cloud and target detection results are displayed on the interface through the human-computer interaction module, and the user operation button is provided to set the robotic arm position and target weight. The above solution steps are explained in detail below.

[0106] (1) Hardware module

[0107] (1) Industrial camera module

[0108] This invention uses a high-precision binocular industrial camera (resolution 2448×1536) to construct a stereoscopic vision system. This system connects to an industrial control computer via dual shielded USB 3.0 ports, achieving binocular image synchronization. At the software level, a C++ control program developed based on the camera's native SDK dynamically adjusts key imaging parameters, including exposure time, gain, and white balance. The current scene is displayed in real time on a user interface.

[0109] To ensure the relative position between the binocular camera and the end-of-arm remains fixed, the two industrial cameras are securely mounted on a dedicated base. This base not only provides necessary support but also effectively connects the camera base to the gripper via a connecting rod. This is designed to reduce potential vibration and displacement of the cameras during operation, ensuring that the camera's viewing angle and posture remain consistent throughout operation. This ensures that the binocular cameras accurately capture the target object's position and posture during stereo vision processing and object grasping, improving system stability and reliability. Furthermore, the base's design takes into account the requirements of easy maintenance and adjustment, allowing for convenient camera calibration and position adjustment during actual use.

[0110] (2) Robotic arm control module

[0111] The present invention is used in complex and varied scenarios, and tobacco leaves are placed in various postures. In order to achieve complete spatial posture control capabilities, the present invention adopts a six-degree-of-freedom robotic arm. It has three translational degrees of freedom (X / Y / Z movement) and three rotational degrees of freedom (rotation around the X / Y / Z axis), and can independently control the position and direction of the end. When grabbing tobacco leaves, no matter how the petiole is oriented (such as tilted or inverted), the gripper angle can be adjusted by rotating the end joint to accurately align the petiole direction vector.

[0112] The six-degree-of-freedom robotic arm used in this invention is equipped with a teach pendant. During manual commissioning, the teach pendant is used to select between joint motion and rectangular coordinate movement, and the arm's joints are moved to the target position. Simultaneously, the teach pendant reads the three-dimensional X / Y / Z coordinates of the current end position in the arm's base coordinate system, as well as the rotation angles Rx, Ry, and Rz for each axis. This allows the rotation and translation matrices of the end position relative to the base coordinate system to be determined.

[0113] By programming the robot arm instructions on the teach pendant and writing the TCP server program in C++, the communication protocol between the robot arm and the industrial control computer is implemented. The industrial control computer can send the target point posture calculated by the vision system to the robot arm in the form of instructions, thereby controlling the robot arm to automatically move to the target point for grasping.

[0114] (3) Gripper control module

[0115] The proposed two-finger linear gripper is driven by a stepper motor. A precision stepper motor is integrated in the center of the gripper as the power source, directly driving a vertically mounted lead screw mechanism through a coupling. The motor's rotational motion is converted by the lead screw into linear displacement of the coil sleeve, driving the two-finger linkage mechanism for synchronous opening and closing. Upward movement of the coil sleeve corresponds to radial contraction (grasping) of the two fingers, while downward movement of the coil sleeve corresponds to radial expansion (release) of the two fingers.

[0116] The two-finger linear gripper is controlled by an industrial control computer, which generates the target opening and closing distance and motion parameters (motor revolutions, speed, and acceleration / deceleration curves) based on the grasping area width calculated by the vision system. Commands are encapsulated using the Modbus-RTU protocol and sent to the stepper motor driver via a USB to RS-485 serial bus. The driver parses the commands and converts them into pulse signals, driving the stepper motor to open and close the gripper. The number of revolutions precisely controls the gripper's opening and closing distance, and the speed of the rotation can be adjusted to adjust the speed of the grasping action.

[0117] To achieve quantitative grasping, a linear mapping relationship between gripper opening distance and motor revolutions was pre-established, and the degree of opening and closing was calibrated. After receiving the grasping width calculated by the detection module, the system infers the required number of motor revolutions N and sends a Modbus command. The driver performs precise positioning, enabling the industrial control computer to control the gripper.

[0118] (2) Software Module

[0119] (1) Vision system calibration module

[0120] When performing target grasping tasks, the accuracy of the target's pose calculation is a key factor affecting the mission's success rate. Especially when the target is a complex object, the accuracy of pose estimation is even more stringent. During calibration, it is necessary to accurately obtain the internal parameters of the two industrial cameras and the transformation relationship between the two camera coordinate systems.

[0121] This system uses the Eye-to-Hand calibration mode, which requires pre-planning 30 spatially evenly distributed robotic arm postures (covering the typical motion range of the workspace) to ensure the integrity and posture diversity of the calibration plate image acquisition. The visual system calibration flow chart is as follows: Figure 3 The subsequent calibration steps are based on these preset poses.

[0122] 1) Concentric circle coded target feature point positioning

[0123] The calibration target used for vision system calibration is a high-precision coded target with concentric circles. It combines the advantages of multiple calibration targets. Center positioning improves noise and distortion immunity, and supports robust calibration and feature point matching under partial occlusion. During feature point extraction, the self-polar triangle property of concentric circles is used to locate the center of each concentric circle target, resulting in a more accurate center.

[0124] Suppose there are two concentric circles C1 and C2. Let point x and line a be the common pole and common polar line of C1 and C2. Figure 4 As shown. Then it satisfies:

[0125]

[0126] Where λ is a non-zero parameter and I is the identity matrix. The common poles of C1 and C2 can be obtained by The eigenvalues ​​and eigenvectors of are solved. Its eigenvalues ​​satisfy λ1=λ2≠λ3, and the eigenvector v3 corresponding to λ3 is the coordinate of the center of the concentric circles.

[0127] On the projected image plane, the common autopolar triangle property of the two ellipses still holds:

[0128]

[0129] Where H represents the homography transformation matrix between the calibration template plane and the camera imaging plane. E1 and E2 are the elliptical contours obtained by projecting C1 and C2 through the homography matrix transformation. Therefore, we can solve The eigenvalues ​​and eigenvectors are used to calculate the center projection coordinates of the concentric circle target, that is, the feature point coordinates of the coded target, for subsequent camera calibration.

[0130]

[0131] Among them, P is the coordinate of the feature point to be sought, Among the eigenvalues ​​λ′1, λ′2, λ′3 (where λ′1=λ′2≠λ′3), the eigenvector corresponding to λ′3.

[0132] 2) Camera internal and external parameter calibration

[0133] Before using the camera for measurement, it is necessary to complete the camera calibration, that is, obtain the camera parameters. Figure 5 As shown in the figure, the calibration plate is freely moved within the camera field of view, and a calibration plate image is obtained from different angles. The coordinates of each feature corner point in the calibration plate image are detected, and n (n≥4) feature corner points of each image are extracted with sub-pixel accuracy. The correspondence between the feature points is used to solve the world coordinate system OX in each perspective. W Y W Z WWith pixel coordinate system o u -uv homography matrix. Take the plane where the calibration plate is located as the world coordinate system Z W = 0. Then for the points on the calibration plate plane:

[0134]

[0135] Among them, r1, r2, r3 represent the column vectors of the rotation matrix R between the camera imaging plane and the calibration plate; t represents the translation vector between the camera imaging plane and the calibration plate, which is also the world coordinate of the camera optical center; K is the camera internal parameter matrix, and s is the scale factor. Let Then formula (4) can be written as:

[0136]

[0137] Given the spatial coordinates and image coordinates of the calibration points, the camera's intrinsic and extrinsic parameters and distortion coefficients are obtained by solving the least squares equation and further optimizing it with the Levenberg-Marquarat algorithm, and converging to the global optimal solution through multiple iterations. Figure 6 The image of some calibration plates and the calibration results of the visual system are shown, where left_K and right_K are the intrinsic coefficient matrices of the left and right cameras respectively, the corresponding D is the distortion coefficient matrix of the corresponding camera, and R and T are the rotation and translation matrices between the left and right cameras.

[0138] (2) Robotic arm system calibration module

[0139] Based on the calibration plate image collected at the aforementioned preset pose and the visual system calibration results, in order to obtain the pose information of the observed object in the robotic arm coordinate system, it is necessary to obtain the relative relationship between the camera and the robotic arm end coordinate system, so as to finally convert the object pose information from the camera coordinate system to the robotic arm coordinate system. Figure 7 This is a schematic diagram of the hand-eye calibration of the robotic arm. The camera coordinate system and the robotic arm end coordinate system satisfy the hand-eye equation:

[0140] AX=XB

[0141] Among them, the A matrix represents the transformation between the end coordinate system of the robot arm and the base coordinate system. The B matrix represents the transformation between the camera coordinate system and the world coordinate system, that is, the calibration plate coordinate system. X represents the transformation relationship between the end coordinate system of the robot arm and the camera coordinate system. The A and B matrices are known conditions. It can be seen from the hand-eye equation that X is an unknown parameter. In the process of obtaining X, considering the uniqueness of the solution, the data is collected multiple times during the experiment. A in the formula represents the relative posture relationship when the camera moves from A1 to A2, and B can be read from the robot arm controller. Expanding the matrix in the equation, the above formula can be expressed as:

[0142]

[0143] In the process of solving the above equation, the only parameter to be solved is R gc , t gc , that is, the rotation matrix and translation vector in the hand-eye matrix, and R in the above formula a ,t a ,R b ,t b are known values ​​and are all orthogonal unit matrices. They are solved by the TASI two-step method to obtain the required transformation matrix R between the camera and robotic arm coordinate systems. gc , t gc .

[0144] (3) Scene 3D reconstruction module

[0145] In a stereo vision system, after the visual system is calibrated, the images of the left and right cameras are first subjected to epipolar correction to ensure that the matching points of the two camera images are at the same vertical coordinate. The obtained epipolar correction map is input into the stereo matching network to extract depth information. In traditional stereo matching, the search direction between the two corrected images is only on the epipolar line. For non-ideal stereo correction, CREStereo adopts a 2D-1D alternating local search strategy to improve matching accuracy. In order to maintain the details of high-resolution input while maintaining robustness, CREStereo constructs a recursive update module (RUM) based on the GRU module and the adaptive group correlation layer (AGCL). Through the process of recursive refinement, the disparity value is continuously adjusted to reduce the matching error, thereby improving the quality of the disparity map. Finally, a high-quality disparity map is output. Once the final disparity value is obtained, the internal and external parameters of the camera are used to calculate the three-dimensional coordinates. Through the known camera parameters and combined with the disparity value, the two-dimensional coordinates of each image point can be converted into three-dimensional coordinates to reconstruct a complete three-dimensional point cloud containing the depth information of objects in the scene and reflecting the shape and structure of the objects. Figure 8 This is a flowchart of scene stereo matching and 3D reconstruction, where the network inputs I1 and I2 are the images after epipolar correction of the left and right cameras. Figure 9 The 3D reconstruction result.

[0146] (4) Tobacco leaf grabbing area detection module

[0147] This module uses a six-degree-of-freedom robotic arm to guide a binocular vision system to collect images of tobacco leaves in different positions and postures, construct an Oriented Bounding Box (OBB) annotation dataset, and train a lightweight object detection network to achieve accurate detection of petioles and leaf tips, providing position and direction information support for subsequent automated grasping. The specific implementation steps are as follows:

[0148] 1) Dataset creation and annotation

[0149] The present invention originally collected about 380 images of tobacco leaves, including tobacco leaf image data in different placement positions and postures. All image data were cleaned (fuzzy / occluded invalid samples were eliminated), and 1,350 high-quality images were retained through data set expansion. Since the automated and precise capture of tobacco leaves not only depends on the accuracy of tobacco leaf area positioning, but also requires directional information of tobacco leaf placement. Therefore, a rotated bounding box (OBB) is used. Compared with the traditional axis-aligned bounding box (AABB, Axis-Aligned Bounding Boxes), OBB allows the rotation of the object bounding box. This advantage makes the directional bounding box OBB more accurate in surrounding the target object in the picture, and at the same time, when encountering situations such as random placement of tobacco leaves, its specific direction can be obtained.

[0150] Use the rotation bounding box annotation tool rolabelimg to perform rotation bounding box annotation on the collected tobacco leaf images, such as Figure 10 As shown. Label the petiole area vein and the leaf tip area tac of each image respectively, and obtain the corresponding annotation information as follows:

[0151] B=(x c ,y c ,w,h,θ)

[0152] Among them, (x c ,y c ) is the coordinate of the center of the bounding box, w is the width of the bounding box, h is the height of the bounding box, and θ is the rotation angle. θ is usually the angle measured counterclockwise from the horizontal line, ranging from [-90°, 90°]. Figure 11 As shown in the figure, a JSON file corresponding to each image is generated. Through dataset conversion, the JSON file is converted into the txt file required for deep learning network training. Finally, the dataset is divided into training, validation, and test sets in an 8:1:1 ratio.

[0153] 2) Network training and testing

[0154] Because actual industrial production places high demands on real-time detection, this paper uses the state-of-the-art YOLO (You Only Look Once) algorithm as a baseline model and uses the YOLO11-OBB model to train the network on a constructed tobacco leaf rotated bounding box dataset. The YOLO11-OBB model is an improvement on YOLO, specifically designed for processing rotated bounding boxes. This model effectively combines the advantages of OBB and demonstrates superior performance in complex object detection tasks. During training, the number of epochs is set to 400, and the model with the best evaluation metrics is saved. Figure 12 The figure shows the changes of indicators such as loss function during network training. The schematic diagram of some effects of the model on the test set is as follows Figure 13 shown.

[0155] To apply the object detection network to actual projects, the present invention also deploys the trained model on the C++ side. The specific operation includes converting the network weight file (.pt) trained on the Python side into the ONNX format and using the ONNX link dependency library to call the network model on the C++ side.

[0156] (5) Quantitative grabbing module

[0157] By using three-dimensional measurement and target detection technology and gravity sensor technology, the present invention designs a quantitative grasping method. The flow chart is as follows Figure 14 The specific steps are as follows:

[0158] 1) Tobacco leaf pose estimation: Use target detection technology to detect tobacco leaves and obtain the rotation bounding box B of the petiole and tip areas. vein and B tac .

[0159] 2) Determine the gripper posture: Connect the center points of the two rotating bounding boxes to generate a vector D = C indicating the overall placement direction of the tobacco leaves. tac -C vein , where C tac and C vein The center coordinates of the two rotating bounding boxes are set. The direction of the connecting rod of the end effector (gripper) is set to be parallel to the obtained overall placement direction vector of the tobacco leaf, ensuring that the posture of the gripper approaching the tobacco leaf is consistent with the natural placement direction of the tobacco leaf.

[0160] 3) Initial gripper opening setting: Based on the detected petiole region boundary box, a fixed width W1 is selected from the leftmost end, and the width is combined with the petiole region boundary box height h vein The jointly defined area is used as the initial grasping target area B1, and the fixed width is set as the predetermined distance for the initial grasping hand to open.

[0161] 4) Grasping point calculation: Based on the point cloud data obtained by stereo matching, the coordinates P of the three-dimensional center point of the initial grasping target area B1 defined in step 3 in the camera coordinate system are calculated. c , this point is the grasping target point that the robot arm needs to locate; at the same time, the actual distance W of the width W1 in three-dimensional space is calculated c1 , this distance is the actual width of the initial gripper opening.

[0162] 5) Coordinate transformation and execution: Using the pre-calibrated hand-eye transformation matrix R camera_to_end and T camera_to_end , convert the three-dimensional coordinates of the grasping target point obtained in step 4 from the camera coordinate system to the robot arm base coordinate system:

[0163] P end =R camera_to_end P c +T camera_to_end

[0164] P base =R end_to_base P end +T end_to_base

[0165] Among them, P end 、P base The three-dimensional coordinates of the point are represented in the manipulator's end coordinate system and the manipulator's base coordinate system, respectively. The converted target point coordinates, tobacco leaf direction vector, gripper opening width, and other parameters are sent to the manipulator and gripper controller, respectively. The manipulator is controlled to move to the target point and drive the gripper to grasp the tobacco leaf at a preset posture and opening.

[0166] 6) Quantitative Control: The weighing sensor records and accumulates the weight of each grab. When the required remaining weight is less than a certain value, the volume v required for the remaining weight is estimated using the tobacco leaf density model, and then the final gripper opening is calculated.

[0167] 7) Calculation of the last grip opening:

[0168] (i) Height measurement: Using the three-dimensional point cloud data obtained by binocular vision stereo matching technology, the average tobacco leaf thickness h in the vertical section of the petiole area is calculated.

[0169] (ii) Length measurement: Based on the distance between the center points of the rotated bounding box of the petiole and the tip area and the frame height, the total length l of the tobacco leaf is approximately calculated:

[0170] l=(h tac +h vein ) / 2+||C tac -C vein ||

[0171] h tac is the height of the bounding box of the tip area;

[0172] (iii) Calculation of the final grip opening: Based on the volume required for the remaining weight, the length and average thickness of the tobacco leaves, calculate the width W2 required for the final grip:

[0173] W2=v / (hl)

[0174] Set it as the last gripper opening. Repeat steps 4) and 5).

[0175] (6) Human-computer interaction module

[0176] A comprehensive human-computer interaction interface (HUI) has been designed to enhance the user experience with the system. This interface, which incorporates a stereo matching network and an object detection network, generates and displays the scene structure point cloud output by the stereo matching network in real time. This allows users to visually observe the distribution of tobacco leaves in three-dimensional space. Users can freely view the scene point cloud through interactive methods such as zooming and rotating, and can choose to save the current point cloud data for later analysis or reference. Furthermore, the interface displays a real-time image of the tobacco leaf placement detected in the binocular camera image, allowing users to intuitively observe the algorithm's performance in real-world applications.

[0177] Through the various control buttons on the interface, users can easily set the initial position, release position and other posture parameters of the robot arm. These settings ensure that the robot arm can accurately position and achieve the expected results when performing grasping tasks. At the same time, users can also set the target grasping weight so that the system can effectively automate the grasping operation based on these settings. The specific design diagram of the human-computer interaction interface is shown below. Figure 15 shown.

[0178] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0179] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A vision-guided automatic quantitative feeding system for redried tobacco leaves, characterized in that: include: Six-degree-of-freedom robotic arm, binocular industrial camera, two-finger translational gripper, weighing sensor and industrial control computer; The six-degree-of-freedom robotic arm, binocular industrial camera, two-finger translation gripper and weighing sensor are respectively connected to the industrial control computer; The binocular industrial camera is connected to a connector via a fixed base, and the connector is connected to the upper end of the two-finger translation gripper; The load cell is connected to a standard flange at the end of the six-degree-of-freedom robotic arm; The two-finger translation gripper is connected to the end of the six-degree-of-freedom robotic arm through a weight sensor, and the weight of the tobacco leaves grasped by the gripper is measured by the tension between the end of the robotic arm and the gripper. Two binocular industrial cameras are connected via a base to form a stereo vision system, which is then connected to a two-finger translational gripper via a connector. The stereo vision system uses concentric circle coded targets to extract feature points, thereby calibrating the intrinsic parameters and distortion coefficients of the two industrial cameras. The stereo vision system serves as the image data acquisition device of the feeding system, used to collect images of tobacco leaves. The tobacco leaves delivered from the warehouse are temporarily stored on the operating platform. The robotic arm moves to the photo-taking position and uses a stereo vision system to capture tobacco leaf images for target detection and three-dimensional measurement. The optimal grasping posture and gripper opening degree are calculated, and the posture is transferred to the robotic arm. The robotic arm moves to the target position and uses a weighing sensor to measure the weight to achieve fixed-weight tobacco leaf grasping.

2. The vision-guided automatic quantitative feeding system for redried tobacco leaves according to claim 1 is characterized in that: The stereo vision system adopts the eye-on-hand calibration mode, pre-plans multiple spatially evenly distributed robotic arm postures, and during the feature point extraction process, uses the self-polar triangle property of concentric circles to locate the center point of each concentric circle target; Suppose there are a pair of concentric circles C1 and C2, and suppose point x and line a are the common pole and common polar line of C1 and C2, then: Where λ is a non-zero parameter, I is the identity matrix; the common poles of C1 and C2 are expressed by Solve the eigenvalues ​​and eigenvectors of ; On the projected image plane, the common autopolar triangle property of the two ellipses still holds: Where H represents the homography transformation matrix between the calibration template plane and the camera imaging plane; E1 and E2 are the elliptical contours obtained by projecting C1 and C2 through the homography matrix transformation; Therefore, it can be solved by The eigenvalues ​​and eigenvectors are used to calculate the center projection coordinates of the concentric circle target, that is, the feature point coordinates of the coded target, for subsequent camera calibration; Among them, P is the coordinate of the feature point to be sought, for The characteristic value of Before using the camera for measurement, it is necessary to complete the camera calibration, that is, obtain the camera parameters; The calibration plate is freely moved within the camera field of view, and a calibration plate image is obtained from different angles. The coordinates of each feature corner point in the calibration plate image are detected, and multiple feature corner points of each image are extracted with sub-pixel accuracy. The correspondence between the feature points is used to solve the world coordinate system OX in each perspective. W Y W Z W With pixel coordinate system o u -The homography matrix between uv, taking the plane where the calibration plate is located as the world coordinate system Z W = 0, then for the points on the calibration plate plane: Among them, r1, r2, r3 represent the column vectors of the rotation matrix R between the camera imaging plane and the calibration plate; t represents the translation vector between the camera imaging plane and the calibration plate, that is, the world coordinates of the camera optical center; K is the camera internal parameter matrix, s is the scale factor; let Then we have: Given the spatial coordinates and image coordinates of the calibration points, the camera's intrinsic and extrinsic parameters and distortion coefficients are obtained by solving the least squares equation and further optimizing it with the Levenberg-Marquarat algorithm, and converging to the global optimal solution through multiple iterations.

3. The vision-guided automatic quantitative feeding system for redried tobacco leaves according to claim 1 is characterized in that: The camera coordinate system and the robotic arm end coordinate system satisfy the hand-eye equation: AX=XB Among them, the A matrix is ​​the transformation between the end coordinate system of the manipulator and the base coordinate system; the B matrix is ​​the transformation between the camera coordinate system and the world coordinate system, that is, the calibration plate coordinate system; X is the transformation relationship between the end coordinate system of the manipulator and the camera coordinate system; Expand the matrix in the equation and express it as: R a ,t a ,R b ,t b are known values ​​and are all orthogonal unit matrices. They are solved by the TASI two-step method to obtain the rotation matrix R in the hand-eye matrix gc and the translation vector t gc .

4. The vision-guided automatic quantitative feeding system for redried tobacco leaves according to claim 1 is characterized in that: In a stereo vision system, after the vision system is calibrated, the images of the left and right cameras are firstly subjected to epipolar correction to ensure that the matching points of the two camera images are at the same vertical coordinate. The obtained epipolar correction map is input into the stereo matching network to extract the depth information and obtain the disparity value. According to the obtained disparity value, the three-dimensional coordinates are calculated using the internal and external parameters of the camera. The two-dimensional coordinates of each image point are converted into three-dimensional coordinates through the known camera parameters and the disparity value to reconstruct a complete three-dimensional point cloud.

5. The vision-guided automatic quantitative feeding system for redried tobacco leaves according to claim 1 is characterized in that: Using a six-degree-of-freedom robotic arm to guide a stereo vision system, we collected images of tobacco leaves in different positions and postures, constructed a rotational bounding box annotation dataset, and trained a lightweight object detection network to achieve accurate detection of petioles and leaf tips. Specifically: Perform data cleaning on all image data and retain a preset number of images through dataset expansion; Use the rotating bounding box annotation tool rolabelimg to perform rotating bounding box annotation on the collected tobacco leaf images, and annotate the petiole area vein and leaf tip area tac of each image respectively. The corresponding annotation information is as follows: B=(x c ,y c ,w,h,θ) Among them, (x c ,y c ) is the coordinate of the center of the bounding box, w is the width of the bounding box, h is the height of the bounding box; θ is the rotation angle, ranging from [-90°, 90°]; Generate a json file corresponding to each image, convert the json file into a txt file required for deep learning network training through dataset conversion, and finally divide the dataset into training set, validation set and test set in a ratio of 8:1:1; The YOLO algorithm was used as the baseline model, and the YOLO11-OBB model was used to train the network on the constructed tobacco leaf rotated bounding box dataset. During the training process, the number of epochs was set to 400, and the model with the best evaluation index was saved. The network weight file obtained by training on the Python side was converted to the ONNX format, and the network model was called on the C++ side using the ONNX link dependency library.

6. The vision-guided automatic quantitative feeding system for redried tobacco leaves according to claim 1 is characterized in that: Quantitative grasping is performed using 3D measurement, target detection, and gravity sensor technology. The specific steps are as follows: Step 1: Use target detection technology to detect tobacco leaves and obtain the rotation bounding box B of its petiole area and leaf tip area vein and B tac ; Step 2: Connect the center points of the two rotated bounding boxes to generate a vector D = C indicating the overall placement direction of the tobacco leaves tac -C vein , where C tac and C vein are the center coordinates of the two rotating bounding boxes; the direction of the end effector's connecting rod is set to be parallel to the obtained overall placement direction vector of the tobacco leaf, ensuring that the gripper's posture approaching the tobacco leaf is consistent with the natural placement direction of the tobacco leaf; Step 3: Based on the detected petiole region bounding box, select a fixed width W1 from the leftmost end and compare the width with the petiole region bounding box height h vein The jointly defined area is used as the initial grasping target area B1, and this fixed width is set as the predetermined distance for the initial gripper opening; Step 4: Calculate the coordinates P of the three-dimensional center point of the initial capture target area B1 defined in step 3 in the camera coordinate system based on the point cloud data obtained by stereo matching. c , this point is the grasping target point that the robot arm needs to locate; at the same time, the actual distance W of the width W1 in three-dimensional space is calculated c1 , this distance is the actual width of the initial gripper opening; Step 5: Use the pre-calibrated hand-eye transformation matrix R camera_to_end and T camera_to_end , convert the three-dimensional coordinates of the grasping target point obtained in step 4 from the camera coordinate system to the robot arm base coordinate system: P end =R camera_to_end P c +T camera_to_end P base =R end_to_base P end +T end_to_base Among them, P end 、P base Respectively represent the three-dimensional coordinates of the point in the robot arm end coordinate system and the robot arm base coordinate system; send the converted target point coordinates, tobacco leaf direction vector, and gripper opening width parameters to the robot arm and gripper controller respectively, control the robot arm to move to the target point, and drive the gripper to grab the tobacco leaf with a preset posture and opening; Step 6: The weighing sensor records the weight of each grab and accumulates it. When the remaining weight required for grabbing is less than the value for one grab, the volume v required for the remaining weight is estimated using the tobacco leaf density model, and then the final gripper opening is calculated; Step 7: Using the 3D point cloud data obtained using binocular stereo matching technology, calculate the average tobacco leaf thickness h on the vertical cross section of the petiole area. Calculate the total tobacco leaf length l based on the center point distance and frame height of the rotated bounding box between the petiole and the tip area: l=(h tac +h vein ) / 2+||C tac -C vein || h tac is the height of the bounding box of the tip area; Based on the required volume of the remaining weight, the length and average thickness of the tobacco leaves, calculate the width W2 required for the final grab: W2=v / (hl) And set it as the last gripper opening.

7. A method for implementing a visually guided automatic quantitative feeding system for redried tobacco leaves according to any one of claims 1 to 6, characterized in that: The steps include: Use industrial control computers to control binocular industrial cameras to collect image data; The camera's internal and external parameters are obtained through stereo vision system calibration, and image dedistortion processing is performed on the tobacco leaf placement image data; Perform hand-eye calibration to obtain the coordinate system transformation relationship between the binocular industrial camera and the six-degree-of-freedom robotic arm; Stereo matching is performed on tobacco leaf image data collected by two binocular industrial cameras to generate a 3D point cloud and reconstruct the 3D scene; Perform grasping area detection on tobacco leaves in different postures and obtain the rotating bounding boxes of the petiole and tip of the tobacco leaves; Combining the 3D point cloud with the regional detection results, the position and posture of the grasping point and the degree of gripper opening and closing are calculated, and quantitative grasping is achieved with the help of weight sensors. The three-dimensional point cloud and target detection results are displayed on the interface through the human-computer interaction module, and user operation buttons are provided to set the six-degree-of-freedom robotic arm position and target weight.

Images