Ginger bulbil intelligent identification and positioning method and system based on deep learning

Abstract

The invention discloses a ginger bulbil intelligent identification and positioning method and system based on deep learning. The method comprises the following steps: data acquisition and enhancement; training and deploying a lightweight target detection model; performing instance segmentation to obtain bulbil masks; camera calibration and coordinate conversion; performing PCA and direction disambiguation; performing hand-eye calibration and mechanical arm coordinate system conversion; calculating a correction angle; and a driving and executing stage. According to the invention, high-precision detection and positioning of bulbil can be realized under a complex background, limitation of a traditional image processing method under morphological diversity and environmental interference is effectively overcome, it is ensured that the system meets millisecond-level response requirements in real-time image processing, reliable guarantee is provided for high-speed mechanical operation, and the method is suitable for popularization and application. The whole process automation from bulbil detection, pose calculation to angle correction is realized, the manual intervention requirement is remarkably reduced, the operation efficiency is improved, the bulbil angle correction accuracy is ensured, and a reliable basis is provided for subsequent ginger directional seeding.

Description

Technical Field

[0001] This invention relates to the field of agricultural machine vision and intelligent equipment technology, and in particular to a method and system for intelligent identification and localization of ginger buds based on deep learning. Background Technology

[0002] The orientation of ginger buds during planting directly affects the uniformity of emergence and later yield. Traditional methods mainly rely on manual judgment and placement, which is inefficient, labor-intensive, and results in poor consistency.

[0003] The ginger variety identification method based on color and morphological features proposed in Chinese patent CN110178481B is susceptible to changes in ambient light, surface contamination of ginger, and morphological diversity, resulting in insufficient accuracy and robustness. Furthermore, traditional mechanisms often employ stepper motors to drive single-axis rotation, which lacks flexibility and makes it difficult to achieve precise multi-degree-of-freedom posture adjustment.

[0004] In recent years, deep learning-based object detection and instance segmentation methods have shown significant advantages in agricultural vision tasks. However, they still face challenges in scenarios with small, multi-shaped, and heavily occluded targets, such as ginger seed buds. Moreover, existing methods mostly remain at the detection stage and have failed to form a complete closed-loop control system with the actuator.

[0005] Therefore, in view of the problems existing in the prior art, a method for bud recognition and orientation control that integrates a lightweight YOLOv8n detection model, SAM segmentation network, PCA orientation estimation and Delta parallel robotic arm to achieve full-process automation from visual perception to mechanical execution is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a method and system for intelligent identification and localization of ginger buds based on deep learning.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A deep learning-based intelligent identification and localization method for ginger bud scales includes the following steps:

[0009] Step 1: Data Acquisition and Augmentation

[0010] Ginger images were collected under various lighting conditions, backgrounds, and angles. Labelimg was used to annotate the bounding boxes and polygons of the buds and ginger bodies. Data augmentation methods were used to improve the generalization ability of the model.

[0011] Step 2: Training and Deployment of Lightweight Object Detection Model

[0012] A lightweight target detection architecture, YOLOv8n, was used to train a detection model for ginger body and ginger scales. Transfer learning was used for training, and cross-validation was used to evaluate the stability of the model. The trained model was then deployed on an embedded processing unit.

[0013] Step 3: Instance segmentation to obtain bud masks

[0014] The Segment Anything Model is used to finely segment the detected bud region and extract pixel-level masks.

[0015] Step 4: Camera Calibration and Coordinate Transformation

[0016] The camera intrinsic parameter matrix K and distortion coefficients are obtained using the Zhang Zhengyou calibration method, and the three-dimensional points in the camera coordinate system are determined.

[0017] Step 5: PCA and Orientation Disambiguation

[0018] PCA and orientation disambiguation are performed on the mask point set in the camera coordinate system to determine the directed unit vector. ;

[0019] Step 6: Hand-eye calibration and robotic arm coordinate system transformation

[0020] Perform a transformation from the camera coordinate system to the robot arm base coordinate system;

[0021] Step 7: Calculate the correction angle

[0022] The directed unit vector obtained in step five Transform to the robot arm base coordinate system and calculate the correction angle;

[0023] Step 8: Driving and Execution Phase

[0024] The actuator consists of a Delta robotic arm. When the robotic arm controller receives the ginger pose information and correction angle from the vision recognition unit, it plans a motion path based on the ginger pose information and controls three servo motors mounted on the base to synchronously drive the active arm to rotate. The active arm drives the passive link to move through the hinge, and then transmits the motion to the moving platform through the parallel link mechanism, so that it generates spatial displacement and accurately positions itself on top of the ginger to adsorb and fix the ginger. After the suction operation is completed, the moving platform drives the ginger to the target position under the drive of the servo motor.

[0025] The servo motor drives the end rotating shaft to rotate precisely according to the correction angle, and the pneumatic suction head adjusts the orientation of the ginger scales to be consistent with the preset direction.

[0026] The pneumatic suction head releases ginger and places it stably on the conveyor belt platform in a corrected posture, completing the automated orientation placement process.

[0027] Preferably, in the first step, random rotation, brightness and contrast adjustment, and noise augmentation methods are applied to improve the model's generalization ability.

[0028] Preferably, in the fourth step, the Zhang Zhengyou calibration method is used to obtain the camera intrinsic parameter matrix K and distortion coefficients; let the pixel point The depth at the corresponding pixel is Then the three-dimensional point in the camera coordinate system is .

[0029] Preferably, in the fifth step, after centering the mask, a covariance matrix is ​​constructed and its first principal vector is calculated to obtain a unit vector with an unspecified direction coinciding with the principal axis of the bud. , =1, the PCA principal axis is only defined as the straight line direction, which has a 180° sign ambiguity;

[0030] Calculate the body mass of ginger coordinates With the scaly bud core coordinates Form a directed centroid line vector Calculate the dot product Determine the sign of the principal axis by the dot product sign. ,vector It is a directed unit vector, representing the estimated direction from the base of the bud to the tip of the bud.

[0031] Preferably, in the sixth step, the transformation from the camera coordinate system to the robot arm base coordinate system is as follows: ,in, Let be a rotation matrix. It is a translation vector.

[0032] Preferably, in the seventh step, the directed unit vector obtained in step five is... Transform to the robot arm base coordinate system to obtain The current bud angle is calculated based on this vector. Set the target angle as Calculate the correction angle The angle range is (-180°, 180°), with counterclockwise rotation defined as positive and clockwise rotation as negative.

[0033] Preferably, in the eighth step, the Delta robotic arm includes a base, an active arm, a passive link, a moving platform, an end effector shaft, and a pneumatic suction head.

[0034] A deep learning-based intelligent identification and positioning system for ginger buds includes an RGB-D camera fixed on a platform, an embedded processing unit, and a Delta-type parallel robotic arm.

[0035] The present invention achieves the following technical effects compared to the prior art:

[0036] (1) By combining the lightweight deep learning target detection model YOLOv8n with the image segmentation technology SAM, this invention can achieve high-precision detection and localization of buds in complex backgrounds, effectively overcoming the limitations of traditional image processing methods in the face of morphological diversity and environmental interference.

[0037] (2) The present invention adopts a lightweight YOLOv8n model and deploys it on a high-performance embedded platform to ensure that the system meets the millisecond-level response requirements in real-time image processing, providing a reliable guarantee for high-speed mechanized operations;

[0038] (3) By seamlessly integrating visual recognition results with robotic arm control, this invention achieves full automation of the entire process from bud detection and pose calculation to angle correction, significantly reducing the need for manual intervention and improving work efficiency.

[0039] (4) The excellent robustness of this invention is reflected in its ability to adapt to complex environments such as changes in illumination, and the model generalization performance improved by data augmentation technology.

[0040] (5) Based on the PCA principal axis direction disambiguation algorithm and precise coordinate transformation method, this invention ensures the accuracy of scale bud angle correction and provides a reliable basis for subsequent ginger directional sowing. Attached Figure Description

[0041] Figure 1 This is a flowchart of the ginger bud identification and orientation correction method of the present invention;

[0042] Figure 2 This is a structural diagram of the Delta robotic arm of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the direction disambiguation of the present invention;

[0044] Figure 4 This is the output diagram of the calculation of the bud orientation angle of this invention;

[0045] Figure 5 This is a schematic diagram of the model recognition results of the present invention;

[0046] Figure 6 This is an overall view of the orientation device of the present invention;

[0047] Figure 7 This is a diagram showing the identification results and angle calculation results of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example 1:

[0050] like Figure 1 As shown, the system mainly includes an RGB-D camera fixed on a stand, an embedded processing unit (such as a high-performance computer), and a Delta-type parallel robotic arm; the end effector of the robotic arm is a negative pressure air suction head connected to a hollow stepper motor.

[0051] The system operates according to the following steps:

[0052] S1, a depth camera fixed above the platform, acquires RGB images and depth information of ginger within the working area in real time. The acquired visual information is immediately transmitted to the embedded processing unit. This unit has a pre-set and running ginger bud recognition model. This model simultaneously identifies the ginger body and buds in the image and outputs their respective bounding box information and confidence scores.

[0053] As attached Figure 2 As shown, the system successfully identified the ginger body (labeled as body) and the bulb bud (labeled as bud), providing a foundation for subsequent processing.

[0054] S2. After obtaining the bounding box, the system calls the Segment Anything Model to perform pixel-level segmentation on each identified region and generate an accurate mask.

[0055] S3. Principal component analysis (PCA) is performed on the segmented bud mask point set to extract its first principal component vector. This vector defines the principal axis direction of the bud, but there is a 180° directional ambiguity.

[0056] S4. To determine the unique positive direction (i.e. from the bud base to the bud tip), the system calculates the centroid (C_body) of the ginger body mask and the centroid (C_bud) of the bud mask.

[0057] Calculate the centroid vector V_centroid pointing from C_body to C_bud. Perform a dot product operation between the principal axis vector V_pca obtained from PCA and its reverse vector -V_pca, and V_centroid. The direction with a positive dot product result is determined as the positive direction of the principal axis.

[0058] S5. Based on the determined positive direction vector, calculate the current angle of the bud relative to the reference direction of the world coordinate system using the arctan2 function.

[0059] S6, the vision system will send the calculated three-dimensional coordinates of the ginger body's centroid and the scale bud correction angle directly to the Delta parallel robotic arm through the calibrated hand-eye relationship. The robotic arm will then move to the target position, and the end air suction head will complete the precise suction of the ginger.

[0060] S7. After adsorption is complete, the hollow stepper motor, which is coaxially connected to the air suction head, receives the Correction Angle command and precisely rotates the corresponding angle to correct the buds to the target orientation.

[0061] S8 Finally, the robotic arm moves the ginger, whose orientation has been corrected, to directly above the target placement point (such as the conveyor belt of a seeder), releases the negative pressure, and completes one work cycle.

[0062] Figure 3 This is a diagram illustrating the disambiguation of the direction of the scale bud. 1 is the positive direction of the principal axis; 2 is the ginger body; 3 is the centroid of the ginger body; 4 is the centroid vector pointing from the centroid of the ginger body to the centroid of the scale bud; 5 is the scale bud; 6 is the centroid of the scale bud; 7 is the principal axis of the PCA.

[0063] This diagram provides a more intuitive description of the bud orientation process. First, the SAM image segmentation model is used to obtain the masks of the ginger body and the buds. Based on the bud masks, PCA analysis is performed to obtain the 7pca principal axis. Although this axis can reflect the main direction of bud growth, there is a 180° directional ambiguity. To eliminate the ambiguity, the centroids of the ginger body (3) and the buds (6) are calculated, and the centroid vector (4) is obtained. Then, the dot product of this vector (4) with the two possible directions of the PCA principal axis is calculated, and the direction with a dot product result greater than zero is taken as the positive direction of the principal axis, that is, the actual growth direction of the buds. The basis for this is that the buds have a natural tendency to deviate from the centroid of the ginger body during the growth process.

[0064] Figure 4 This is the output diagram of the calculation of the bud orientation angle of the present invention. 1 is the growth direction of ginger buds; 2 is the centroid vector of the ginger body centroid pointing to the bud centroid; 3 is the ginger body centroid; 4 is the bud centroid.

[0065] To verify the feasibility of the ginger scale angle algorithm, a simulation experiment was conducted. Ginger image data was input into the algorithm, and a visualized result was output. The image retained the calculation process traces (1234) and output the current scale angle. In this experiment, the horizontal rightward direction of the image was defined as 0°, and 0° was assumed to be the target direction for calculating the correction angle.

[0066] Figure 5 This invention illustrates the model recognition results. It uses an improved YOLOv8 model to identify the ginger body and buds. The ginger body is labeled as "body," and the buds are labeled as "bud." A Gemini 335 camera is used to acquire ginger data in real time. After importing the data into the recognition model, the ginger body and buds are detected in real time, and the real-time recognition results are visualized.

[0067] like Figure 6 As shown in the figure, the entire experimental device platform, including the depth camera, processing computer, Delta robotic arm, and end effector, is clearly displayed, intuitively presenting the hardware configuration of the present invention.

[0068] like Figure 7 The figure shown is a diagram of the identification results and angle calculation results of the present invention.

[0069] Example 2:

[0070] This embodiment provides an engineered device for converting a benchtop verification system into a fully automated field seeding system.

[0071] While retaining the core recognition-orientation algorithm, this device has undergone adaptive improvements to the actuator and material conveying method to meet the needs of field operations.

[0072] The field device may include a tracked self-propelled chassis, a ginger seed supply frame supported on top of the chassis, a first adjustable-speed seeding conveyor belt connected to the discharge side of the frame, a vision recognition module mounted above the conveyor belt, a SCARA-type robotic arm cantilevered to the rear of the chassis, and a second conveyor belt independent of the suction operation area. The second conveyor belt may be configured within the operating range of the robotic arm to receive the oriented seed ginger and transport it to the seed furrow. A hollow stepper motor-driven negative pressure suction head may be configured at the end of the robotic arm.

[0073] The system operates according to the following steps:

[0074] S1, the ginger seed frame pre-stores seed ginger for planting. During operation, the first conveyor belt intermittently transports the ginger seeds in a single row to the visual inspection area. The visual recognition module uses a dustproof and waterproof RGB-D camera, which is fixed in place. When the ginger seeds arrive at the inspection area, image acquisition is triggered, obtaining RGB images and depth information, which are then transmitted to the built-in ginger scale and bud recognition model via the onboard processing unit. The model simultaneously outputs the bounding boxes and confidence scores of the ginger body and scales.

[0075] S2, after obtaining the bounding box, the system calls the instance segmentation model to generate a pixel-level mask.

[0076] S3. Principal component analysis (PCA) is performed on the bud mask point set to extract its first principal component vector. To improve robustness in complex field environments, the algorithm can add morphological screening conditions: when the mask slenderness ratio is less than a preset threshold, it is determined to be an invalid target, triggering a re-examination or rejection process.

[0077] S4, the direction disambiguation step employs the dual centroid vector method. In this embodiment, the calculation of the centroid of the ginger body C_body and the centroid of the bud C_bud can be corrected using depth information to compensate for coordinate deviations caused by the non-absolute horizontality of the conveyor belt surface. Specifically: the centroid vector V_centroid pointing from C_body to C_bud is calculated, and the PCA principal axis vector V_pca and its reverse vector -V_pca are respectively multiplied by V_centroid. The direction with a positive dot product result is determined as the positive direction of the principal axis. This calculation rule is consistent with the bench system.

[0078] S5, the correction angle calculation is based on the direction of the seeder's movement or the direction of the seed furrow, and the angle deviation of the current scale bud relative to the target orientation is obtained through the arctan2 function.

[0079] S6, the vision system sends the three-dimensional coordinates and correction angle of the ginger body to the SCARA robotic arm controller. Compared to the Delta robotic arm, the SCARA robotic arm has greater extension flexibility in the horizontal plane and a more compact structure, making it suitable for installation on space-constrained mobile chassis. The robotic arm moves to the suction position according to the path planning algorithm, and the end suction head completes the suction under negative pressure. The suction height and vacuum level can be adaptively adjusted according to the size and skin characteristics of the ginger.

[0080] S7. After successful adsorption, the hollow stepper motor drives the air suction head to rotate according to the Correction Angle command. This embodiment can be configured with an incremental encoder to achieve closed-loop angle feedback, ensuring that the correction accuracy meets agronomic requirements.

[0081] S8. After calibration, the robotic arm moves the seed ginger to the temporary storage position above the second conveyor belt. Negative pressure is released, allowing the seed ginger to fall smoothly onto the second conveyor belt. The second conveyor belt carries the oriented seed ginger synchronously backward to the planting position behind the furrow opener. The seed ginger falls into the furrow by its own weight or with the aid of a guide finger, completing the oriented placement. The operating speed of the second conveyor belt can be synchronized with that of the first conveyor belt to ensure coordinated operation.

[0082] This embodiment achieves autonomous movement in the field via a tracked chassis, uses dual conveyor belts for ginger seed supply and directional transport respectively, and optimizes the spatial layout with a SCARA robotic arm, forming a complete field directional sowing operation chain. Since the core recognition-orientation algorithm is derived from the same platform system, this embodiment only requires adaptation to conventional mechanical structures and parameter adjustments for implementation. It is expected to achieve the same orientation success rate as the platform system, meeting the efficiency requirements of mechanized field sowing.

[0083] The tracked chassis, conveyor belt, SCARA robotic arm, and other specific structures mentioned in this embodiment are conventional selections in the field, and their detailed design does not constitute an essential technical feature of the present invention.

[0084] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the technical scope of the present invention. Therefore, any minor modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for intelligent identification and localization of ginger bud scales based on deep learning, characterized in that, Includes the following steps: Step 1: Data Acquisition and Augmentation Ginger images were collected under various lighting conditions, backgrounds, and angles. Labelimg was used to annotate the bounding boxes and polygons of the buds and ginger bodies. Data augmentation methods were used to improve the generalization ability of the model. Step 2: Training and Deployment of Lightweight Object Detection Model A lightweight target detection architecture, YOLOv8n, was used to train a detection model for ginger body and ginger scales. Transfer learning was used for training, and cross-validation was used to evaluate the stability of the model. The trained model was then deployed on an embedded processing unit. Step 3: Instance segmentation to obtain bud masks The Segment Anything Model is used to finely segment the detected bud region and extract pixel-level masks. Step 4: Camera Calibration and Coordinate Transformation The camera intrinsic parameter matrix K and distortion coefficients are obtained using the Zhang Zhengyou calibration method, and the three-dimensional points in the camera coordinate system are determined. Step 5: PCA and Orientation Disambiguation PCA and orientation disambiguation are performed on the mask point set in the camera coordinate system to determine the directed unit vector. ; Step 6: Hand-eye calibration and robotic arm coordinate system transformation Perform a transformation from the camera coordinate system to the robot arm base coordinate system; Step 7: Calculate the correction angle The directed unit vector obtained in step five Transform to the robot arm base coordinate system and calculate the correction angle; Step 8: Driving and Execution Phase The actuator consists of a Delta robotic arm. When the robotic arm controller receives the ginger pose information and correction angle from the vision recognition unit, it plans a motion path based on the ginger pose information and controls three servo motors mounted on the base to synchronously drive the active arm to rotate. The active arm drives the passive link to move through the hinge, and then transmits the motion to the moving platform through the parallel link mechanism, so that it generates spatial displacement and accurately positions itself on top of the ginger to adsorb and fix the ginger. After the suction operation is completed, the moving platform drives the ginger to the target position under the drive of the servo motor. The servo motor drives the end rotating shaft to rotate precisely according to the correction angle, and the pneumatic suction head adjusts the orientation of the ginger scales to be consistent with the preset direction. The pneumatic suction head releases ginger and places it stably on the conveyor belt platform in a corrected posture, completing the automated orientation placement process.

2. The method for intelligent identification and localization of ginger buds based on deep learning according to claim 1, characterized in that, In the first step, random rotation, brightness and contrast adjustment, and noise augmentation methods are applied to improve the model's generalization ability.

3. The method for intelligent identification and localization of ginger buds based on deep learning according to claim 1, characterized in that, In the fourth step, the Zhang Zhengyou calibration method is used to obtain the camera intrinsic parameter matrix K and distortion coefficients; let the pixel point The depth at the corresponding pixel is Then the three-dimensional point in the camera coordinate system is .

4. The method for intelligent identification and localization of ginger buds based on deep learning according to claim 1, characterized in that, In the fifth step, after centering the mask, a covariance matrix is ​​constructed and its first principal vector is calculated to obtain a unit vector with an unspecified direction coinciding with the principal axis of the bud. , =1, the PCA principal axis is only defined as the straight line direction, which has a 180° sign ambiguity; Calculate the body mass of ginger coordinates With the scaly bud core coordinates Form a directed centroid line vector Calculate the dot product Determine the sign of the principal axis by the dot product sign. ,vector It is a directed unit vector, representing the estimated direction from the base of the bud to the tip of the bud.

5. The method for intelligent identification and localization of ginger scales based on deep learning according to claim 1, characterized in that, In the sixth step, the transformation from the camera coordinate system to the robot arm base coordinate system is as follows: ,in, For rotation matrix, It is a translation vector.

6. The method for intelligent identification and localization of ginger scales based on deep learning according to claim 1, characterized in that, In the seventh step, the directed unit vector obtained in step five will be... Transform to the robot arm base coordinate system to obtain The current bud angle is calculated based on this vector. Set the target angle as Calculate the correction angle The angle range is (-180°, 180°), with counterclockwise rotation defined as positive and clockwise rotation as negative.

7. The method for intelligent identification and localization of ginger buds based on deep learning according to claim 1, characterized in that, In the eighth step, the Delta robotic arm includes a base, an active arm, a passive link, a moving platform, an end effector, and a pneumatic suction head.

8. A deep learning-based intelligent identification and localization system for ginger bud scales, characterized in that, The system includes an RGB-D camera fixed on a stand, an embedded processing unit, and a Delta-type parallel robotic arm.

Images