An apple picking mechanical hand assisted grabbing method based on key point detection
By training a YOLOv8-point convolutional neural network model for key point detection, the rotation angle and grasp size of the apple are determined, solving the problem of inaccurate positioning of the robotic arm during apple picking. This enables precise grasping by the robotic arm, improving picking efficiency and reducing fruit damage.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG AGRICULTURAL UNIVERSITY
- Filing Date
- 2023-09-15
- Publication Date
- 2026-07-03
AI Technical Summary
Existing mechanized apple picking equipment is prone to damaging apples when positioning is inaccurate. How to achieve accurate picking by robotic arms is an urgent problem to be solved.
By training a YOLOv8-point convolutional neural network model to detect key points, the key point information of the apple is obtained. A coordinate system is established to determine the rotation angle and gripping size of the robotic arm, and the robotic arm is controlled to perform precise gripping.
It improved the accuracy and efficiency of apple picking, reduced fruit damage, and enabled precise grasping by the robotic arm.
Smart Images
Figure CN117173244B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of apple picking technology, specifically to an apple picking robotic arm assisted grasping method based on key point detection. Background Technology
[0002] Apple harvesting is a crucial step in apple cultivation. The labor involved in apple harvesting accounts for two-thirds of the total labor in the entire production process, and manual harvesting is extremely labor-intensive and costly. For non-dwarf apple varieties, harvesters need to use ladders to reach the trees, and falls are a frequent occurrence.
[0003] Existing technologies utilize mechanized apple harvesting equipment to overcome the aforementioned problems, and mechanized harvesting can also improve the production efficiency of the apple industry and reduce labor costs. Mechanized harvesting generally requires a robotic arm to grasp the apples and control it to complete the harvesting process. However, if the apples are not accurately positioned during the harvesting process, they can be damaged.
[0004] Therefore, how to accurately control the robotic arm to accurately pick apples is a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0005] In order to solve the above-mentioned technical problems, this application proposes the following technical solution:
[0006] In a first aspect, embodiments of this application provide a method for apple-picking robotic arms to assist in grasping based on key point detection, including:
[0007] A keypoint detection model was obtained by training the YOLOv8-point convolutional neural network model using a dataset of orchard apple images under natural conditions.
[0008] The key point detection model is used to detect key points in apple images to obtain key point information for each fruit.
[0009] A coordinate system is established with the initial position of the robot arm as the rotation zero point. The rotation angle and gripping size of the robot arm are determined based on the position of the key points in the coordinate system.
[0010] The robotic arm is controlled to rotate and open to grasp the apple based on the rotation angle and the grasping size.
[0011] In one possible implementation, the step of training a YOLOv8-point convolutional neural network model using a dataset of orchard apple images in their natural state to obtain a keypoint detection model includes:
[0012] Collect a predetermined number of apple fruit images under natural lighting conditions;
[0013] The collected images were labeled using the Labelme annotation tool. The key points to be labeled include three labels: fruit stem, fruit navel, and fruit shoulder. The fruit shoulder corresponds to two key points.
[0014] The apple fruit image was augmented sequentially by geometric transformation, color space transformation and pixel manipulation, which increased the complexity of the background and the posture of the fruit, thus avoiding overfitting of the data.
[0015] Using the augmented dataset, we trained the YOLOv8-point convolutional neural network model.
[0016] In one possible implementation, the step of detecting keypoint information for each fruit in the apple image using the keypoint detection model includes:
[0017] The trained YOLOv8-point keypoint detection model is used to detect keypoint information for each fruit in an apple image, including the navel keypoint, two shoulder keypoints, and stem keypoint.
[0018] The position of the robotic arm's grasping fingertips is determined by key points.
[0019] In one possible implementation, the establishment of the coordinate system uses the initial position of the robotic arm as the rotation zero point. The rotation angle and grasping size of the robotic arm are determined based on the positions of the key points in the coordinate system. This includes: assuming the initial position of the robotic arm is the rotation zero point in the coordinate system, if three key points—the fruit stem, the fruit navel, and the fruit shoulder—are detected, and the pixel coordinates of these three key points are (x1, y1); (x2, y2); and (x3, y3) respectively, then: the rotation angle of the robotic arm... Preparing to grasp the size of the robotic arm
[0020] In one possible implementation, the establishment of the coordinate system uses the initial position of the robotic arm as the rotation zero point. The rotation angle and grasping size of the robotic arm are determined based on the positions of the key points in the coordinate system. This includes: assuming the initial position of the robotic arm is the rotation zero point in the coordinate system, if two fruit stem key points are detected, the coordinates of the two key points are (x... A ,y A ) and (x B ,y B Let α be the angle between the line connecting the initial positions of the two fingers on the robotic arm and the line connecting the two key points of the fruit shoulder, and let D be the actual distance between the two key points of the fruit shoulder.
[0021] or,
[0022] If the fruit shoulder key point (x) is detected A ,y A ) and key points of the fruit stalk (x C ,y C Given that the angle β between the line connecting the two key points of the fruit shoulder and the line connecting the key points of the fruit shoulder and stem on the robot arm in the initial position is known, and taking the line connecting the two key points as the hypotenuse of a triangle, with the midpoint of the hypotenuse as the center of rotation, and using the initial position of the robot arm as the reference for calculating the rotation angle, we take... The distance between the two key points is taken as the pre-grasping size D of the robotic arm, and the following is calculated.
[0023] or,
[0024] Only the fruit stalk key point (x) was detected C ,y C ) and the key point of the fruit navel (x D ,y D Let the angle between the line connecting the two key points and the line connecting the fruit stalk and the fruit navel at the initial position of the robot be taken as the rotation angle α. The actual distance between the two key points is the pre-grasping size D of the robotic arm. The result is...
[0025] or,
[0026] Only one fruit shoulder keypoint (x) was detected A ,y A ) and the key point of the fruit navel (x D ,y D Given that the angle β between the line connecting the fruit navel and fruit stalk key points on the robot arm and the line connecting the fruit shoulder and fruit navel key points in the preset initial position is also known, and taking the line connecting the two key points as the hypotenuse of a triangle, with the midpoint of the hypotenuse as the center of rotation, and taking the initial position of the robot arm as the reference for calculating the rotation angle, we take... The distance between the two key points is taken as the pre-grasping size D of the robotic arm, and the following is calculated.
[0027] In one possible implementation, controlling the rotation and opening of the robotic arm to grasp the apple based on the rotation angle and the grasping size includes: transmitting two parameter information, the determined rotation angle α and the robotic arm's intended grasping size D, to the robotic arm control device, and using the control device to control the rotation and opening of the robotic arm to grasp the apple.
[0028] Secondly, embodiments of this application provide an apple-picking robotic arm-assisted grasping system based on key point detection, including:
[0029] The model training module is used to train the YOLOv8-point convolutional neural network model using a dataset of orchard apple images in natural conditions to obtain a keypoint detection model.
[0030] The key point detection module is used to detect key point information corresponding to each fruit in the apple image through the key point detection model.
[0031] The grasping parameter determination module is used to establish a coordinate system with the initial position of the robot as the rotation zero point, and to determine the rotation angle and grasping size of the robot based on the position of the key points in the coordinate system.
[0032] The grasping control module is used to control the robotic arm to rotate and open to grasp the apple according to the rotation angle and grasping size.
[0033] Thirdly, embodiments of this application provide a device, including: a Delta robotic arm, with a three-jaw drive manipulator and a servo motor respectively disposed at both ends of the Delta robotic arm, the three-jaw drive manipulator and the servo motor being connected by a transmission rod, and the controller of the servo motor executing the method described in any possible implementation of the first aspect to realize the picking of apples.
[0034] Fourthly, embodiments of this application provide a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method described in any possible implementation of the first aspect.
[0035] In this embodiment, a trained keypoint detection model is used to detect keypoint information of the apple, thereby determining different rotation angles and gripping sizes for the robotic arm. This achieves accurate acquisition of the apple's keypoints and precise gripping of the robotic arm following the apple's posture. This improves the accuracy and efficiency of apple harvesting and reduces fruit damage. Attached Figure Description
[0036] Figure 1 A flowchart illustrating an apple-picking robotic arm-assisted grasping method based on key point detection, provided for an embodiment of this application;
[0037] Figure 2 This is a schematic diagram of the location of key points of the apple target provided in an embodiment of this application;
[0038] Figure 3 A schematic diagram illustrating the determination of the apple's posture baseline under four occlusion conditions provided in the embodiments of this application;
[0039] Figure 4 A schematic diagram illustrating the rotation of key points (posture) of an apple followed by a robotic arm, provided in an embodiment of this application.
[0040] Figure 5 A schematic diagram of an apple-picking robotic arm-assisted grasping system based on key point detection provided in this application embodiment;
[0041] Figure 6 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation
[0042] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.
[0043] See Figure 1 The apple-picking robotic arm-assisted grasping method based on key point detection provided in this embodiment includes:
[0044] S101, a key point detection model is obtained by training the YOLOv8-point convolutional neural network model using a dataset of orchard apple images under natural conditions.
[0045] In this embodiment, a predetermined number of apple fruit images were acquired under natural lighting conditions. The acquired images were labeled using the Labelme annotation tool. The key points of the annotations included three labels: the stem, the navel, and the shoulder. The shoulder corresponds to two key points, such as... Figure 2 As shown, the apple fruit image was augmented sequentially through geometric transformation, color space transformation, and pixel manipulation, increasing the complexity of the background and the posture of the fruit, thus avoiding overfitting. The augmented dataset was then used to train a YOLOv8-point convolutional neural network model.
[0046] S102, the key point detection model is used to detect key point information corresponding to each fruit in the apple image.
[0047] The trained YOLOv8-point keypoint detection model is used to detect keypoint information for each apple in an image, including the navel keypoint, two shoulder keypoints, and stem keypoint; the keypoints are then used to determine the gripping fingertip position of the robotic arm.
[0048] S103, Establish a coordinate system with the initial position of the robot as the rotation zero point, and determine the rotation angle and gripping size of the robot based on the position of the key points in the coordinate system.
[0049] In this embodiment, there are multiple scenarios. If three key points—the fruit stem, the fruit navel, and the fruit shoulder—are detected, and the pixel coordinates of these three key points are (x1, y1); (x2, y2); and (x3, y3) respectively, then: the rotation angle of the robotic arm... Preparing to grasp the size of the robotic arm
[0050] The above describes the scenario where all keypoints are detected. If the apple is obscured by leaves, different combinations of keypoints will be detected, such as... Figure 3 As shown.
[0051] Case 1: If two key points of the fruit stalk are detected, and the coordinates of the two key points are (x... A ,y A ) and (x B ,y B Let α be the angle between the line connecting the initial positions of the two fingers on the robotic arm and the line connecting the two key points of the fruit shoulder, and let D be the actual distance between the two key points of the fruit shoulder.
[0052] Case 2, if the fruit shoulder key point (x) is detected A ,y A ) and key points of the fruit stalk (x C ,y C Given that the angle β between the line connecting the two key points of the fruit shoulder and the line connecting the key points of the fruit shoulder and stem on the robot arm in the initial position is known, and taking the line connecting the two key points as the hypotenuse of a triangle, with the midpoint of the hypotenuse as the center of rotation, and using the initial position of the robot arm as the reference for calculating the rotation angle, we take... The distance between the two key points is taken as the pre-grasping size D of the robotic arm, and the following is calculated.
[0053] Case 3: Only the fruit stalk key point (x) was detected. C ,y C ) and the key point of the fruit navel (x D ,y D Let the angle between the line connecting the two key points and the line connecting the fruit stalk and the fruit navel at the initial position of the robot be taken as the rotation angle α. The actual distance between the two key points is the pre-grasping size D of the robotic arm. The result is...
[0054] Case 4: Only one fruit shoulder keypoint (x) is detected. A ,y A ) and the key point of the fruit navel (x D ,y D Given that the angle β between the line connecting the fruit navel and fruit stalk key points on the robot arm and the line connecting the fruit shoulder and fruit navel key points in the preset initial position is also known, and taking the line connecting the two key points as the hypotenuse of a triangle, with the midpoint of the hypotenuse as the center of rotation, and taking the initial position of the robot arm as the reference for calculating the rotation angle, we take... The distance between the two key points is taken as the pre-grasping size D of the robotic arm, and the following is calculated.
[0055] S104, the robotic arm is controlled to rotate and open to grasp the apple according to the rotation angle and the grasping size.
[0056] The determined rotation angle α and the pre-grabbing size D of the robot arm are transmitted to the robot arm control device, which then controls the rotation and opening of the robot arm to grasp the apple.
[0057] like Figure 4 As shown, Figure 4 This includes the robotic arm's pre-grabbing position and the key point positions of the fruit detected by the model. Using the key point detection model, the rotation angle α and the pre-grabbing size for each apple are calculated. Servo motors enable the robotic arm to rotate precisely to the corresponding angle, ready for the picking action. Based on the pre-grabbing size, the robotic arm predicts the required gripping force, thus achieving precise apple grasping.
[0058] In this embodiment, a vision module monitors the real-time situation during the grasping process, such as whether the apple has been successfully grasped. The success signal is transmitted to the robotic arm control system, enabling real-time control and adjustment of the robotic arm to ensure the safety and stability of the harvesting process. The key point detection and grasping performance during the harvesting process are evaluated, and the model and algorithm are optimized and improved based on the actual results. The training dataset is continuously optimized to improve the accuracy and stability of the key point detection model.
[0059] Corresponding to the key point detection-based apple picking robot assisted grasping method provided in the above embodiments, this application also provides an embodiment of a key point detection-based apple picking robot assisted grasping system.
[0060] See Figure 5 The apple-picking robotic arm-assisted grasping system 20 based on key point detection provided in this embodiment specifically includes:
[0061] The model training module 201 is used to train the YOLOv8-point convolutional neural network model using a dataset of orchard apple images in natural conditions to obtain a key point detection model.
[0062] The key point detection module 202 is used to detect key point information corresponding to each fruit in the apple image through the key point detection model;
[0063] The grasping parameter determination module 203 is used to establish a coordinate system with the initial position of the robot as the rotation zero point, and to determine the rotation angle and grasping size of the robot based on the position of the key points in the coordinate system.
[0064] The grasping control module 204 is used to control the robotic arm to rotate and open to grasp the apple according to the rotation angle and grasping size.
[0065] See Figure 6 This embodiment also provides a device, including: Delta robotic arm 1, with a three-jaw drive manipulator 2 and a servo motor 3 respectively provided at both ends of the Delta robotic arm 1. The three-jaw drive manipulator 2 and the servo motor 3 are connected by a transmission rod. The controller of the servo motor 3 executes the method described in the above embodiment to realize the picking of apples.
[0066] Due to the limited flexibility of Delta arm 1, a three-jaw driven robotic arm 2 with a rotation mechanism (servo motor driven) was designed to ensure that the robotic arm has flexible grasping ability and can accurately grasp apples of different sizes and postures. Delta robotic arm 1 is responsible for positioning and movement, while the three-jaw driven robotic arm 2 is responsible for the actual apple grasping operation.
[0067] In this embodiment, the YOLOv8-point convolutional neural network model is used to detect key point information of the apple, thereby determining the rotation angle between the apple's posture and the robotic arm's pre-grabbing posture. Simultaneously, the fruit size is sent to the robotic arm to determine the pre-grabbing size. A servo motor controls the robotic arm to rotate to the same posture as the target, and the Delta robotic arm moves to the target position before grasping the fruit. Then, the servo motor controls the robotic arm to rotate 360° to separate the fruit stem from the branch, completing the harvesting process.
[0068] Corresponding to the above embodiments, this application also provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, wherein when the program runs, it can control the device where the computer-readable storage medium is located to execute some or all of the steps in the above method embodiments. Specifically, the computer-readable storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0069] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0070] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.
Claims
1. A method for assisting apple picking with a robotic arm based on key point detection, characterized in that, include: A keypoint detection model was obtained by training the YOLOv8-point convolutional neural network model using a dataset of orchard apple images under natural conditions. The keypoint detection model is used to detect keypoints in apple images to obtain keypoint information for each fruit, including: The trained YOLOv8-point keypoint detection model is used to detect keypoint information for each fruit in an apple image, including the navel keypoint, two shoulder keypoints, and stem keypoint. The position of the robotic arm's grasping fingertips is determined by key points; A coordinate system is established with the robot's initial position as the rotation zero point. The rotation angle and gripping size of the robot are determined based on the positions of the key points in the coordinate system. This includes: assuming the robot's initial position is the rotation zero point in the coordinate system, if two shoulder key points are detected, the coordinates of the two key points are... and The rotation angle is defined as the angle between the line connecting the initial positions of the two fingers on the robotic arm and the line connecting the key points of the two fruit shoulders. The actual distance between the two key points on the shoulder is the size of the robotic arm's pre-grab size. , ; ; or, If the key point of the fruit shoulder is detected and fruit stalk key points Given that the initial coordinates of the line connecting the two fingertips of the robotic arm grasping the fruit shoulder in the preset initial position are connected to the line connecting the key points of the fruit shoulder and stem, with an angle β, and considering the line connecting the two key points as the hypotenuse of a triangle, and the midpoint of the hypotenuse as the center of rotation, the initial position of the robotic arm is taken as the reference for calculating the rotation angle. The distance between the two key points is used as the pre-grabbing size of the robotic arm. Seeking ; ; or, Only the key points of the fruit stalk were detected. and the key point of the fruit navel The angle between the line connecting the two key points and the central axis of symmetry of the robot arm at its initial position is taken as the rotation angle. ,Pick The actual distance between the two key points is the size at which the robotic arm is prepared to grasp. Seeking ; ; or, Only one fruit shoulder keypoint was detected. Key points of the fruit navel The angle between the line connecting the robotic arm's fruit navel grasping fingertip and the rotation center point of the fruit stem in the preset initial position, and the line connecting the key points of the fruit shoulder and fruit navel, is also [value missing]. Given that the line connecting the two key points is the hypotenuse of a triangle, and the midpoint of the hypotenuse is the center of rotation, and the initial position of the robot is taken as the reference for calculating the rotation angle, then... The distance between the two key points is used as the pre-grabbing size of the robotic arm. Seeking ; ; The robotic arm is controlled to rotate and open to grasp apples based on the rotation angle and the size of the grasp.
2. The apple-picking robotic arm-assisted grasping method based on key point detection according to claim 1, characterized in that, The keypoint detection model is obtained by training the YOLOv8-point convolutional neural network model using a dataset of orchard apple images under natural conditions, including: Collect a predetermined number of apple fruit images under natural lighting conditions; The collected images were labeled using the Labelme annotation tool. The key points to be labeled include three labels: fruit stem, fruit navel, and fruit shoulder. The fruit shoulder corresponds to two key points. The apple fruit image was augmented sequentially by geometric transformation, color space transformation and pixel manipulation, which increased the complexity of the background and the posture of the fruit, thus avoiding overfitting of the data. Using the augmented dataset, we trained the YOLOv8-point convolutional neural network model.
3. The apple-picking robotic arm-assisted grasping method based on key point detection according to claim 1, characterized in that, The establishment of the coordinate system uses the initial position of the robotic arm as the rotation zero point. The rotation angle and grasping size of the robotic arm are determined based on the positions of the key points in the coordinate system. This includes: assuming the initial position of the robotic arm is the rotation zero point in the coordinate system, if three key points—the fruit stem, the fruit navel, and the fruit shoulder—are detected, their pixel coordinates are as follows: ; ; Then: the rotation angle of the robotic arm The size of the robotic arm to be grasped .
4. The apple-picking robotic arm-assisted grasping method based on key point detection according to claim 3, characterized in that, The process of controlling the robotic arm to rotate and open to grasp the apple based on the rotation angle and the grasping size includes: determining the rotation angle... and the size of the robotic arm's pre-grab Two parameter information are transmitted to the robot arm control device, which then controls the robot arm's rotation and opening / closing to grasp the apple.
5. An apple-picking robotic arm-assisted grasping system based on key point detection, characterized in that, include: The model training module is used to train the YOLOv8-point convolutional neural network model using a dataset of orchard apple images in natural conditions to obtain a keypoint detection model. The keypoint detection module is used to detect keypoint information for each fruit in an apple image using a keypoint detection model, including: The trained YOLOv8-point keypoint detection model is used to detect keypoint information for each fruit in an apple image, including the navel keypoint, two shoulder keypoints, and stem keypoint. The position of the robotic arm's grasping fingertips is determined by key points; The grasping parameter determination module is used to establish a coordinate system, taking the robot's initial position as the rotation zero point. Based on the positions of key points in the coordinate system, it determines the robot's rotation angle and grasping size. This includes: assuming the robot's initial position is the rotation zero point in the coordinate system, if two shoulder key points are detected, the coordinates of the two key points are... and The rotation angle is defined as the angle between the line connecting the initial positions of the two fingers on the robotic arm and the line connecting the key points of the two fruit shoulders. The actual distance between the two key points on the shoulder is the size of the robotic arm's pre-grab size. , ; ; or, If the key point of the fruit shoulder is detected and fruit stalk key points The angle between the initial coordinates of the line connecting the two fingertips of the robotic arm grasping the fruit shoulder in the preset initial position and the line connecting the key points of the fruit shoulder and stem is... Given that the line connecting the two key points is the hypotenuse of a triangle, and the midpoint of the hypotenuse is the center of rotation, and the initial position of the robot is taken as the reference for calculating the rotation angle, then... The distance between the two key points is used as the pre-grabbing size of the robotic arm. Seeking ; ; or, Only the key points of the fruit stalk were detected. and the key point of the fruit navel The angle between the line connecting the two key points and the central axis of symmetry of the robot arm at its initial position is taken as the rotation angle. ,Pick The actual distance between the two key points is the size at which the robotic arm is prepared to grasp. Seeking ; ; or, Only one fruit shoulder keypoint was detected. Key points of the fruit navel The angle between the line connecting the robotic arm's fruit navel grasping fingertip and the rotation center point of the fruit stem in the preset initial position, and the line connecting the key points of the fruit shoulder and fruit navel, is also [value missing]. Given that the line connecting the two key points is the hypotenuse of a triangle, and the midpoint of the hypotenuse is the center of rotation, and the initial position of the robot is taken as the reference for calculating the rotation angle, then... The distance between the two key points is used as the pre-grabbing size of the robotic arm. Seeking ; ; The gripping control module is used to control the robotic arm to rotate and open to grip the apple based on the rotation angle and gripping size.
6. A device, characterized in that, include: The Delta robotic arm has a three-jaw drive manipulator and a servo motor at each end. The three-jaw drive manipulator and the servo motor are connected by a transmission rod. The controller of the servo motor executes the method described in any one of claims 1-4 to realize the picking of apples.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 4.