Apple picking method and system based on fruit stem and calyx recognition and picking device
By identifying the fruit stem and navel in apple tree images, the posture category of the apple is determined and the grasping direction is adjusted, solving the problem of inaccurate apple picking in existing technologies and realizing non-damaging mechanical picking.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG AGRICULTURAL UNIVERSITY
- Filing Date
- 2023-04-03
- Publication Date
- 2026-06-16
AI Technical Summary
Current technology cannot accurately determine the apple's posture, making the fruit and tree susceptible to damage during mechanical harvesting.
By identifying the fruit stem and navel in apple tree images, a pre-trained recognition and detection model is used to determine the pose category of the target apple, and the grasping direction is determined based on the feature information. An end effector is then used for precise grasping.
It enables precise mechanical harvesting of apples, avoiding damage to the fruit and the tree.
Smart Images
Figure CN116343127B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, specifically to an apple grasping method, system, and picking equipment based on fruit stem and fruit navel recognition. Background Technology
[0002] Fruit tree cultivation is a key industry in my country, with the country ranking first in the world in both apple planting area and output, and this trend continues to grow steadily. While my country's agricultural mechanization is developing rapidly, the level of mechanization in fruit harvesting remains low. Apple harvesting is the most time-consuming and labor-intensive stage of the apple production cycle, characterized by strong seasonality, high labor intensity, and high labor costs.
[0003] Apples are single-fruited fruits that can be directly harvested. However, apple trees are densely packed with fruit of various shapes and sizes. Traditional apple harvesting is done manually, which is inefficient and the fruit is easily damaged. To improve apple harvesting, mechanical harvesting has begun to be used in traditional techniques. Chinese Patent CN112840862A discloses a harvesting robot and its method suitable for harvesting various fruits. The robot includes a vision device, a movement device, an execution device, and a collection device. Through the adjustable gripping space of its execution device and the reciprocating motion of a cylinder driving the instantaneous swing of a square rod, the robot uses this instantaneous swinging force to complete the harvesting action of multiple grippers. This invention, a harvesting robot and its method suitable for harvesting various fruits, can adapt to harvesting fruits of various sizes and effectively avoids damage to the fruit during the harvesting process by utilizing instantaneous swinging force.
[0004] However, during mechanized harvesting, the orientation of the target apple must be determined to pick the fruit in the correct gripping direction, preventing damage to the fruit and the tree. While the above-mentioned technical solutions enable mechanized fruit harvesting, they cannot determine the fruit's orientation, thus hindering precise fruit picking. Summary of the Invention
[0005] In view of this, this application provides an apple grasping method, system and picking device based on fruit stem and fruit navel recognition, in order to solve the problems in the prior art.
[0006] In a first aspect, embodiments of this application provide an apple grasping method based on fruit stem and fruit navel recognition, including:
[0007] A pre-trained recognition and detection model is used to identify apples and apple feature information in apple tree images, the feature information including the fruit stem and the fruit navel;
[0008] The target apple is the one closest to the end effector of the harvester.
[0009] The pose category of the target apple is determined based on the first feature information identified in the apple image.
[0010] The grasping direction of the target apple is determined based on the first feature information and the target apple image, and the grasping direction is used by the end effector to grasp the target apple.
[0011] In one possible implementation, training the recognition and detection model includes:
[0012] Sample images of apple trees during the harvest season were acquired using a high-resolution sample acquisition camera;
[0013] Use computer image annotation tools to annotate apples, stems, and navels in sample images to generate masks of apples, stems, and navels.
[0014] The yolact recognition and detection model was trained using the collected sample images and labeled masks.
[0015] In one possible implementation, determining the pose category of the target apple based on first feature information identified in the apple image includes:
[0016] The target apple and the first feature information in the apple image are detected and segmented to generate a mask of the target apple and the first feature;
[0017] If the fruit stalk and fruit navel are not detected in the mask of the first feature, then the posture category is the standard posture;
[0018] or,
[0019] If a fruit stalk or fruit navel is detected in the mask of the first feature, then the pose category is a non-standard pose.
[0020] In one possible implementation, if the pose category is a standard pose, determining the grasping direction of the target apple based on the first feature information and the target apple image includes:
[0021] Calculate the minimum outer rectangle of the target apple mask;
[0022] Determine the centerline of the short side of the smallest outer rectangle;
[0023] Determine any two points on the center line of the short side, taking the point closest to the ground as the starting point, and the direction towards the other point as the grabbing direction.
[0024] In one possible implementation, if the pose category is a non-standard pose, determining the grasping direction of the target apple based on the first feature information and the target apple image includes:
[0025] If a fruit stalk is detected in the mask of the first feature, then the minimum outer rectangle of the target apple mask is calculated;
[0026] Determine the first geometric center of the minimum outer rectangle and the second geometric center of the fruit stalk mask;
[0027] The direction from the first geometric center to the second geometric center is the grasping direction;
[0028] or,
[0029] If a fruit navel is detected in the mask of the first feature, then the minimum outer rectangle of the target apple mask is calculated;
[0030] Determine the first geometric center of the minimum outer rectangle and the third geometric center of the fruit navel mask;
[0031] The direction from the third geometric center to the first geometric center is the grasping direction.
[0032] Secondly, embodiments of this application provide an apple grasping system based on fruit stem and fruit navel recognition, including:
[0033] The acquisition module is used to identify apples and apple feature information in an apple tree image using a pre-trained recognition and detection model, wherein the feature information includes the fruit stem and the fruit navel;
[0034] The first determining module is used to identify the apple with the shortest distance to the end effector of the harvester as the target apple;
[0035] The second determining module is used to determine the pose category of the target apple based on the first feature information identified in the apple image.
[0036] The third determining module is used to determine the grasping direction of the target apple based on the first feature information and the target apple image, the grasping direction being used by the end effector to grasp the target apple.
[0037] Thirdly, embodiments of this application provide a harvesting device, including:
[0038] Controller;
[0039] End effector;
[0040] Memory;
[0041] And a computer program, wherein the computer program is stored in the memory, the computer program including instructions that, when executed by the controller, cause the picking device to perform the method described in any possible implementation of the first aspect, controlling the end effector to grasp the target apple in a determined grasping direction.
[0042] In this embodiment, the target apple's posture is classified based on the identified feature information, whether the identified feature information contains a stem or navel, and the specific grasping direction is determined based on the posture classification, thereby achieving precise mechanical harvesting of the apple and avoiding damage to the apple fruit and the tree. Attached Figure Description
[0043] Figure 1 A flowchart illustrating an apple-grabbing method based on stem and navel recognition provided in this application embodiment;
[0044] Figure 2 This is an example of image annotation provided in an embodiment of this application.
[0045] Figure 3 A schematic diagram of an apple in a standard pose provided for embodiments of this application;
[0046] Figure 4 A schematic diagram of an apple in a non-standard pose provided for an embodiment of this application;
[0047] Figure 5 A schematic diagram of the standard apple grasping direction provided for embodiments of this application;
[0048] Figure 6 A schematic diagram of a non-standard apple grasping direction provided for an embodiment of this application;
[0049] Figure 7 A schematic diagram of another non-standard apple grasping direction provided for an embodiment of this application;
[0050] Figure 8 A schematic diagram of an apple grasping system based on fruit stem and fruit navel recognition provided in an embodiment of this application;
[0051] Figure 9 A schematic diagram of the harvesting equipment provided in the embodiments of this application. Detailed Implementation
[0052] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0053] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0054] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.
[0055] Figure 1 A flowchart illustrating an apple-grabbing method based on stem and navel recognition, provided in this application embodiment, is shown below. Figure 1 The apple grasping method based on fruit stem and fruit navel recognition in this embodiment includes:
[0056] S101, a pre-trained recognition and detection model is used to identify apples and apple feature information in an apple tree image, the feature information including the fruit stem and the fruit navel.
[0057] In this embodiment, high-resolution sample acquisition cameras capture sample images of apple trees during the harvest season. Computer-aided image annotation tools are used to annotate the apples, stems, and navels in the sample images, generating masks for these elements. The captured sample images and the annotated masks are then used to train the YOLACT recognition and detection model. A harvesting robot's recognition camera captures apple tree images in real time, and the trained YOLACT recognition and detection model is used for real-time recognition and detection of apples, stems, and navels, followed by instance segmentation.
[0058] To ensure the accuracy of the recognition model's segmentation, the sample images should be collected in a standardized dwarf rootstock densely planted apple orchard under natural sunlight. The sample acquisition camera should be 1.2 to 1.5 meters away from the apple tree trunk to collect sample images, maintaining a distance similar to that between the recognition camera and the apple tree during the harvesting robot's operation. The apple trees for which the sample images are collected should be in the harvesting season, and the sample image collection time should cover 8:00 to 16:00, with a total of 500 images collected.
[0059] To ensure model recognition accuracy and reduce training time, the acquired image resolution was adjusted to 1050×700, and the image format was set to JPG. The Anaconda integrated environment was installed on the computer. A Labelme virtual environment was created and activated within Anaconda, and packages such as PyQt and Pillow were installed. Finally, the Labelme image annotation tool was installed, and Labelme was launched via Anaconda Prompt. The Labelme tool was used to annotate the acquired sample images, using apples, stems, and navels as labels. The pixel regions containing the apples, stems, and navels were annotated with polygonal regions exceeding four points, generating masks for the apples, stems, and navels. The apple mask is a pixel region containing the fruit body and navel but excluding the stem; the stem mask is the pixel region containing the stem; and the navel mask is the pixel region containing the navel. Figure 2 As shown.
[0060] The collected sample images and the labeled masks are divided into a training set and a test set. 400 sample images and their labeled masks are used as the training set, and 100 sample images and their labeled masks are used as the validation set. The Yolac recognition and detection model is trained using the training set and the validation set to learn the features of ripe apples, stems, and navels, and to learn the connections between features.
[0061] S102, the apple with the shortest distance to the end effector of the harvester is selected as the target apple.
[0062] The picking robot establishes three-dimensional coordinates in the picking area, and the depth camera of the picking robot acquires and identifies the three-dimensional coordinates of the apples, determining the apple with the shortest straight-line distance to the end effector as the target apple.
[0063] S103, determine the pose category of the target apple based on the first feature information of the target apple identified in the apple image.
[0064] In this embodiment, the YOLACT recognition and detection model is used to detect and segment the target apple and its first feature information in the apple image, generating a mask for the target apple and its first feature. If the stem and navel are not detected in the mask of the first feature, it indicates that the stem and navel of the target apple are not captured in the image in the two-dimensional plane direction of the image acquired by the camera, and the navel of the target apple should be on the side facing the ground in the two-dimensional plane direction. In this case, the pose category of the target apple is determined to be a standard pose, such as... Figure 3 The images shown are all of apples in their standard pose.
[0065] If the fruit stem or navel is detected in the mask of the first feature, it indicates that the fruit stem and navel of the target apple are not in the two-dimensional plane direction of the image captured by the camera. In this case, the two-dimensional plane direction of the navel of the target apple is not facing the ground. Therefore, the pose category of the target apple is determined to be a non-standard pose. Figure 4 All of these are apples in non-standard poses.
[0066] S104, determine the grasping direction of the target apple based on the first feature information and the target apple image, the grasping direction being used by the end effector to grasp the target apple.
[0067] In this embodiment, the direction of grasping the target apple needs to be determined according to the situation, which is divided into two cases: standard posture and non-standard posture.
[0068] When the pose type is standard pose, calculate the minimum outer rectangle of the target apple mask; determine the centerline of the short side of the minimum outer rectangle; determine any two points on the centerline of the short side, taking the point closest to the ground as the starting point, and the direction towards the other point as the grasping direction. See also Figure 5 , Figure 5 The apples in the picture are all apples with standard posture, Figure 5 Taking the leftmost apple as an example, the center line L of the short side is determined. Any two points A and B are defined on L, with A being the closest point. The direction from A to B is the grasping direction. When the harvesting robot is performing its harvesting operation, it should grasp the target apple from the near-ground end upwards along the direction from the center line A to B.
[0069] It should be noted that the apples used in this embodiment are from the experimental region and are characterized by a distinctly traditional apple shape. Therefore, the center line of the shorter side is used as a reference in this embodiment. Apples of foreign varieties, such as the Red Delicious, which are slender in shape, are not considered in this embodiment.
[0070] The above applies to a standard pose. If the pose is non-standard, such as when a fruit stem is detected in the mask of the first feature, then the minimum outer rectangle of the target apple mask is calculated; the first geometric center of the minimum outer rectangle and the second geometric center of the fruit stem mask are determined; the direction from the first geometric center to the second geometric center is the grasping direction. See also Figure 6 If the first geometric center is C and the second geometric center is D, then the grabbing direction is from C to D.
[0071] If a fruit navel is detected in the mask of the first feature, then the minimum outer rectangle of the target apple mask is calculated; the first geometric center of the minimum outer rectangle and the third geometric center of the fruit navel mask are determined; the direction from the third geometric center to the first geometric center is designated as the grasping direction. See also Figure 7 If the first geometric center is C and the third geometric center is E, then the grabbing direction is from E to C.
[0072] In the above situation, the grasping direction is the rotation direction of the target apple in the two-dimensional plane. When the end effector of the picking robot is performing the picking operation, it should grasp the target apple at one end of the geometric center point of the target apple mask along the rotation direction of the target apple.
[0073] Corresponding to the apple grasping method based on stem and navel recognition provided in the above embodiments, this application also provides an embodiment of an apple grasping system based on stem and navel recognition.
[0074] See Figure 8 An apple grasping system 20 based on fruit stem and fruit navel recognition includes:
[0075] The acquisition module 201 is used to identify apples and apple feature information in an apple tree image using a pre-trained recognition and detection model, wherein the feature information includes the fruit stem and the fruit navel.
[0076] The first determining module 202 is used to select the apple with the shortest distance to the end effector of the harvester as the target apple.
[0077] The second determining module 203 is used to determine the pose category of the target apple based on the first feature information of the target apple identified in the apple image.
[0078] The third determining module 204 is used to determine the grasping direction of the target apple based on the first feature information and the target apple image, the grasping direction being used by the end effector to grasp the target apple.
[0079] Corresponding to the above embodiments, this application also provides a harvesting device.
[0080] See Figure 9 The harvesting device 30 provided in this application embodiment includes: a controller 301, a memory 302, and an end effector 303. Those skilled in the art will understand that the harvesting device structure shown in the figures does not constitute a limitation on the embodiments of this application, and it may include more components than shown.
[0081] The controller 301 is the control center of the harvesting equipment. It connects various parts of the harvesting equipment through various interfaces and lines. It executes various functions of the harvesting equipment and / or processes data by running or executing software programs and / or modules stored in the memory 302 and calling data stored in the memory.
[0082] Memory 302 is used to store the execution instructions of controller 301. Memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0083] When the execution instructions in the memory 302 are executed by the controller 301, the picking device 30 is able to perform some or all of the steps in the above method embodiment, and control the end effector 303 to pick up the target apple in a determined grasping direction.
[0084] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. An apple grasping method based on fruit stem and fruit navel recognition, characterized in that, include: A pre-trained recognition and detection model is used to identify apples and apple feature information in apple tree images, the feature information including the fruit stem and the fruit navel; The target apple is the one closest to the end effector of the harvester. The pose category of the target apple is determined based on the first feature information identified in the apple tree image. The grasping direction of the target apple is determined based on the first feature information and the target apple image. This grasping direction is used by the end effector to grasp the target apple, including: If the pose category is a standard pose, the grasping direction of the target apple is determined based on the first feature information and the target apple image, including: Calculate the minimum outer rectangle of the target apple mask; determine the center line of the short side of the minimum outer rectangle; determine any two points on the center line of the short side, taking the point closest to the ground as the starting point and the direction towards the other point as the grabbing direction; If the pose category is a non-standard pose, the grasping direction of the target apple is determined based on the first feature information and the target apple image, including: If a fruit stem is detected in the mask of the first feature, the minimum outer rectangle of the target apple mask is calculated; the first geometric center of the minimum outer rectangle and the second geometric center of the fruit stem mask are determined; the direction from the first geometric center to the second geometric center is the grasping direction; If a fruit navel is detected in the mask of the first feature, the minimum outer rectangle of the target apple mask is calculated; the first geometric center of the minimum outer rectangle and the third geometric center of the fruit navel mask are determined; the direction from the third geometric center to the first geometric center is the grasping direction.
2. The apple grasping method based on fruit stem and fruit navel recognition according to claim 1, characterized in that, Training the recognition and detection model includes: Sample images of apple trees during the harvest season were acquired using a high-resolution sample acquisition camera; Use computer image annotation tools to annotate apples, stems, and navels in sample images to generate masks of apples, stems, and navels. The yolact recognition and detection model was trained using the collected sample images and labeled masks.
3. The apple grasping method based on fruit stem and fruit navel recognition according to claim 2, characterized in that, Determining the pose category of the target apple based on the first feature information identified in the apple image includes: The target apple and the first feature information in the apple image are detected and segmented to generate a mask of the target apple and the first feature; If the fruit stalk and fruit navel are not detected in the mask of the first feature, then the posture category is the standard posture; or, If a fruit stalk or fruit navel is detected in the mask of the first feature, then the pose category is a non-standard pose.
4. An apple grasping system based on fruit stem and navel recognition, characterized in that, include: The acquisition module is used to identify apples and apple feature information in an apple tree image using a pre-trained recognition and detection model, wherein the feature information includes the fruit stem and the fruit navel; The first determining module is used to identify the apple with the shortest distance to the end effector of the harvester as the target apple; The second determining module is used to determine the posture category of the target apple based on the first feature information identified by the target apple in the apple tree image; The third determining module is used to determine the grasping direction of the target apple based on the first feature information and the target apple image, the grasping direction being used by the end effector to grasp the target apple, including: If the pose category is a standard pose, the grasping direction of the target apple is determined based on the first feature information and the target apple image, including: Calculate the minimum outer rectangle of the target apple mask; determine the center line of the short side of the minimum outer rectangle; determine any two points on the center line of the short side, taking the point closest to the ground as the starting point and the direction towards the other point as the grabbing direction; If the pose category is a non-standard pose, the grasping direction of the target apple is determined based on the first feature information and the target apple image, including: If a fruit stem is detected in the mask of the first feature, the minimum outer rectangle of the target apple mask is calculated; the first geometric center of the minimum outer rectangle and the second geometric center of the fruit stem mask are determined; the direction from the first geometric center to the second geometric center is the grasping direction; If a fruit navel is detected in the mask of the first feature, the minimum outer rectangle of the target apple mask is calculated; the first geometric center of the minimum outer rectangle and the third geometric center of the fruit navel mask are determined; the direction from the third geometric center to the first geometric center is the grasping direction.
5. A harvesting device, characterized in that, include: Controller; End effector; Memory; And a computer program, wherein the computer program is stored in the memory, the computer program including instructions that, when executed by the controller, cause the picking device to perform the method of any one of claims 1 to 3, controlling the end effector to grasp the target apple in a determined grasping direction.