Method for identifying growth postures of pineapples in field environment
By constructing a pineapple key point data set and a pineapple recognition method based on the yolov7 network model, and combining the depth information of the ZED camera to perform pineapple position detection, the problem that existing pineapple picking equipment is difficult to accurately identify pineapple growth postures, realizing accurate identification and lossless picking of pineapples, and improving the quality of pineapples and the income of growers.
Patent Information
- Application Number
- CN202411725570.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-03-11
AI Technical Summary
Existing pineapple picking equipment is difficult to accurately identify the growth posture of pineapples, which makes it easy to damage the pineapple during picking, affecting the quality of pineapples and the income of growers.
A method of identifying pineapple growth posture in a field environment is adopted. By constructing a pineapple key point data set, pineapple identification and key point positioning are performed based on the yolov7 network model, and pineapple position detection and calculation are carried out in combination with the depth information of the ZED camera to achieve accurate identification and lossless picking of pineapples.
It has achieved accurate identification and lossless picking of pineapples, improved the quality of pineapples and the income of growers, and accelerated the mechanization process of the pineapple industry.
Smart Images
Figure CN119600359A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of pineapple posture recognition, and in particular to a method for recognizing the growth posture of a pineapple in a field environment. Background Art
[0002] Pineapple, also known as pineapple, is one of the representative tropical fruits. In recent years, the increasing demand has led to a continuous increase in the pineapple planting area, and the industry scale has achieved a rapid snowball expansion. In recent years, the domestic pineapple production scale, planting area and overall output have continued to expand and increase. However, the pineapple industry model and overall development are still immature. Although some links of field mechanization have made certain progress, due to the lack of harvesting equipment in the pineapple field, the labor efficiency is often low and the personnel cost is high during the harvest period, which directly causes the imbalance of supply and demand in the pineapple industry, and then leads to a sharp decline and fluctuation in some prices, forming a serious unsalable situation. Therefore, the "inorganic availability" in the harvesting link is the fundamental reason for the obstruction of the development of the pineapple industry. Therefore, the mechanized picking of pineapples can effectively promote the development of the pineapple industry.
[0003] At present, some large-scale pineapple planting areas use some smart devices for picking, but these smart devices are prone to damage the pineapples when picking, and cannot accurately identify the growth posture of the pineapples. After picking, there will be many low-quality and damaged pineapples, which seriously affect the quality of the pineapples and reduce the income costs of the growers. Therefore, realizing accurate identification and lossless picking of pineapples can promote the development of the pineapple planting industry. Summary of the invention
[0004] The present invention provides a method for identifying the growth posture of pineapples in a field environment, which can accurately locate and identify the growth posture of pineapples, and on this basis, non-destructive pineapple picking can be achieved in conjunction with picking equipment.
[0005] To achieve the above object, the present invention adopts the following technical solution:
[0006] A method for identifying the growth posture of pineapples in a field environment, the method comprising the following steps:
[0007] A. Constructing a pineapple key point dataset: Use a pineapple image acquisition device to collect pineapple images, clean the collected pineapple images, and then design pineapple instance category labels, pineapple key points and labels on the pineapple images. Then, use a dataset filtering algorithm to check the images and annotations to obtain a correctly labeled pineapple key point dataset.
[0008] B. Pineapple recognition model: Based on the pineapple key point dataset and the yolov7 network model, the key point category prediction branch and the key point coordinate regression branch are added to the coordinate box prediction branch and the pineapple category prediction branch. The recognition of pineapples in images and the positioning of pixel coordinates as well as the recognition and positioning of pineapple key points are achieved by training the deep learning model.
[0009] C. Pineapple pose detection and calculation in field environment: The pineapple recognition model is used to identify the pineapple and its key point coordinates in the ZED camera image, and then the key point depth information convolution filtering algorithm and the pineapple pose Euler angle calculation algorithm are used to detect and calculate the pineapple pose. Finally, the depth information of the ZED camera is combined to select the graspable pineapple, and the rotation Euler angle of the pineapple relative to the camera plane is calculated to obtain the grasping pose of the pineapple relative to the camera coordinate system.
[0010] Preferably, the pineapple key points marked and designed in step A are, in sequence, the center position of the top of the pineapple crown, the middle point where the root of the pineapple crown meets the top of the pineapple body, the left edge position of the middle fruit body of the pineapple body, the right edge position of the middle fruit body of the pineapple body, and the bottom center position of the pineapple body. The pineapple whose first four unobstructed pineapple key points are clearly visible at the same time is marked and classified as pineapple-all, and the remaining pineapple key points are blocked pineapples, and their marked classification is represented by pineapple-occlusion.
[0011] Preferably, the steps of the data set filtering algorithm in step A are as follows: (1) obtaining an image and its annotation as well as label information of a pineapple instance; (2) determining whether the type of the pineapple instance is a designed label: if yes, proceeding to the next step, and then determining whether its coordinate point is ≥ 2; if no, indicating that the coordinate point is less than 2, then the labeled image instance is wrong, and the wrong image is transferred; if yes, proceeding to the next step; if no, indicating that the pineapple instance is not a designed label, then determining whether it is a key point label; if no, indicating that the annotation is wrong, then transferring the wrong image; if yes, indicating that it is a key point, proceeding to the next step, determining whether its coordinate point is 1; if no, indicating that the key point annotation is wrong; (3) storing relevant label information, then filtering out the pineapple category information in the label, and performing the next step of processing; (4) obtaining the annotation box information and the instance ID according to the pineapple category label in the label; (5) determining whether the pineapple label and the key point annotation are consistent with the definition according to the definition of pineapple-all and pineapple-occlusion and their relationship with the key point, combined with the key point information of the ID number instance, if consistent, indicating that the annotation is correct; if inconsistent, indicating that the annotation is wrong, then transferring the wrong image.
[0012] Preferably, the steps of the key point depth information convolution filtering algorithm in step C are: (1) calculating the coordinate information of the key point according to the pixel coordinates of the pineapple key point in combination with the convolution algorithm; (2) obtaining the 9 pixel coordinates of each key point based on the convolution kernel; (3) obtaining the three-dimensional coordinate information of the 9 pixel coordinates using the correspondence between the pixel coordinates of the ZED camera and the three-dimensional point cloud; (4) sorting the depth distance information of each key point; (5) filtering the sorted depth distance information using the majority value filtering algorithm, and obtaining the value z0 with the highest probability of occurrence as the final depth distance of the key point; (6) constructing the three-dimensional coordinate information (x, y, z0) of the pineapple key point.
[0013] Preferably, the steps of the graspable pineapple pose algorithm in step C are: (1) opening the ZED camera to obtain the color image of the left camera and its corresponding depth point cloud information; (2) inputting the color image into the pineapple detection yolov7-pineapple model, and outputting all detected pineapples and their pixel coordinate information and key point coordinate information; (3) based on the detected pineapples and their pixel coordinates and key point coordinates, the key point depth information convolution algorithm is used to filter the depth information of the neck point; (4) based on the filtered neck point depth information, the pineapple closest to the camera is rotated to calculate the pose information of the pineapple; (6) the pose calculation model is used to calculate the pose information of the selected pineapple
[0014] Compared with the prior art, no report on the definition of pineapple key points and key point filtering algorithm has been found so far. Secondly, the beneficial effect of the present invention is that it can realize the three-dimensional spatial positioning of pineapple instances and realize the accurate calculation of pineapple posture based on pineapple key points while ensuring the real-time calculation. On this basis, it can realize lossless pineapple picking in cooperation with picking equipment, thereby accelerating the progress of the whole process of pineapple planting. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 Construct a framework diagram for the pineapple key point data set of the present invention;
[0016] Figure 2 It is a diagram of the data set filtering algorithm of the present invention;
[0017] Figure 3 It is a schematic diagram of pineapple recognition model training of the present invention;
[0018] Figure 4 It is a flow chart of pineapple posture calculation of the present invention;
[0019] Figure 5 This is a diagram of the key point depth information convolution filtering algorithm of the present invention;
[0020] Figure 6 It is a schematic diagram of the convolution calculation of the key point pixel coordinates of the present invention;
[0021] Figure 7 It is a schematic diagram of the key point convolution of the present invention;
[0022] Figure 8 It is a schematic diagram of pineapple categories and key point markings of the present invention;
[0023] Fig. 9 A schematic diagram for designing the labeling rules of the present invention;
[0024] Fig.10 It is a schematic diagram of the key point design of the pineapple of the present invention. DETAILED DESCRIPTION
[0025] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0026] It should be noted that when a component / part is referred to as being "set on" another component / part, it may be directly set on the other component / part or there may also be an intermediate component / part. When a component / part is referred to as being "connected / coupled" to another component / part, it may be directly connected / coupled to the other component / part or there may be an intermediate component / part at the same time. The term "connection / coupling" as used herein may include electrical and / or mechanical physical connections / couplings. The term "includes / comprising" as used herein refers to the presence of a feature, step or component / part, but does not exclude the presence or addition of one or more other features, steps or components / parts. The term "and / or" as used herein includes any and all combinations of one or more relevant listed items.
[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application.
[0028] See also Figures 1 to 10 As shown, a method for identifying the growth posture of pineapples in a field environment comprises the following steps:
[0029] A. Constructing a pineapple key point dataset: Use a pineapple image acquisition device to collect pineapple images, clean the collected pineapple images, and then design pineapple instance category labels, pineapple key points and labels on the pineapple images. Then, use a dataset filtering algorithm to check the images and annotations to obtain a correctly labeled pineapple key point dataset.
[0030] B. Pineapple recognition model: Based on the pineapple key point dataset and the yolov7 network model, the key point category prediction branch and the key point coordinate regression branch are added to the coordinate box prediction branch and the pineapple category prediction branch. The recognition of pineapples in images and the positioning of pixel coordinates as well as the recognition and positioning of pineapple key points are achieved by training the deep learning model.
[0031] C. Pineapple pose detection and calculation in field environment: The pineapple recognition model is used to identify the pineapple and its key point coordinates in the ZED camera image, and then the key point depth information convolution filtering algorithm and the pineapple pose Euler angle calculation algorithm are used to detect and calculate the pineapple pose. Finally, the depth information of the ZED camera is combined to select the graspable pineapple, and the rotation Euler angle of the pineapple relative to the camera plane is calculated to obtain the grasping pose of the pineapple relative to the camera coordinate system.
[0032] The yolov7 network model and ZED camera are existing technologies and are already well-known technologies or equipment. I will not go into details here. If you have any questions, you can check them online.
[0033] The present invention needs to locate the positions of different pineapples in the image and then infer their three-dimensional coordinate information based on the depth information. Therefore, it belongs to the target detection task at the image recognition level. Secondly, due to the demand for accurate and lossless pineapple picking, it is necessary to estimate the position information of the pineapple and the boundary area between the pineapple body and the pineapple crown. Therefore, key points that are easy to distinguish are designed. The five key points are represented by head, neck, left, right, and bottom, which are significant and can significantly represent the pineapple. Figure 8 As shown;
[0034] Head: the center of the top of the pineapple crown;
[0035] Neck: The midpoint where the root of the pineapple crown meets the top of the pineapple fruit.
[0036] left: the left edge of the middle fruit body of the pineapple;
[0037] right: the right edge of the middle fruit body of the pineapple;
[0038] Bottom: The bottom center of the pineapple fruit.
[0039] Based on points 1 (head), 2 (neck) or 2 (neck), 3 (left), 4 (right), the offset of the pineapple from the vertical direction can be estimated, thereby estimating the approximate position of the pineapple, which is convenient for the robot to grasp.
[0040] In the process of pineapple category design, according to the actual grasping environment, the effective working distance of the manipulator and the grasping steps, priority is given to grasping the complete pineapple individuals in the field of view. Therefore, two auxiliary subcategories are designed based on pineapples, and the occlusion and unocclusion labels are marked. The pineapples with the first four unobstructed pineapple key points that are clearly visible at the same time are labeled as pineapple-all, and the remaining pineapple key points are occluded pineapples, and their labeling category is pineapple-occlusion, and they are labeled using the open source labelme software.
[0041] like Figure 1 As shown in , in order to obtain high-quality labeled data for pineapples, the labeling rules are designed as follows:
[0042] (i) Pineapples that are far away in the image and still blurry after being magnified do not need to be labeled, but all others need to be labeled;
[0043] (ii) Regardless of whether the pineapple is complete or blocked, as long as it is clearly visible to the human eye, it must be labeled;
[0044] (3) Pineapples that are too dark or too blurry to be quickly identified will not be marked;
[0045] (iv) It is not marked when the human eye cannot tell that it is a pineapple without inference;
[0046] (V) All visible key points of the pineapple need to be marked and named according to the defined category names;
[0047] (6) The coordinates of the key points must be within their marked boxes;
[0048] (VII) Pineapples are divided into two categories: pineapple-all and pineapple-occlusion. The key point category names are: head, neck, left, right, and bottom, with a total of 5 key points.
[0049] See also Figure 2 As shown in the figure, different people's cognition often leads to incorrect pineapple labeling during the labeling process. Therefore, in order to filter out the wrong labels and improve the quality of the data set, the present invention proposes a pineapple data set filtering algorithm for screening out incorrect labeling information.
[0050] The steps of the data set filtering algorithm are as follows: Step 1 is to obtain the image and its annotation as well as the label information label of the pineapple instance; Step 2 is to determine whether the type of the label label is the designed label. If yes, determine whether the coordinate point is ≥ 2 (if yes, proceed to the next step, if no, it indicates that the pineapple instance in the image is incorrectly labeled, then the wrong image is transferred); if no, determine whether it is a key point label (if no, it indicates an incorrect labeling, then the wrong image is transferred, if yes, it indicates that it is a key point, then proceed to the next step); Step 3 is to store the relevant label information (such as labels, ids, axis_data), then filter out the pineapple category information in labels, and proceed to the next step; Step 4 is to obtain the annotation box information and the instance id according to the pineapple category label in labels; Step 5 is based on the definition of pineapple-all (head, neck, left, right are visible at the same time) and the rest are pineapple-occlusion. Combined with the key point information of the id numbered instance, determine whether the pineapple label and the key point annotation are consistent with the definition. If consistent, it indicates that the annotation is correct. If inconsistent, it indicates that the annotation is incorrect, then the wrong image is transferred.
[0051] See also Figure 3 As shown in the figure, the pineapple recognition model is mainly based on the yolov7 network model. On the basis of the coordinate box prediction branch and the pineapple category prediction branch, the key point category prediction branch and the key point coordinate regression branch are added. The learning mathematical model of the key point coordinate regression prediction branch is shown in formulas (1) to (2). Formula (1) is mainly aimed at improving the ability of the yolov7-pineapple network to deal with small and medium range errors of pineapple key points during training. The s in formula (2) i and s' i are the predicted key point coordinate information and the real key point coordinate information respectively. Formula (3) is the key point visible category loss function.
[0052]
[0053] Loss L (s)=∑ i wing(s i -s' i ) (2)
[0054] L point_cls =-(ylogp(x)+(1-y)log(1-p(x))) (3)
[0055] The pineapple detection algorithm is mainly based on the ZED depth camera to realize the acquisition of image and three-dimensional point cloud information. The images acquired by the ZED camera are mainly used for pineapple recognition and detection, and the three-dimensional point cloud map is mainly used for pineapple posture calculation and depth distance information acquisition. The pineapple detection algorithm in the field environment is mainly composed of the image acquisition algorithm of the key point data set, the pineapple recognition model, the key point depth information convolution filtering algorithm, the pineapple posture calculation algorithm, etc.
[0056] The key point depth information convolution filtering algorithm is mainly used to calculate the depth information of the key points. At the same time, it can reduce the interference of branch and leaf occlusion on the depth information and improve the accuracy of depth distance estimation. Secondly, it can greatly reduce the amount of calculation and improve the efficiency of the algorithm. It mainly targets the current situation in which the single-point depth distance acquisition in the process of key point depth distance acquisition is affected by light, branch and leaf occlusion and other factors and has large errors.
[0057] See also Figures 5 to 7 As shown in the figure, the steps of the key point depth information convolution filtering algorithm are: (1) calculating the coordinate information of the key point according to the pixel coordinates of the pineapple key point combined with the convolution algorithm; (2) obtaining the 9 pixel coordinates of each key point based on the convolution kernel; (3) obtaining the three-dimensional coordinate information of the 9 pixel coordinates by using the correspondence between the pixel coordinates of the ZED camera and the three-dimensional point cloud; (4) sorting the depth distance information of each key point; (5) filtering the sorted depth distance information using the majority value filtering algorithm, and obtaining the value z0 with the highest probability of occurrence as the final depth distance of the key point; (6) constructing the three-dimensional coordinate information (x, y, z0) of the pineapple key point.
[0058] See also Figure 4 As shown in the figure, the Euler angle calculation algorithm for the pineapple posture is mainly used to calculate the Euler deflection angle and depth three-dimensional coordinates of the pineapple. The steps of the pineapple posture algorithm are as follows: (1) Open the ZED camera to obtain the color image of the left camera and its corresponding depth point cloud information; (2) Use the color image as input, send it to the pineapple detection yolov7-pineapple model, and output all detected pineapples and their pixel coordinate messages and key point coordinate information; (3) Based on the detected pineapples and their pixel coordinates and key point coordinates, the key point depth information convolution algorithm is used to filter the depth information of the neck point; (4) Based on the filtered neck point depth information, the pineapple closest to the camera is rotated to calculate the posture information of the pineapple; (5) Calculate the posture information of the selected pineapple.
[0059] The calculation of the Euler angle of the pineapple posture is based on the grasping requirements of the manipulator. The present invention calculates the rotation angle around the z-axis and the deflection angle around the x-axis based on the Euler angle coordinate system, and the calculation formulas are shown below (4) to (6). Formula (4) is used to calculate the rotation angle around the z-axis, and γ in the formula represents the threshold of the difference between the head point and the neck depth distance. When it is less than the threshold, the head and neck are used to calculate the rotation angle around the z-axis, otherwise the middle point between the left and right points is used to calculate the angle; formulas (5) to (6) are mainly used to calculate the Euler rotation angle around the x-axis.
[0060]
[0061] The present invention can accurately locate and identify the growth posture of pineapples, and on this basis, in conjunction with picking equipment, can achieve non-destructive picking of pineapples.
[0062] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A method for identifying the growth posture of pineapples in a field environment, characterized in that: The identification method comprises the following steps: Construct a pineapple key point dataset: Use a pineapple image acquisition device to collect pineapple images, clean the collected pineapple images, and then design pineapple instance category labels, pineapple key points and labels on the pineapple images. Then use a dataset filtering algorithm to check the images and annotations to obtain a correctly labeled pineapple key point dataset. Pineapple recognition model: Based on the pineapple key point dataset and the yolov7 network model, the key point category prediction branch and key point coordinate regression branch are added to the coordinate box prediction branch and the pineapple category prediction branch. The deep learning model is trained to realize the recognition of pineapples in images, the positioning of pixel coordinates, and the recognition and positioning of pineapple key points. Pineapple pose detection and calculation in field environment: The pineapple recognition model is used to identify the pineapples and their key point coordinates in the ZED camera image, and then the key point depth information convolution filtering algorithm and the pineapple pose Euler angle calculation algorithm are used to detect and calculate the pineapple pose. Finally, the depth information of the ZED camera is combined to select the graspable pineapples, and the rotation Euler angle of the pineapple relative to the camera plane is calculated to obtain the grasping pose of the pineapple relative to the camera coordinate system.
2. The method for identifying pineapple growth posture under field environment according to claim 1, characterized in that, The pineapple key points marked and designed in step A are, in sequence, the center position of the top of the pineapple crown, the middle point where the root of the pineapple crown meets the top of the pineapple body, the left edge position of the middle fruit body of the pineapple body, the right edge position of the middle fruit body of the pineapple body, and the bottom center position of the pineapple body. The pineapples with the first four unobstructed pineapple key points clearly visible at the same time are marked and classified as pineapple-all, and the remaining pineapple key points are blocked pineapples, and their marked classification is represented by pineapple-occlusion.
3. The method for identifying pineapple growth posture under field environment according to claim 1, characterized in that, The steps of the data set filtering algorithm in step A are as follows: (i) obtaining the image and its annotation and the label information of the pineapple instance; (ii) determining whether the type of the pineapple instance is the designed label: if yes, proceed to the next step, and then determine whether its coordinate point is ≥ 2. If no, it indicates that the coordinate point is less than 2, then the labeled image instance is wrong, and the wrong image is transferred. If yes, proceed to the next step; if no, it indicates that the pineapple instance is not the designed label, then determine whether it is a key point label. If no, it indicates that the annotation is wrong, and the wrong image is transferred. If yes, it indicates that it is a key point, then proceed to the next step, and determine whether its coordinate point is 1. At this time, if no, it indicates that the key point is incorrectly labeled; (iii) Store the relevant label information, then filter out the pineapple category information in the label and proceed to the next step of processing; (iv) Obtain the annotation box information and instance ID based on the pineapple category label in the label; (v) Based on the definitions of pineapple-all and pineapple-occlusion and their relationship with key points, combined with the key point information of the ID numbered instance, determine whether the pineapple label and key point annotation are consistent with the definition. If they are consistent, it indicates that the annotation is correct. If they are inconsistent, it indicates that the annotation is wrong, and the erroneous image is transferred.
4. The method for identifying pineapple growth posture under field environment according to claim 1, characterized in that, The steps of the key point depth information convolution filtering algorithm in step C are: (i) calculating the coordinate information of the key point according to the pixel coordinates of the pineapple key point in combination with the convolution algorithm; (ii) obtaining the 9 pixel coordinates of each key point based on the convolution kernel; (iii) obtaining the three-dimensional coordinate information of the 9 pixel coordinates using the correspondence between the pixel coordinates of the ZED camera and the three-dimensional point cloud; (iv) sorting the depth distance information of each key point; (v) filtering the sorted depth distance information using the majority value filtering algorithm, and obtaining the value z0 with the highest probability of occurrence as the final depth distance of the key point; and (vi) constructing the three-dimensional coordinate information of the pineapple key point.
5. The method for identifying pineapple growth posture under field environment according to claim 1, characterized in that, The steps of the graspable pineapple pose algorithm in step C are: (i) open the ZED camera to obtain the color image of the left camera and its corresponding depth point cloud information; (ii) input the color image into the pineapple detection yolov7-pineapple model, and output all detected pineapples and their pixel coordinate messages and key point coordinate information; (iii) based on the detected pineapples and their pixel coordinates and key point coordinates, the key point depth information convolution algorithm is used to filter the depth information of the neck point; (iv) based on the filtered neck point depth information, the pineapple closest to the camera is rotated to calculate the pose information of the pineapple; (v) the pose calculation model is used to calculate the pose information of the selected pineapple.