A key point auxiliary-based tea tree new shoot picking position detection method

Abstract

The application discloses a tea shoot picking position detection method based on key point assistance, and specifically comprises the following steps: collecting tea bud images in real time and inputting the images into a key point detection model Mask R-CNN to predict tea bud key points; according to the types and quantity of the tea bud key points identified by the key point detection model, combining bud leaf growth characteristic parameters, selecting different picking position detection schemes to determine the picking position of the tea buds, and the picking position detection schemes are divided into four categories: overall non-shielding condition, bud and leaf vertex non-shielding condition, bud and leaf connection point non-shielding condition, and bud and leaf vertex and connection point partial shielding condition. According to the types and quantity of the tea bud key points identified by the key point detection model, combining the bud leaf growth characteristic parameters, and selecting different picking position detection schemes to determine the picking position of the tea buds, the precision and efficiency of the tea bud picking position detection under the shielding condition are greatly improved, and accurate picking is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent harvesting technology, specifically relating to a method for detecting the harvesting position of tea tree shoots based on key point assistance. Background Technology

[0002] In robotic intelligent harvesting, the location of buds and leaves is obtained through visual perception technology; target identification and localization are fundamental to harvesting. In recent years, vision-based automated harvesting robots have been used for harvesting premium teas, but the automatic identification and localization of their harvesting locations has become a key limitation and challenge to their development. Tea buds and leaves are small, densely distributed, and exhibit diverse postures, leading to visual interference issues such as overlapping buds and leaves and shading by branches and leaves. Furthermore, their soft texture makes them easily disturbed and highly susceptible to environmental factors such as light and wind speed. Moreover, the harvesting time for premium teas is tight and demanding; therefore, an effective harvesting solution is currently lacking. To achieve rapid identification and localization of harvesting locations and ensure the efficiency and high quality requirements of mechanized harvesting of premium teas, it is necessary to develop a method for acquiring harvesting location information for premium teas. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides a method for detecting the picking location of tea shoots based on key point assistance, so as to achieve rapid and accurate identification and positioning of tea picking location.

[0004] The method for detecting the picking position of tea shoots based on key point assistance includes the following steps:

[0005] S1. Real-time acquisition of tea bud images and input of the images into the Mask R-CNN keypoint detection model for tea bud and leaf keypoint prediction;

[0006] S2. Based on the key point detection model, identify the type and number of key points of tea buds and leaves. Combined with the growth characteristic parameters of buds and leaves, select different picking location detection schemes to determine the picking location of tender buds. There are four types of picking location detection schemes: 1) No overall obstruction; 2) Bud and leaf apex not obstructed; 3) Bud and leaf connection point not obstructed; 4) Bud and leaf apex and connection point partially obstructed.

[0007] Furthermore, the key point types of tea tree buds and leaves include bud tip A, leaf tip B, one bud and one leaf connection point C, and one bud and two leaves connection point D; the picking positions of the tender buds include the picking position P1 for one bud and one leaf and the picking position P2 for one bud and two leaves.

[0008] Furthermore, for the case of 1) overall unobstructed view, first calculate the distance d between the two points based on the positions of the connection point C (one bud and one leaf) and the connection point D (one bud and two leaves). CDThe predicted picking position for one bud and one leaf is determined based on the confidence level of the two connection points, either P1 or P2. That is, if the confidence level of the connection point C for one bud and one leaf is greater than that of the connection point D for one bud and two leaves, the predicted picking position is P1 for one bud and one leaf; otherwise, the predicted picking position is P2 for one bud and two leaves.

[0009] Furthermore, for case 2) where the bud and leaf apex are not obscured, a prediction model is established based on the relationships between bud apex A, leaf apex B, the connection point between one bud and one leaf C, and the connection point between one bud and two leaves D, specifically including three cases:

[0010] If the connection point C between a bud and a leaf is blocked, then a linear equation is established based on the bud apex A and the connection point D between a bud and two leaves, or based on the leaf apex B and the connection point D between a bud and two leaves to predict the picking position P2 of a bud and two leaves.

[0011] If the connection point D between one bud and two leaves is blocked, then a linear equation is established based on the bud apex A and the connection point C between one bud and one leaf, or based on the leaf apex B and the connection point C between one bud and one leaf to predict the picking position P1 of one bud and one leaf.

[0012] If both the one-bud-one-leaf connection point C and the one-bud-two-leaf connection point D are obscured, then prediction will not be made for the time being.

[0013] Furthermore, regarding case 3) where the bud-leaf junction is not occluded, three cases are identified based on the number of occluded vertices:

[0014] If the bud tip A is blocked and the leaf tip B is not blocked, then a straight line equation is established based on the leaf tip B and the connection point C between the bud and the leaf to predict the picking position P1 of the bud and the leaf, or a straight line equation is established based on the connection point C between the bud and the leaf and the connection point D between the bud and the leaf to predict the picking position P1 of the bud and the leaf and the picking position P2 of the leaf.

[0015] If the leaf apex B is obscured and the bud apex A is not obscured, then a straight line equation is established based on the connection point C of one bud and one leaf and the connection point D of one bud and two leaves to predict the picking position P1 of one bud and one leaf and the picking position P2 of one bud and two leaves.

[0016] If both bud apex A and leaf apex B are obscured, then a linear equation is established based on the connection point C between one bud and one leaf and the connection point D between one bud and two leaves to predict the picking position P1 for one bud and one leaf and the picking position P2 for one bud and two leaves.

[0017] Furthermore, regarding the case where the bud tip and connecting point are partially obscured (4), there are four specific scenarios:

[0018] If the leaf apex B and the connection point D between one bud and two leaves are obscured, then a straight line equation is established based on the apex A of the bud and the connection point C between one bud and one leaf to predict the picking position P1 of one bud and one leaf.

[0019] If the leaf apex B and the connection point C between one bud and one leaf are obscured, then a straight line equation is established based on the apex A of the bud and the connection point D between one bud and two leaves to predict the picking position P2 of one bud and two leaves.

[0020] If the bud apex A and the connection point D between the bud and two leaves are obscured, then a straight line equation is established based on the leaf apex B and the connection point C between the bud and one leaf to predict the picking position P1 of the bud and one leaf.

[0021] If the bud apex A and the connection point C between the bud and the leaf are obscured, then a linear equation is established based on the leaf apex B and the connection point D between the bud and the leaf to predict the picking position P1 of the bud and leaf.

[0022] Compared with the prior art, the present invention has the following advantages: Based on the key point detection model, the present application identifies the type and number of key points of tea buds and leaves, and combines the growth characteristic parameters of buds and leaves to select different picking position detection schemes to determine the picking position of tender buds, which greatly improves the accuracy and efficiency of tea bud and leaf picking position detection under shading conditions and achieves precise picking. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the overall process of the harvesting location detection method of the present invention.

[0024] Figure 2 This is a schematic diagram defining the key points and harvesting locations of new shoot buds and leaves in this invention. The key points specifically include the bud tip, leaf tip, one bud and one leaf connection point, and one bud and two leaf connection point.

[0025] Figure 3 This is an overall schematic diagram of the harvesting location detection method of the present invention. The "√" in the diagram indicates that the part is not obscured.

[0026] Figure 4 This invention presents a completely unobstructed view of the new shoots and key points.

[0027] Figure 5 The diagram shows the situation where the bud and leaf tips are not obscured. (1) The connection point C of one bud and one leaf is obscured; (2) The connection point D of one bud and two leaves is obscured; (3) Both the connection point C of one bud and one leaf and the connection point D of one bud and two leaves are obscured.

[0028] Figure 6 The following is a schematic diagram of the new shoot connection point not being blocked: (1) the bud tip A is blocked and the leaf tip B is not blocked; (2) the leaf tip B is blocked and the bud tip A is not blocked; (3) both the bud tip A and the leaf tip B are blocked.

[0029] Figure 7The diagram shows the situation where the tips and connecting points of the new shoot buds and leaves are partially obscured. (1) The leaf tip B and the connecting point D of one bud and two leaves are obscured. (2) The leaf tip B and the connecting point C of one bud and one leaf are obscured. (3) The bud tip A and the connecting point D of one bud and two leaves are obscured. (4) The bud tip A and the connecting point C of one bud and one leaf are obscured. Detailed Implementation

[0030] To enable those skilled in the art to more clearly understand the technical solution of the present invention, the following description, in conjunction with the accompanying drawings, further illustrates a method for detecting the picking position of tea shoots based on key point assistance.

[0031] like Figures 1-3 As shown, a keypoint-assisted method for detecting the picking location of tea shoots includes the following steps: First, real-time images of tea buds are acquired using an industrial camera, and the images are input into the MaskR-CNN keypoint detection model for predicting keypoints of tea buds and leaves. The model outputs the predicted keypoints and their corresponding confidence scores. The keypoints of tea buds and leaves include bud vertex A, leaf vertex B, one-bud-one-leaf connection point C (i.e., the connection point between bud and leaf), and one-bud-two-leaf connection point D (i.e., the connection point between one-bud-two-leaf and stem). Then, based on the type and number of keypoints of tea buds and leaves identified by the keypoint detection model, and combined with the bud and leaf growth characteristic parameters, different picking location detection schemes are selected to determine the picking location of the tender buds. This method enables the estimation of the picking location of buds and leaves even when they are partially obscured, thus achieving accurate picking. The specific harvesting location detection scheme is divided into four types based on the different degrees of bud and leaf shading: First, no overall shading; second, bud and leaf tips not shading; third, bud-leaf connection point not shading; and fourth, bud and leaf tips and connection point partially shading. The harvesting locations for tender buds include location P1 (one bud and one leaf) and location P2 (one bud and two leaves).

[0032] like Figure 4 As shown, for the first scenario, where there is no overall obstruction, the distance d between the two points is first calculated based on the positions of the connection point C (one bud and one leaf) and the connection point D (one bud and two leaves). CD The predicted picking position for one bud and one leaf, P1, or one bud and two leaves, is determined based on the confidence level of the two connection points. That is, if the confidence level of the one bud and one leaf connection point C is greater than the confidence level of the one bud and two leaves connection point D, the predicted picking position is one bud and one leaf, P1; otherwise, the predicted picking position is one bud and two leaves, P2.

[0033] It is understandable that, for an overall unobstructed situation, the harvesting positions P1 (one bud and one leaf) and P2 (one bud and two leaves) are predicted based on the straight line CD (i.e., the connection point C between one bud and one leaf and the connection point D between one bud and two leaves). In a specific embodiment, if the confidence level of the connection point C between one bud and one leaf is greater than the confidence level of the connection point D between one bud and two leaves, then 0.5 * d is taken down from the connection point C between one bud and one leaf. CD The distance is used to predict the picking position P1 for one bud and one leaf; otherwise, the picking position P2 for one bud and two leaves is predicted by taking 5mm downward from the connection point D of one bud and two leaves (based on the growth parameters of the bud and leaves and the picking standard).

[0034] like Figure 5 As shown, for the second scenario, where the bud and leaf apex are not obscured, a prediction model is established based on the relationships between bud apex A, leaf apex B, the connection point between one bud and one leaf C, and the connection point between one bud and two leaves D. This model includes three specific cases: Figure 5 As shown in (1), if the connection point C between a bud and a leaf is blocked, the picking position P2 of a bud and two leaves can be predicted by establishing a straight line equation using the two-point formula based on either the bud apex A and the connection point D between a bud and two leaves, or the leaf apex B and the connection point D between a bud and two leaves. Figure 5 As shown in (2), if the connection point D between a bud and two leaves is blocked, a straight line equation is established based on the bud apex A and the connection point C between a bud and one leaf, or based on the leaf apex B and the connection point C between a bud and one leaf to predict the picking position P1 of a bud and one leaf; Figure 5 As shown in (3), if both the one-bud-one-leaf connection point C and the one-bud-two-leaf connection point D are blocked, then prediction will not be made for the time being.

[0035] It is understandable that, for cases where the bud and leaf apex are not obscured, different analyses are performed depending on the scenario: if the connection point C between a bud and a leaf is obscured, the harvesting position P2 of the bud and two leaves is predicted based on the straight line AD (i.e., the two points between the bud apex A and the connection point D between the bud and two leaves) or the straight line BD (i.e., the two points between the leaf apex B and the connection point D between the bud and two leaves); in a specific embodiment, such as Figure 5 (1) The equation of the straight line established according to point AD is y = -5.88*x + 74.07 (the equation of the straight line in the xoy coordinate system in the figure, and the equation of the straight line in the following text is the same). The equation of the straight line established according to point BD is y = 7.66*x - 82.44, in mm. After establishing the equation, take a position about 4 mm down according to the equation of AD or BD (based on the growth parameters of the bud and leaves and the picking standard, the same below) as the picking position. If the connection point D of one bud and two leaves is blocked, predict the picking position P1 of one bud and one leaf based on the straight lines AC (i.e., the two points of bud apex A and one bud and one leaf connection point C) and BC (i.e., the two points of leaf apex B and one bud and one leaf connection point C); in a specific embodiment, such as Figure 5(2) The equation of the straight line established based on point AC is y = -3.7x + 63.52, and the equation of the straight line established based on point BC is y = 8.05x - 88.84, both in mm. After establishing the equations, a position approximately 4 mm below the point based on the equation of line AC or line BC is taken as the picking position. If both the connection point C (one bud and one leaf) and the connection point D (one bud and two leaves) are obscured, no prediction is made for the time being.

[0036] like Figure 6 As shown, for the third case, where the bud-leaf connection point is not occluded, there are three cases based on the number of occluded vertices: such as Figure 6 As shown in (1), if the bud apex A is obscured and the leaf apex B is not obscured, then a straight line equation is established based on the leaf apex B and the connection point C between the bud and the leaf to predict the picking position P1 of the bud and the leaf, or a straight line equation is established based on the connection point C between the bud and the leaf and the connection point D between the bud and the leaf to predict the picking positions P1 and P2 of the bud and the leaf; Figure 6 As shown in (2), if the leaf apex B is obscured and the bud apex A is not obscured, then a straight line equation is established based on the connection point C between one bud and one leaf and the connection point D between one bud and two leaves to predict the picking position P1 of one bud and one leaf and the picking position P2 of one bud and two leaves; Figure 6 As shown in (3), if both the bud apex A and the leaf apex B are blocked, then... Figure 6 Similar to (2), a linear equation is established based on the connection point C of one bud and one leaf and the connection point D of one bud and two leaves to predict the picking position P1 of one bud and one leaf and the picking position P2 of one bud and two leaves.

[0037] It is understandable that, for cases where the bud-leaf connection point is not obstructed, different analyses are performed depending on the scenario: if the bud apex A is obstructed and the leaf apex B is not obstructed, the picking position P1 for a bud-leaf and the picking position P2 for a bud-two-leaf are predicted based on line BC (i.e., the leaf apex B and the connection point C between a bud and a leaf) or line CD (i.e., the connection point C between a bud and a leaf and the connection point D between a bud and two leaves); in a specific embodiment, Figure 6 The equation of the straight line established according to point BC is y = 8.05*x - 88.84, and the equation of the straight line established according to point CD is y = 6.7*x - 71.35, in mm. After establishing the equations, a position approximately 4 mm downwards is taken as the picking position. If the leaf apex B is obscured and the bud apex A is not obscured, the picking position P1 for one bud and one leaf and the picking position P2 for one bud and two leaves are predicted based on the straight line CD (i.e., the connection point C between one bud and one leaf and the connection point D between one bud and two leaves). In a specific embodiment, Figure 6The equation of the straight line established according to point CD is y = 6.7*x - 71.35, in mm. After establishing the equation, take a position about 4 mm below point C or D along the straight line as the picking position. If both the bud apex A and the leaf apex B are blocked, predict the picking position P1 for one bud and one leaf and the picking position P2 for one bud and two leaves based on the straight line CD (i.e., the connection point C for one bud and one leaf and the connection point D for one bud and two leaves).

[0038] like Figure 7 As shown, the fourth type, where the bud tip and connecting point are partially obscured, specifically includes four scenarios: such as... Figure 7 As shown in (1), if the leaf apex B and the connection point D between one bud and two leaves are obscured, then a straight line equation is established based on the apex A of the bud and the connection point C between one bud and one leaf to predict the picking position P1 of one bud and one leaf; Figure 7 As shown in (2), if the leaf apex B and the connection point C between a bud and a leaf are obscured, then a straight line equation is established based on the bud apex A and the connection point D between a bud and a leaf to predict the picking position P2 of a bud and a leaf; Figure 7 As shown in (3), if the bud apex A and the connection point D between the bud and two leaves are obscured, then a straight line equation is established based on the leaf apex B and the connection point C between the bud and one leaf to predict the picking position P1 of the bud and one leaf; Figure 7 As shown in (4), if the bud apex A and the connection point C between the bud and the leaf are blocked, a straight line equation is established based on the leaf apex B and the connection point D between the bud and the leaf to predict the picking position P1 of the bud and the leaf.

[0039] It is understandable that different analyses are performed depending on the scenario when the bud and leaf apex and connection point are partially obscured: If the leaf apex B and the connection point D of one bud and two leaves are obscured, the picking position P1 of one bud and one leaf is predicted based on the straight line AC (i.e., the two points A and C). As shown in the figure, the equation of the straight line established according to point AC is y = -3.7*x + 63.52, in mm. After establishing the equation, the picking position is taken about 4 mm downwards based on the straight line equation. If the leaf apex B and the connection point C of one bud and one leaf are obscured, the picking position P2 of one bud and two leaves is predicted based on the straight line AD (i.e., the two points A and D). In a specific embodiment, the equation of the straight line established according to point AD is y = -5.88*x + 74.07, in mm. After establishing the equation, the picking position is taken about 4 mm downwards based on the straight line equation. If the bud apex A and the connection point D between the bud and two leaves are obscured, the picking position P1 for the bud and one leaf is predicted based on the straight line BC (i.e., the leaf apex B and the connection point C between the bud and one leaf). In a specific embodiment, as shown in the figure, the equation of the straight line established according to point BC is y = 8.05*x - 88.84, and the equation of the straight line established according to point CD is y = 6.7*x - 71.35, in mm. After establishing the equation, the picking position is taken approximately 4 mm downwards according to the straight line equation. If the bud apex A and the connection point C between the bud and one leaf are obscured, the picking position P1 for the bud and one leaf is predicted based on the straight line BD (i.e., the leaf apex B and the connection point D between the bud and two leaves). In a specific embodiment, as shown in the figure, the equation of the straight line established according to point BD is y = 7.66*x - 82.44, in mm. After establishing the equation, the picking position is taken approximately 4 mm downwards according to the straight line equation.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting the picking position of tea shoots based on key point assistance, characterized in that, The specific steps of the detection method include: S1. Real-time acquisition of tea bud images and input of the images into the Mask R-CNN key point detection model for tea bud and leaf key point prediction; the key point types of tea bud and leaf include bud vertex A, leaf vertex B, one bud and one leaf connection point C, and one bud and two leaves connection point D; the bud picking position includes one bud and one leaf picking position P1 and one bud and two leaves picking position P2. S2. Based on the key point detection model, identify the type and number of key points of tea buds and leaves. Combined with the growth characteristic parameters of buds and leaves, select different picking location detection schemes to determine the picking location of tender buds. There are four types of picking location detection schemes: 1) No overall obstruction; 2) Bud and leaf apex not obstructed; 3) Bud and leaf connection point not obstructed; 4) Bud and leaf apex and connection point partially obstructed. For case 1) where there is no obstruction, first calculate the distance d between the two points based on the positions of the connection point C (one bud and one leaf) and the connection point D (one bud and two leaves). CD The prediction of the picking position of one bud and one leaf, P1 or the picking position of one bud and two leaves, is determined based on the confidence level of the two connection points. That is, if the confidence level of the one bud and one leaf connection point C is greater than the confidence level of the one bud and two leaves connection point D, the predicted picking position is the one bud and one leaf picking position P1; otherwise, the predicted picking position is the one bud and two leaves picking position P2. For case 2) where the bud and leaf apex are not obscured, a prediction model is established based on the relationships between bud apex A, leaf apex B, the connection point between one bud and one leaf C, and the connection point between one bud and two leaves D. This model includes three specific scenarios: If the connection point C between a bud and a leaf is blocked, then a linear equation is established based on the bud apex A and the connection point D between a bud and two leaves, or based on the leaf apex B and the connection point D between a bud and two leaves to predict the picking position P2 of a bud and two leaves. If the connection point D between one bud and two leaves is blocked, then a linear equation is established based on the bud apex A and the connection point C between one bud and one leaf, or based on the leaf apex B and the connection point C between one bud and one leaf to predict the picking position P1 of one bud and one leaf. If both the one-bud-one-leaf connection point C and the one-bud-two-leaf connection point D are obscured, then prediction will not be made for the time being.

2. The method for detecting the picking position of tea shoots based on key point assistance according to claim 1, characterized in that, Regarding case 3), where the bud-leaf connector is not occluded, there are three cases based on the number of occluded vertices: If the bud tip A is blocked and the leaf tip B is not blocked, then a straight line equation is established based on the leaf tip B and the connection point C between the bud and the leaf to predict the picking position P1 of the bud and the leaf, or a straight line equation is established based on the connection point C between the bud and the leaf and the connection point D between the bud and the leaf to predict the picking position P1 of the bud and the leaf and the picking position P2 of the leaf. If the leaf apex B is obscured and the bud apex A is not obscured, then a straight line equation is established based on the connection point C of one bud and one leaf and the connection point D of one bud and two leaves to predict the picking position P1 of one bud and one leaf and the picking position P2 of one bud and two leaves. If both bud apex A and leaf apex B are obscured, then a linear equation is established based on the connection point C between one bud and one leaf and the connection point D between one bud and two leaves to predict the picking position P1 for one bud and one leaf and the picking position P2 for one bud and two leaves.

3. The method for detecting the picking position of tea shoots based on key point assistance according to claim 1, characterized in that, Regarding case 4), where the bud tip and connecting point are partially obscured, there are four specific scenarios: If the leaf apex B and the connection point D between one bud and two leaves are obscured, then a straight line equation is established based on the apex A of the bud and the connection point C between one bud and one leaf to predict the picking position P1 of one bud and one leaf. If the leaf apex B and the connection point C between a bud and a leaf are obscured, then a straight line equation is established based on the bud apex A and the connection point D between a bud and a leaf to predict the picking position P2 of a bud and a leaf. If the bud tip A and the connection point D between the bud and two leaves are obscured, then a straight line equation is established based on the leaf tip B and the connection point C between the bud and one leaf to predict the picking position P1 of the bud and one leaf. If the bud apex A and the connection point C between the bud and the leaf are obscured, then a linear equation is established based on the leaf apex B and the connection point D between the bud and the leaf to predict the picking position P1 of the bud and leaf.

Images