A green asparagus picking recognition method

Abstract

A method for identifying harvested green asparagus involves first training a green asparagus instance segmentation model. Then, a binocular depth camera on a harvesting robot acquires color images and 3D point clouds of the current area. The green asparagus instance segmentation model then obtains the coordinates of the green asparagus detection box and the original mask image of the green asparagus. The original mask image of the green asparagus is cropped to obtain the original binary image of a single green asparagus. Morphological processing is performed on the original binary image of a single green asparagus to extract its morphological skeleton. The centerline of the green asparagus is obtained by fitting the morphological skeleton, and the pixel length of the centerline and the average pixel diameter of the green asparagus are calculated. This morphological skeleton fitting method is applicable to green asparagus in different poses. Finally, using the fitted centerline and average pixel diameter of the green asparagus as constraints, the 3D point cloud information is dynamically mapped to obtain a scaling factor. The actual length of the asparagus and the harvesting point are then calculated, effectively improving the accuracy of the identification.

Description

Technical Field

[0001] This invention relates to the field of harvesting and identification technology, and in particular to a method for identifying green asparagus harvesting. Background Technology

[0002] Harvesting green asparagus is a labor-intensive agricultural activity, characterized by high manual labor intensity and costs. Therefore, automated harvesting robots are urgently needed to alleviate this problem. However, due to the uncertain growth location and inconsistent ripening time of green asparagus, selective harvesting is necessary. The maturity of green asparagus is primarily determined by its length; therefore, a reliable method for measuring the length of green asparagus is required.

[0003] With the continuous development of machine vision technology, image-based localization and discrimination methods have become a key research direction for automated fruit and vegetable harvesting. The research results of deep learning have elevated image processing to a new level. Through deep learning algorithms, it is possible to better identify and distinguish crops based on their features, textures, and colors in images. Therefore, it is very suitable as a preprocessing method for measuring and calculating crop morphological parameters.

[0004] To accurately obtain the actual location and three-dimensional information of the target to be identified, it is usually necessary to use a depth camera to simultaneously collect the depth information or three-dimensional point cloud of the area to be collected. However, due to the small diameter of green asparagus or the influence of outdoor light, the three-dimensional point cloud of green asparagus collected by the depth camera is incomplete, which affects the calculation of the length of green asparagus and the harvesting time. Summary of the Invention

[0005] The technical problem solved by this invention is to provide a method for identifying green asparagus during harvesting, so as to solve the problems in the background art mentioned above.

[0006] The technical problem solved by this invention is achieved by the following technical solution:

[0007] A method for identifying green asparagus during harvesting, with the following specific steps:

[0008] 1) Training a green asparagus instance segmentation model

[0009] I. Collect green asparagus image data. The green asparagus field images are RGB color images with a resolution of 2208×1242 pixels. The camera is positioned to the side of the green asparagus planting direction, 450mm from the ground and 400-600mm from the green asparagus being photographed. The camera lens is angled 15° towards the ground.

[0010] II. Use Labelme annotation software to annotate the green asparagus images collected in step I to create an image dataset. Use polygon labels to annotate each green asparagus in the green asparagus image. Each label is the edge outline of an asparagus. The image dataset includes multiple two-dimensional images.

[0011] III. A green asparagus instance segmentation model was obtained by training a pre-trained YOLO-V9 instance segmentation model in conjunction with an image dataset.

[0012] 2) Measurement and positioning of green asparagus length

[0013] The environment for harvesting green asparagus should have good lighting conditions. The specific steps are as follows:

[0014] Step 1: The harvesting robot arrives at the designated location and uses a binocular depth camera mounted on it to acquire color images and 3D point clouds of the current area. The acquired real-time images of green asparagus are then input into the green asparagus instance segmentation model trained in Step 1. When the green asparagus instance segmentation model detects one or more green asparagus, the coordinates (x, y, y) of the green asparagus detection box are obtained. i ,y i A i B i ), where (x i ,y i ) represents the pixel coordinates of the top-left corner of the green asparagus detection box, A i B is the width of the detection frame for green asparagus (x direction). i Set the height (y-direction) of the green asparagus detection box; simultaneously acquire the original mask image of the green asparagus;

[0015] Step 2: The green asparagus instance segmentation model crops the original mask image of the green asparagus according to the position of the green asparagus detection box, and obtains the original binary image of a single green asparagus.

[0016] Step 3: Perform morphological processing on the original binary image of a single green asparagus obtained in Step 2 to eliminate asparagus morphological noise and obtain the green asparagus morphological outline.

[0017] Step 4: Extract the asparagus shape skeleton from the green asparagus shape contour obtained in Step 3, and perform polynomial fitting based on the pixel coordinates of the asparagus shape skeleton to reduce redundant pixels and smooth the green asparagus shape skeleton, so as to obtain the green asparagus center line and the fitting polynomial y(x) of the green asparagus center line.

[0018] Step 5: Calculate the pixel length (L) of the green asparagus centerline based on its pixel coordinates. pixel Then, based on the center line and outline of the green asparagus, the average pixel diameter (w) of the green asparagus is calculated. A );

[0019] Step 6: Dynamically calculate the scaling factor, that is, using the center line of the green asparagus and the average pixel diameter of the green asparagus as constraints, determine multiple local key measurement points (x) on the outline of the green asparagus shape. pixel ,y pixel) and the circumference used for measurement, and local key measurement points (x pixel ,y pixel This includes the reference measurement point O and various measurement points (M1 to M2) on the circumference. k ), and map local key measurement points (x pixel ,y pixel ) The 3D point cloud at the corresponding location (x real ,y real ,z real Then calculate the measurement points (M1~M) on the circumference. k Distance (L) from the reference measurement point O (pixel distance) k ) and spatial distance (d k The ratio of the calculated proportions is taken as the average of the ratios to obtain the proportionality factor (k). R );

[0020] Step 7: Calculate the length and picking point of green asparagus using a scaling factor to identify the picking location of green asparagus.

[0021] Furthermore, in step 1, the binocular depth camera is a ZED2 binocular camera.

[0022] Further, in step 2, the green asparagus instance segmentation model segments the original mask image of green asparagus. Specifically, the original mask image of green asparagus is identified into two categories: the pixels where green asparagus is located in the color image are segmented into the asparagus category, and the pixels where the background is located are segmented into the background category; the original binary image of a single green asparagus is the mask segmented by the green asparagus instance segmentation model by cropping the original mask image of green asparagus, and the asparagus category region is set as a white pixel region, and the background category is set as a black pixel region.

[0023] Furthermore, in step 3, the specific steps for morphological processing of the original binary image of a single green asparagus are as follows:

[0024] Step 3.1: Fill the holes by filling the black pixels surrounded by white pixels with white pixels;

[0025] Step 3.2: Remove small connected components, ensuring the number of pixels is less than the minimum restricted area A. min Deleting connected components (setting them to black pixels), minimum area constraint A min The area A of the connected domain of the main asparagus max The decision is as shown in formula (1):

[0026] A min =0.5·A max (1)

[0027] To remove any possible segmentation errors or the influence of other non-subject asparagus parts in the surrounding area;

[0028] Step 3.3: Perform a morphological opening operation on the original binary image of a single green asparagus using a disk-shaped structuring element to eliminate noise at the image edges, and then perform an opening operation using a rhombus-shaped structuring element to make the edges of the green asparagus smoother; the radius r of the disk-shaped structuring element... i The side length s of the rhombus structure element i The area A of the connected domain of the main asparagus max The decision is as shown in formulas (2) and (3):

[0029]

[0030] In formulas (2) to (3), k s k d k is a constant. s k d They are 5500 and 2000 respectively;

[0031] This yields the outline of the green asparagus.

[0032] Furthermore, in step 4, the specific steps for obtaining the center line of the green asparagus and the fitting polynomial y(x) of the center line of the green asparagus are as follows:

[0033] Step 4.1: Delete the pixels at the boundary of the area where the asparagus is located in the green asparagus shape outline obtained in Step 3. Repeat this operation while ensuring that the connected components are not separated. The remaining pixels constitute the skeleton of the asparagus shape.

[0034] Step 4.2: Set the pixel coordinates (x, y) of the asparagus morphological skeleton. k ,y k Polynomial fitting was performed on k = 1, 2, ..., m to obtain the center line of the green asparagus, as shown in formulas (4) to (6):

[0035] y(x)=p1x n +p2x n-1 +...+p n x+p n+1 (4)

[0036]

[0037] p = V\y (6)

[0038] In formulas (4) to (6), y(x) is the fitted polynomial, p is the coefficient matrix of the fitted polynomial, and V is the m-row + n-column matrix in formula (5). n is a constant 3.

[0039] Further, in step 5, the pixel length (L) of the green asparagus is calculated. pixel ) and the average pixel diameter of green asparagus (w ASpecifically:

[0040] Step 5.1: Calculate the pixel length of the green asparagus based on the pixel coordinates of the center line. First, calculate the pixel length of the green asparagus's center line at adjacent pixels ((xf...). k ,yf k The differences (dX and dY) in the x and y directions between k = 1, 2, ..., m are calculated, and then the length (L) of each small straight segment of the center line of the green asparagus is calculated using the Pythagorean theorem. s Finally, the lengths of each small straight line segment are added together to obtain the pixel length (L) of the center line of the green asparagus. pixel As shown in formulas (7) to (10):

[0041] dX = [xf2-xf1 xf3-xf2 ... xf m -xf m-1 (7)

[0042] dY = [yf2-yf1 yf3-yf2 ... yf m -yf m-1 (8)

[0043]

[0044] Step 5.2: Select j equally spaced points q1, q2, ... q along the center line of the green asparagus. j And draw perpendicular lines from each of the equal division points to this point, and calculate the values ​​of q1 and q... j In addition, the distances w2, w3, ..., w of the overlapping portion of each vertical line with the area containing the green asparagus (white pixels) j-1 Then, calculate the average distance between the overlapping portion of each vertical line and the area where the green asparagus is located (white pixels), and calculate the average pixel diameter (w) of the green asparagus. A As shown in formula (11):

[0045]

[0046] In formula (11), j is set to a constant 10.

[0047] Furthermore, in step 6, the specific steps for dynamically calculating the scaling factor include:

[0048] Step 6.1: Determine the reference measurement point O. Since the 3D point cloud at the base of the green asparagus will not be missing, start searching upwards along the center line of the green asparagus from the base. When the 3D point cloud is not empty, draw a circle with the reference measurement point O as the center. To constrain the circle to the green asparagus, the starting point of the circle is one time the average pixel diameter of the green asparagus (w) from the bottom endpoint of the center line of the green asparagus. A At point ), the radius is the average pixel diameter of the green asparagus (w)A Half of the circle, obtain k measurement points (M1~M) on the circumference. k ), and the distance from the reference measurement point O to the bottom endpoint of the center line of the green asparagus is 1 to 1.5 times the average pixel diameter of the green asparagus (w A Between ), k is a constant of 16; the pixel coordinates (x, y) of the reference measurement point O are obtained. Op ,y Op ) and the pixel coordinates (x) of the measurement point on the circumference kp ,y kp );

[0049] Step 6.2: Based on the pixel coordinates (x, y) of the reference measurement point O determined in Step 6.1 Op ,y Op ) and the pixel coordinates (x) of the measurement point on the circumference kp ,y kp The spatial coordinates (x, y) of the reference measurement point O are obtained by mapping. O ,y O ,z O ) and the spatial coordinates of the measurement point (x k ,y k ,z k ), calculate the spatial distance (d) between the reference measurement point O and each measurement point. k As shown in formula (12):

[0050]

[0051] In formula (12), k = 1, 2, ..., 16;

[0052] Step 6.3: Calculate the scaling factor based on the spatial distance (d) between the reference measurement point O and each measurement point obtained in Step 6.2. k ), calculate the measurement points (M1~M) on the circumference. k Distance (L) from the reference measurement point O (pixel distance) k ) and spatial distance (d k The proportion of ), where pixel distance (L) k ) equals 0.5 times the average pixel diameter of green asparagus (i.e., L) k =0.5w A Finally, the average of the calculated proportions is taken to obtain the proportion factor (k) of the location of the green asparagus. R As shown in formula (13):

[0053]

[0054] Furthermore, in step 7, the length of the green asparagus and the picking point are calculated using a scaling factor. The specific steps include:

[0055] Step 7.1: Based on the pixel length (L) of the green asparagus centerline obtained in Step 5.1 pixel ) and the scaling factor (k) obtained in step 6.3 R ), calculate the actual length (L) of the green asparagus. real As shown in formula (14):

[0056] L real =L pixel k R (14)

[0057] Step 7.2: Constrain the green asparagus picking point on the center line of the green asparagus, and then determine the stubble length (L) after picking. c The distance (d) from the bottom endpoint of the green asparagus centerline was calculated using the ratio factor and the green asparagus picking point. c Finally, based on the fitting polynomial y(x) of the green asparagus centerline obtained in step 4, the coordinates (x, y, x) of the green asparagus picking point are solved. c ,y c The calculations are shown in formulas (15) and (16):

[0058]

[0059] In formula (15), the stubble length L c The range is 0.5cm < L c <1.5cm.

[0060] Beneficial effects: This invention uses the two-dimensional parameters of each green asparagus as a constraint to dynamically select local key measurement points, effectively ensuring the reliability of the data; and by calculating the scaling factor between the two-dimensional parameters of the asparagus location and the three-dimensional point cloud, it avoids the influence of directly using incomplete three-dimensional point clouds on the actual length measurement, improves the robustness of the algorithm, and provides a guarantee for the accuracy of calculating the length of green asparagus and the harvesting time. Attached Figure Description

[0061] Figure 1 This is a flowchart of a preferred embodiment of the present invention.

[0062] Figure 2 This is a diagram illustrating the morphological opening operation process in a preferred embodiment of the present invention.

[0063] Figure 3 This is a diagram illustrating the calculation process of the center line and two-dimensional parameters of green asparagus in a preferred embodiment of the present invention.

[0064] Figure 4 This is a diagram showing the relationship between local key measurement points and corresponding three-dimensional point cloud in a preferred embodiment of the present invention.

[0065] Figure 5This is a diagram of the actual harvesting environment for green asparagus in a preferred embodiment of the present invention.

[0066] Figure 6 This is a diagram illustrating the intermediate process of actual harvesting in a preferred embodiment of the present invention.

[0067] Figure 7 This is a diagram showing the final result of actual harvesting environment identification and positioning in a preferred embodiment of the present invention.

[0068] Explanation of reference numerals in the attached figures: Figure 2 In the image, (a) is the original binary image obtained in step 2, and (b) and (c) are the binary images after morphological processing. Figure 3 In the image, (a) is the morphological skeleton of the green asparagus extracted in step 4.1, (b) is the center line of the green asparagus obtained by fitting calculation in step 4.2), (c) is the two-dimensional parameter measurement performed in step 5.2, and (d) is an enlarged view of the green asparagus pixel diameter measurement. Figure 6 In the diagram, (a) is the result after morphological processing in step 3, (b) is the result after processing in step 4, (c) is the result after processing in step 5, and (d) is the result after localization in step 7.2. Detailed Implementation

[0069] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific embodiments.

[0070] A method for identifying green asparagus during harvesting, with the following specific steps:

[0071] 1) Training a green asparagus instance segmentation model

[0072] I. Collect green asparagus image data. The green asparagus field images are RGB color images with a resolution of 2208×1242 pixels. The camera is positioned to the side of the green asparagus planting direction, 450mm from the ground and 400-600mm from the green asparagus being photographed. The camera lens is angled 15° towards the ground.

[0073] II. Use Labelme annotation software to annotate the green asparagus images collected in step I to create an image dataset. Use polygon labels to annotate each green asparagus in the green asparagus image. Each label is the edge outline of an asparagus. The image dataset includes multiple two-dimensional images.

[0074] III. Based on the pre-trained YOLO-V9 instance segmentation model, a green asparagus instance segmentation model is obtained by training the image dataset. In this implementation example, the deep learning framework is PyTorch 2.4.0, and the network input size is 640×640.

[0075] 2) Measurement and positioning of green asparagus length

[0076] The environment for harvesting green asparagus should have good lighting conditions. The specific steps are as follows:

[0077] Step 1: The harvesting robot arrives at the designated location and uses a binocular depth camera mounted on it to acquire color images and 3D point clouds of the current area. The image size is 2280×1242. The acquired real-time images of green asparagus are then input into the green asparagus instance segmentation model trained in Step 1. When the green asparagus instance segmentation model detects one or more green asparagus, the coordinates (x, y, y) of the green asparagus detection box are obtained. i ,y i A i B i ), where (x i ,y i ) represents the pixel coordinates of the top-left corner of the green asparagus detection box, A i B is the width of the detection frame for green asparagus (x direction). i Set the height (y-direction) of the green asparagus detection box; simultaneously acquire the original mask image of the green asparagus;

[0078] Preferably, the binocular depth camera is a ZED2 binocular camera;

[0079] Step 2: As Figure 2 As shown in (a), the green asparagus instance segmentation model crops the original mask image of the green asparagus according to the position of the green asparagus detection box to obtain the original binary image of a single green asparagus.

[0080] The green asparagus instance segmentation model segments the original mask image of green asparagus. Specifically, it identifies the original mask image of green asparagus into two categories: the pixels where green asparagus is located in the color image are segmented into the asparagus category, and the pixels where the rest of the background is located are segmented into the background category.

[0081] The original binary image of a single green asparagus is a mask segmented from the original mask image of the green asparagus by the green asparagus instance segmentation model, with the asparagus category region set as a white pixel region and the background category set as a black pixel region;

[0082] Step 3: Perform morphological processing on the original binary image of a single green asparagus obtained in Step 2 to eliminate noise in the asparagus shape and obtain the outline of the green asparagus shape. The result is as follows: Figure 6 As shown in (a);

[0083] Step 3.1: Fill the holes by filling the black pixels surrounded by white pixels with white pixels;

[0084] Step 3.2: Remove small connected components, ensuring the number of pixels is less than the minimum restricted area A. min Deleting connected components (setting them to black pixels), minimum area constraint A minThe area A of the connected domain of the main asparagus max The decision is as shown in formula (1):

[0085] A min =0.5·A max (1)

[0086] To remove any possible segmentation errors or the influence of other non-subject asparagus parts in the surrounding area;

[0087] Step 3.3: As Figure 2 As shown in (b), a disk-shaped structuring element is used to perform a morphological opening operation on the original binary image of a single green asparagus to eliminate noise at the image edges. Then, a rhombus-shaped structuring element is used for the opening operation to make the edges of the green asparagus smoother. Figure 2 As shown in (c); the radius r of the disk-shaped structural element i The side length s of the rhombus structure element i The area A of the connected domain of the main asparagus max The decision is as shown in formulas (2) and (3):

[0088]

[0089] In formulas (2) to (3), k s k d k is a constant. s k d They are 5500 and 2000 respectively;

[0090] This yields the morphological outline of the green asparagus;

[0091] Step 4: As Figure 6 As shown in (b), the asparagus morphological skeleton is extracted from the green asparagus morphological outline. Then, a polynomial fitting is performed based on the pixel coordinates of the asparagus morphological skeleton to reduce redundant pixels and smooth the green asparagus morphological skeleton, thus obtaining the green asparagus centerline.

[0092] Step 4.1: As Figure 3 As shown in (a), delete the pixels of the boundary of the area where the asparagus is located (white pixels) in the green asparagus shape outline obtained in step 3, repeat this operation, and ensure that the connected components are not separated. The remaining pixels constitute the asparagus shape skeleton.

[0093] Step 4.2: As Figure 3 As shown in (b), the pixel coordinates (x) of the asparagus morphological skeleton are... k ,y k Polynomial fitting was performed on k = 1, 2, ..., m to obtain the center line of the green asparagus, as shown in formulas (4) to (6):

[0094] y(x)=p1x n +p2xn-1 +...+p n x+p n+1 (4)

[0095]

[0096] p = V\y (6)

[0097] In formulas (4) to (6), y(x) is the fitted polynomial, p is the coefficient matrix of the fitted polynomial, and V is the m-row + n-column matrix in formula (5). n is a constant 3;

[0098] Step 5: As Figure 6 As shown in (c), the pixel length (L) of the green asparagus centerline is calculated based on the pixel coordinates of the green asparagus centerline. pixel Then, based on the center line and outline of the green asparagus, the average pixel diameter (w) of the green asparagus is calculated. A Specifically:

[0099] Step 5.1: Calculate the pixel length of the green asparagus based on the pixel coordinates of the center line. First, calculate the pixel length of the green asparagus's center line at adjacent pixels ((xf...). k ,yf k The differences (dX and dY) in the x and y directions between k = 1, 2, ..., m are calculated, and then the length (L) of each small straight segment of the center line of the green asparagus is calculated using the Pythagorean theorem. s Finally, the lengths of each small straight line segment are added together to obtain the pixel length (L) of the center line of the green asparagus. pixel As shown in formulas (7) to (10):

[0100] dX = [xf2-xf1 xf3-xf2 ... xf m -xf m-1 (7)

[0101] dY = [yf2-yf1 yf3-yf2 ... yf m -yf m-1 (8)

[0102]

[0103] Step 5.2: As Figure 3 As shown in (c), j equally divided points q1, q2, ... q are selected along the center line of the green asparagus. j And draw perpendicular lines from each of the points of equal division to this point; such as Figure 3 As shown in (d), calculate excluding q1 and q j In addition, the distances w2, w3, ..., w of the overlapping portion of each vertical line with the area containing the green asparagus (white pixels)j-1 Then, calculate the average distance between the overlapping portion of each vertical line and the area where the green asparagus is located (white pixels), and calculate the average pixel diameter (w) of the green asparagus. A As shown in formula (11):

[0104]

[0105] In formula (11), j is set to a constant of 10;

[0106] Step 6: As Figure 4 As shown, the scaling factor is dynamically calculated, that is, multiple local key measurement points (x, y, x) are determined on the outline of the green asparagus, constrained by the center line of the green asparagus and the average pixel diameter of the green asparagus. pixel ,y pixel ), local key measurement points (x) pixel ,y pixel This includes the reference measurement point O and various measurement points (M1 to M2) on the circumference. k ), and map the corresponding 3D point cloud (x) to the location. real ,y real ,z real Then calculate the measurement points (M1~M) on the circumference. k Distance (L) from the reference measurement point O (pixel distance) k ) and spatial distance (d k The ratio of the calculated ratios is taken as the average of the ratios, which gives the scaling factor (k). R To address the impact of incomplete 3D point cloud data on the upper part of green asparagus on direct measurement, the specific steps include:

[0107] Step 6.1: Determine the reference measurement point O. Since the 3D point cloud at the base of the green asparagus will not be missing, start searching upwards along the center line of the green asparagus from the base. When the 3D point cloud is not empty, draw a circle with the reference measurement point O as the center. To constrain the circle to the green asparagus, the starting point of the circle is one time the average pixel diameter of the green asparagus (w) from the bottom endpoint of the center line of the green asparagus. A At point ), the radius is the average pixel diameter of the green asparagus (w) A Half of the circle, obtain k measurement points (M1~M) on the circumference. k ), and the distance from the reference measurement point O to the bottom endpoint of the center line of the green asparagus is 1 to 1.5 times the average pixel diameter of the green asparagus (w A Between ), k is a constant of 16; the pixel coordinates (x, y) of the reference measurement point O are obtained. Op ,y Op ) and the pixel coordinates (x) of the measurement point on the circumference kp ,y kp );

[0108] Step 6.2: Based on the pixel coordinates (x, y) of the reference measurement point O determined in Step 6.1 Op ,y Op ) and the pixel coordinates (x) of the measurement point on the circumference kp ,y kp The spatial coordinates (x, y) of the reference measurement point O are obtained by mapping. O ,y O ,z O ) and the spatial coordinates of the measurement point (x k ,y k ,z k ), calculate the spatial distance d between the reference measurement point O and each measurement point. k As shown in formula (12):

[0109]

[0110] In formula (12), k = 1, 2, ..., 16;

[0111] Step 6.3: Calculate the scaling factor based on the spatial distance (d) between the reference measurement point O and each measurement point obtained in Step 6.2. k ), calculate the measurement points (M1~M) on the circumference. k Distance (L) from the reference measurement point O (pixel distance) k ) and spatial distance (d k The proportion of ), where pixel distance (L) k ) equals 0.5 times the average pixel diameter of green asparagus (i.e., L) k =0.5w A Finally, the average of the calculated proportions is taken to obtain the proportion factor (k) of the location of the green asparagus. R As shown in formula (13):

[0112]

[0113] Step 7: Calculate the length and picking point of green asparagus using a dynamic scaling factor to identify the picking location of green asparagus. Specific steps include:

[0114] Step 7.1: Based on the pixel length (L) of the asparagus centerline obtained in Step 5.1 pixel ) and the scaling factor (k) obtained in step 6.3 R ), calculate the actual length (L) of the green asparagus. real As shown in formula (14):

[0115] L real =L pixel k R (14)

[0116] Step 7.2: Constrain the green asparagus picking point on the center line of the green asparagus, and then determine the stubble length (L) after picking. c The distance (d) from the bottom endpoint of the green asparagus centerline was calculated using the ratio factor and the green asparagus picking point. c Finally, based on the fitting polynomial y(x) of the green asparagus centerline obtained in step 4.2, the coordinates (x, y, x) of the green asparagus picking point are solved. c ,y c The location results are as follows: Figure 6 As shown in (d), the calculations are as shown in formulas (15) and (16):

[0117]

[0118] In formula (15), the stubble length L c The length was set to 1cm, but the actual length tested was 0.5cm < L. c <1.5cm;

[0119] To test the effectiveness of the method in this embodiment, the above examples were tested in an actual green asparagus harvesting scenario, such as... Figure 5 As shown, Figure 6 The diagram shows the intermediate process of this example. The final result of the actual harvesting environment identification and positioning is as follows. Figure 7 As shown.

[0120] In the description of this embodiment, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "front end," and "tail end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "a," "b," and "c" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0121] In the description of this embodiment, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

Claims

1. A method for identifying green asparagus during harvesting, characterized in that, The specific steps are as follows: 1) Training green asparagus instance segmentation model First, collect green asparagus image data, then label the collected green asparagus images to make image data sets, and then train a green asparagus instance segmentation model based on a pre-trained YOLO-V9 instance segmentation model combined with the image data set; 2) Green asparagus length measurement and positioning Step 1: the picking robot reaches the position, acquires the color image and three-dimensional point cloud of the current area through the binocular depth camera installed on the picking robot; then inputs the acquired real-time image of green asparagus into the green asparagus instance segmentation model trained in step 1); when the green asparagus instance segmentation model detects one or more green asparagus, the green asparagus detection box coordinates (x i ,y i ,A i ,B i ) are obtained, wherein (x i ,y i ) is the pixel coordinates of the upper left corner of the green asparagus detection box, A i is the width of the green asparagus detection box, i.e. the x direction, and B i is the height of the green asparagus detection box, i.e. the y direction; at the same time, the original mask image of the green asparagus is obtained; Step 2: The green asparagus instance segmentation model crops the original mask image of the green asparagus according to the green asparagus detection box position to obtain a single green asparagus original binary image; Step 3: Perform morphological processing on the single green asparagus original binary image obtained in step 2 to obtain a green asparagus shape contour; Step 4: Extract the asparagus shape skeleton on the green asparagus shape contour obtained in step 3, and perform polynomial fitting according to the asparagus shape skeleton pixel coordinates to obtain the green asparagus center line and the fitting polynomial y(x) of the asparagus center line; Step 5: Calculate the pixel length L of the green asparagus center line according to the pixel coordinates of the green asparagus center line pixel , and then calculate the average pixel diameter w of the green asparagus according to the green asparagus center line and the green asparagus shape contour A ; Step 6: dynamically calculate the scale factor, i.e. determine a plurality of local key measurement points (x pixel ,y pixel ) and a circumference for measurement on the green asparagus shape contour with the green asparagus center line and the green asparagus average pixel diameter as constraints, map the three-dimensional point cloud (x pixel ,y pixel ,z real ) corresponding to the positions of the local key measurement points (x real ,y real ), and then calculate the scale of the pixel distance L k and the spatial distance d k from each measurement point M1-M k on the circumference to the reference measurement point O, take the average of the calculated scales to obtain the scale factor k R ; the local key measurement points (x pixel ,y pixel ) include the reference measurement point O and each measurement point M1-M k on the circumference; Step 7: Calculate the green asparagus length and picking point through the scale factor to realize the identification of green asparagus picking.

2. The green asparagus picking recognition method according to claim 1, characterized in that, In step 1), each green asparagus in the green asparagus image is labeled using a polygon label, and each label is the edge contour of an asparagus. The image data set includes multiple two-dimensional images.

3. The green asparagus picking recognition method according to claim 1, characterized in that, In step 2), the green asparagus instance segmentation model segments the original mask image of the green asparagus. Specifically, the original mask image of the green asparagus is identified as two categories: the pixels where the green asparagus in the color image is located are segmented into the asparagus category, and the pixels where the rest of the background is located are segmented into the background category.

4. The green asparagus picking recognition method according to claim 1 or 3, characterized in that, The single green asparagus original binary image is the mask segmented from the original mask image of the green asparagus instance segmentation model, with the asparagus category area set as a white pixel area and the background category set as a black pixel area.

5. The green asparagus picking recognition method according to claim 1, characterized in that, In step 2), the specific steps of morphological processing on the single green asparagus original binary image are as follows: Step 3.1: Fill the holes by filling the black pixels surrounded by white pixels with white pixels; Step 3.2: Remove small connected domains, and delete connected domains whose pixel number is less than the minimum limited area A min , i.e. set to black pixels, the minimum limited area A min is determined by the connected domain area A max of the main asparagus, as shown in formula (1): Amin = Amain x (1 - 0.5) A min = 0.5 · A max (1) To remove the possible segmentation error areas or the influence of other non-main asparagus around; Step 3.3: Morphological opening operation is performed on the single green asparagus original binary image using a disc-shaped structural element, and then a rhombus-shaped structural element is used to perform opening operation to make the green asparagus edge more smooth; the radius r of the disc-shaped structural element i and the side length s of the rhombus-shaped structural element i The connected domain area A of the main asparagus max is determined, as shown in formulas (2), (3): In the formulas (2) to (3), k s , k d are constants, k s , k d are 5500 and 2000, respectively. Thus obtaining the green asparagus shape contour.

6. The green asparagus picking recognition method according to claim 1, characterized in that, In step 2), the specific steps of obtaining the green asparagus center line and the fitting polynomial y(x) of the green asparagus center line are as follows: Step 4.1: Delete the pixels of the asparagus area boundary in the green asparagus shape contour obtained in step 3, and repeat the operation while ensuring that the connected domains are not separated. The remaining pixels constitute the asparagus shape skeleton; Step 4.2: The pixel point coordinates (x k ,y k ), k = 1, 2,..., m of the asparagus morphological skeleton are polynomial fitted to obtain the green asparagus center line, as shown in formulas (4) to (6): y(x) = pi x + p2x +... + pnx + p0 n + p2x n-1 +... + pn x + p0 n + p2x n+1 (4) p=V\y (6) In formulas (4) to (6), y(x) is a fitted polynomial, p is a coefficient matrix of the fitted polynomial, and V is an m-row and n+1-column matrix in formula (5) n is a constant of 3.

7. The green asparagus picking recognition method according to claim 1, characterized in that, In step 2), the pixel length of the green asparagus and the average pixel diameter of the green asparagus are calculated as follows: Step 5.1: Calculate the pixel length of the green asparagus center line according to the pixel point coordinates of the green asparagus center line. First, calculate the difference dX and dY in the x and y directions between adjacent pixel points (xf k ,yf k ) of the green asparagus center line, k = 1, 2,..., m, then calculate the length L s of each small straight line segment of the green asparagus center line using the Pythagorean theorem, and finally add up the length of each small straight line segment to obtain the pixel length L pixel of the green asparagus center line, as shown in formulas (7)-(10): dX = [xf2-xf1 xf3-xf2... xf m -xf m-1 ] (7) dY = [yf2 - yf1 yf3 - yf2... yf m -yf m-1 ] (8) Step 5.2: Select j equidistant points q1, q2,... qj on the center line of the green asparagus, and draw a perpendicular line through each equidistant point, and calculate the distance w2, w3,... wj of the overlapping part of each perpendicular line and the area where the green asparagus is located, except for q1 and qj. j Step 5.3: Calculate the average value of the distance of the overlapping part of each perpendicular line and the area where the green asparagus is located, and calculate the average pixel diameter w of the green asparagus. j Step 5.3: Calculate the average value of the distance of the overlapping part of each perpendicular line and the area where the green asparagus is located, and calculate the average pixel diameter w of the green asparagus. j-1 Step 5.3: Calculate the average value of the distance of the overlapping part of each perpendicular line and the area where the green asparagus is located, and calculate the average pixel diameter w of the green asparagus. A As shown in formula (11): In formula (11), j is set to a constant 10.

8. The green asparagus picking recognition method according to claim 7, characterized in that, In step 2), the specific steps of dynamically calculating the scale factor include: Step 6.1: determining the reference measurement point O, searching from the bottom of the green asparagus along the center line of the green asparagus, when the three-dimensional point cloud is not empty, taking the O point as the center to make a circle, and the starting point of the circle is 1 times the average pixel diameter w of the green asparagus from the bottom end point of the center line of the green asparagus A , the radius is half of the average pixel diameter w of the green asparagus, and k measurement points M1-M A are obtained on the circumference, the distance between the reference measurement point O and the bottom end point of the center line of the green asparagus is between 1-1.5 times the average pixel diameter w of the green asparagus, and k is a constant 16; the pixel coordinates (x k , y A ) of the reference measurement point O and the pixel coordinates (x Op , y Op ) of the measurement points on the circumference are obtained kp , y kp ) Step 6.2: According to the pixel coordinates (x Op ,y Op ) of the reference measurement point O and the pixel coordinates (x kp ,y kp ) of the measurement points on the circumference determined in step 6.1, the spatial coordinates (x O ,y O ,z O ) of the reference measurement point O and the spatial coordinates (x k ,y k ,z k ) of the measurement points are mapped, and the spatial distances d k between the reference measurement point O and each of the measurement points M1-M k are calculated, as shown in equation (12): d k = (x k -x O )2+ (y k -y O )2+ (z k -z O )2 In formula (12), k=1,2,…,16; Step 6.3: Calculate the scale factor k, which is the average of the ratios of the pixel distance L and the spatial distance d between the reference point O and each of the measuring points M1-M8 k , calculated according to step 6.2 k Step 6.4: Calculate the pixel distance L between the reference point O and each of the measuring points M1-M8 k Step 6.5: Calculate the ratio of the pixel distance L and the spatial distance d between the reference point O and each of the measuring points M1-M8 k Step 6.6: Calculate the scale factor k, which is the average of the ratios of the pixel distance L and the spatial distance d between the reference point O and each of the measuring points M1-M8 R as shown in equation (13): In Equation (13), the pixel distance L k is equal to 0.5 times the average pixel diameter w A of green asparagus.

9. The green asparagus picking recognition method according to claim 1, characterized in that, In step 2), the specific steps of calculating the green asparagus length and picking point through the scale factor include: Step 7.1: Calculate the pixel length L of the center line of green asparagus obtained in Step 5 pixel and the scale factor k obtained in Step 6 R , calculate the actual length L of green asparagus real as shown in equation (14): L real = L pixel k R (14) Step 7.2: The green asparagus harvesting point is constrained on the asparagus center line, and the stubble length L after harvesting is used to calculate the distance d of the green asparagus harvesting point from the bottom end point of the asparagus center line c and the proportion factor c , and finally the coordinates (x c ,y c ) of the green asparagus harvesting point are calculated according to the fitting polynomial y(x) of the asparagus center line obtained in step 4, as shown in formulas (15) and (16): In Equation (15), the stub length L c is in the range 0.5 cm < L c < 1.5 cm.

Images