A method, device and storage medium for identifying crop root and stem tips

Abstract

The present invention discloses a method for identifying the top of a crop root and stem, which includes using a camera to collect images of crops with leaves, manually annotating the images, and defining the leaf orientation as the extension line from the leaf center to the heart-shaped edge of the leaf base. Subsequently, deep learning technology is used to learn the annotation information, and a model is established to detect the leaf center and orientation of each visible leaf. The pixel offset between frames is calculated using SIFT / SURF / ORB feature points and descriptors, the position of the leaf center of the previous frame plus the offset is projected, and the closest feature point pairs are continuously matched to determine the stable leaf center orientation. Finally, based on the position and orientation of the leaf center in multiple frames, the degree of fluctuation of the leaf center orientation change is determined, and the position of the crop apical bud is determined along the orientation extension line. This comprehensive method aims to achieve efficient and accurate identification of the top of crop root and stem.

Description

Technical Field

[0001] The present invention relates to the field of smart agricultural technology, and in particular to a method, device and storage medium for identifying the top of a crop root or stem. Background Art

[0002] Apical dominance refers to the phenomenon that the apical buds of plants grow preferentially while the lateral buds are inhibited. During the growth of some crops, the auxin produced by the apical buds is transported downward and accumulated in large quantities in the lateral buds, causing the auxin concentration in the lateral buds to be too high, thereby inhibiting the growth of the lateral buds. This is the apical dominance phenomenon. Based on this principle in agricultural production, crops such as cotton are pinched when they grow to a certain height so that the apical buds no longer produce auxin, reducing the auxin content in the lateral buds and promoting the lateral buds to develop into side branches, thereby achieving the purpose of increasing production.

[0003] The premise of mechanized topping is to intelligently identify the top of the crop, but it does not mean simply cutting off the highest part of the crop, because the top of the crop root may be blocked by leaves and not at the very top or even not exposed. Figure 3 As shown in Figure 1, the cotton top is not necessarily at the top of the cotton plant, but may be blocked by the leaves on the upper part of the side branches. Figure 4 As shown. Moreover, the height difference between the cotton top and the cotton bud is very small, the cotton bud cannot be cut, and the growth height of each cotton plant is also different, which requires identifying the position of the top of each cotton plant, so as to use the robot arm blade to perform the cotton topping operation. The position of the cotton top is difficult to identify because it will be blocked by the leaves and difficult to distinguish. Although leaf features can be identified using artificial intelligence technologies such as deep learning, such technologies also have the problem of missed detection or false detection. The difficulty is that the method of relying on leaf features to identify the terminal bud requires stable and correct leaf features, such as the leaf heart or leaf orientation. However, whether it is the leaf heart or orientation, it is inevitable that there will be fluctuations in multi-frame detection, especially the leaf orientation, which causes the identification of the terminal bud to fluctuate in multiple frames. Summary of the invention

[0004] In order to solve the above-mentioned technical problems, the present invention realizes obtaining the characteristics of stable leaves to obtain a stable apical bud position, and provides a method for identifying the apex of a crop root.

[0005] The technical solution of the present invention is achieved in this way:

[0006] A method for identifying the top of a crop root stem comprises the following steps:

[0007] S1. Using a camera to obtain images of crops with leaves, manually marking the images of crops with leaves, and marking the leaf centers using rectangular frames or dots;

[0008] S2. Mark the leaf direction, and define the leaf direction as the extension line from the leaf center to the heart-shaped edge of the leaf base;

[0009] S3. Use deep learning technology to learn the annotation information in steps S1 to S2, use the rectangular box detection in target detection to let the model learn to predict the leaf center rectangular box, and learn the angle information of the leaf orientation, so as to obtain a model that can output the leaf center position and leaf orientation;

[0010] S4, through steps S1 to S3, establish a model for detecting the leaf center and orientation of each visible leaf in the image;

[0011] S5. Calculate the feature points and corresponding descriptors of the two frames using sift / surf / orb feature points and descriptors between frames. After matching the features of the two frames, arrange the features in ascending order according to the matching distance, take out a pair of feature matching points with the shortest feature distance, and subtract the upper positions of this pair of feature points in each image to obtain the offsets of u and v of the pixel positions of the image, where uv represents the pixel position.

[0012] S6. After obtaining the inter-frame offset, the leaf center position of the previous frame is added with the offset, which is equivalent to projecting it on the current frame. The pixel position distances of the leaf center projected on the current frame and the leaf center detected by the current frame model are calculated for any two of them to obtain a matrix C with the points of the previous frame and the points of the current frame as columns. The matrix content is the distance between the point pairs of the two frames. The distances are arranged in ascending order, and the point pairs are taken from the minimum distance. Among them, the point pair is not the feature point pair of the previous step, but the point pair of the position of the leaf center of the two frames. If the distance is less than the threshold, it is considered that the point pair belongs to the same point, and the point matching is successful and recorded as Mi;

[0013] S7, continue to take point pairs starting from the current minimum distance, and repeat the steps until the distance taken is greater than the threshold;

[0014] S8. After the above steps, the position and corresponding orientation of each leaf center id in multiple frames are obtained. By taking out all orientations of a single leaf center in multiple frames, the degree of fluctuation of the leaf center orientation change can be determined.

[0015] S9. According to the output stable orientation and position of the leaf center, each leaf center is extended along the orientation by a straight line of a certain length, any two straight lines intersect to generate intersection points, and the intersection points within a certain range are averaged to obtain the final crop terminal bud position.

[0016] A crop root and stem top recognition device includes a cotton topping device, the cotton topping device includes a frame and a visual recognition component, the visual recognition component is fixedly installed on the side wall of the frame, and the visual recognition component is used to recognize the spatial position of the cotton top;

[0017] The side wall of the rack is fixedly connected with a mounting base for mounting a binocular recognition camera, and the top of the mounting base is fixedly connected with a cable hub, which is used for arranging and protecting cables on the binocular recognition camera.

[0018] A storage medium for a crop root and stem top identification method comprises a master control machine, on which a computer program is designed and on which computer program instructions are stored. When the computer program instructions are executed, a crop root and stem top identification method is implemented.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. The present invention provides accurate training data for the deep learning model by manually marking the leaf center and the leaf orientation, so that the model can learn to accurately predict the leaf center position and the leaf orientation, thereby providing reliable information in the subsequent steps. The deep learning technology is used to learn the marking information, and a model that can output the leaf center position and the leaf orientation is established to ensure that the leaf center and the leaf orientation can be accurately detected in each frame of the image;

[0021] 2. By calculating the feature point offset between frames, the translation perspective transformation during crop movement is realized, which reduces the amount of calculation and ensures the continuity and stability of the leaf center position. By matching the successful leaf center point pairs and comparing the historical orientations, the stable leaf center orientation is screened out, avoiding the situation where the orientation fluctuations affect the final result and ensuring the stability of the leaf center orientation.

[0022] 3. According to the output stable leaf center direction and position, the final crop terminal bud position is determined by extending the straight line of the leaf center direction and finding the intersection point, which ensures the stability and accuracy of the terminal bud position;

[0023] In summary, the present invention utilizes deep learning technology and inter-frame information processing methods to effectively obtain stable leaf features and determine a stable apical bud position, thereby reducing the fluctuation of apical bud recognition in multiple frame positions and improving the accuracy and stability of crop root and stem apex recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a process for identifying the top of a crop root or stem according to the present invention;

[0025] Figure 2 This is a schematic diagram of the structure of a crop root and stem top identification device in the present invention;

[0026] Figure 3 is a schematic diagram of a cotton top in the background art;

[0027] Figure 4 It is a schematic diagram of the cotton top in the background technology being blocked;

[0028] Figure 5 This is a schematic diagram of using a rectangular frame to mark the leaf center in Example 1 of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Example 1

[0031] like Figure 1 As shown, a method for identifying the top of a crop root stem comprises the following steps:

[0032] S1. Use a camera to obtain images of crops with leaves, manually annotate the images of crops with leaves, and use a rectangular frame to annotate the leaf center, such as Figure 5 As shown (in other embodiments, the leaf center may also be marked in the form of a dot);

[0033] S2. Label the direction of the blade. Define the direction of the blade as the extension line from the center of the blade to the heart-shaped edge of the blade base. The direction label does not have to be toward the heart-shaped edge of the blade base. It can also be in the opposite direction of the blade base. Because the direction of the blade is defined in different directions, the post-processing logic only needs to be slightly changed, and no constraints are imposed here.

[0034] S3. Use deep learning technology to learn the annotation information in steps S1 to S2, use the rectangular box detection in target detection to let the model learn to predict the leaf center rectangular box, and learn the angle information of the leaf orientation, so as to obtain a model that can output the leaf center position and leaf orientation;

[0035] The target detection method uses the Yolo series and adds a prediction quantity as an angle, which is the angle relative to a certain axis such as the positive x-axis. If the leaf center is marked with a point, the key point detection technology of deep learning is used to detect the key point of the leaf center, replacing the above target rectangular box detection part.

[0036] Prepare images of crops with leaves and manually annotate them, using rectangular boxes or points to mark the position of the leaf center and the angle information of the leaf orientation;

[0037] Select a deep learning model suitable for target detection tasks, such as the YOLO (You Only Look Once) series, and use the labeled crop image dataset to train the deep learning model. During the training process, the model needs to learn to identify the rectangular box of the leaf center and predict the angle information of the leaf orientation, specifically:

[0038] In the target detection task, the leaf center position is represented by the coordinates of the rectangular box. Assume that the leaf center position is the coordinates of the center point of the rectangular box (x_center, y_center), the width of the rectangular box is w, and the height is h.

[0039] Leaf center position: (x_center, y_center) = (x_rect + 0.5*w, y_rect + 0.5*h)

[0040] The blade orientation is expressed as an angle, which can be calculated according to the defined blade orientation angle calculation formula.

[0041] Leaf orientation angle (relative to an axis such as the positive x-axis): angle = arctan ((y_leaf_base-y_leaf_tip) / (x_leaf_base-x_leaf_tip))

[0042] Among them, (x_leaf_base, y_leaf_base) is the coordinate of the heart-shaped edge of the leaf base, and (x_leaf_tip, y_leaf_tip) is the coordinate of the leaf center;

[0043] When matching feature points between frames, the SIFT (Scale Invariant Feature Transform) algorithm is used to extract feature points and descriptors, and then the offset between feature points is calculated by matching distance, specifically:

[0044] 1. SIFT feature point extraction:

[0045] The SIFT algorithm detects local feature points (key points) of an image and generates a descriptor for each key point, which describes the characteristics of the area around the key point. The extraction of key points is achieved through scale space extreme point detection and key point positioning;

[0046] 2. Feature descriptor calculation:

[0047] For each extracted key point, the SIFT algorithm calculates a 128-dimensional descriptor that describes the characteristics of the area around the key point. These descriptors are calculated by the image gradient direction histogram around the key point;

[0048] 3. Feature point matching:

[0049] In the process of matching feature points between different frames, the nearest neighbor matching method is used, that is, for each feature point, the closest feature point in another frame is found as the matching point. The matching distance is measured using the Euclidean distance or other similarity metrics between descriptors;

[0050] 4. Calculate the offset between feature points:

[0051] For successfully matched feature point pairs, the offset between them can be calculated using the pixel coordinates of the matching points in the two images. Assuming that the successfully matched feature point pairs are (x1, y1) and (x2, y2), the offset (u, v) is:

[0052] u=x2-x1

[0053] v=y2-y1

[0054] According to the offset (u, v) obtained by matching the feature points, the leaf center position offset between frames can be calculated. Assuming that the leaf center position of the current frame is (x_prev, y_prev), the leaf center position after offset is:

[0055] New position: (x_curr,y_curr)=(x_prev+u,y_prev+v)

[0056] In the successfully matched leaf center position point pairs, the pixel position distance between the two frame point pairs can be calculated to determine whether they belong to the same point. The distance calculation can use Euclidean distance or other distance measurement methods;

[0057] Design an appropriate loss function so that the model can accurately predict the blade center position and blade orientation, considering the coordinate regression loss of the blade center position and the regression loss of the blade orientation angle;

[0058] S4, through steps S1 to S3, establish a model for detecting the leaf center and orientation of each visible leaf in the image;

[0059] In actual use, each frame uses the model to detect the current frame and obtain the center and orientation of each visible leaf;

[0060] S5. Calculate the feature points and corresponding descriptors of the two frames using sift / surf / orb feature points and descriptors between frames. After matching the features of the two frames, arrange the features in ascending order according to the matching distance, take out a pair of feature matching points with the shortest feature distance, and subtract the upper positions of this pair of feature points in each image to obtain the offsets of u and v of the pixel positions of the image, where uv represents the pixel position.

[0061] The SIFT algorithm is used to extract the feature points and corresponding descriptors of the two frames. For each feature point, the best matching feature point in the second frame is found in the first frame (with the smallest matching distance), specifically:

[0062] 1. Feature point sorting:

[0063] Arrange the matched feature points in ascending order according to the matching distance (Euclidean distance, Hamming distance, etc.) to find the best match.

[0064] 2. Calculate the offset:

[0065] From the sorted feature points, a pair of feature matching points with the shortest distance is selected and recorded as feature point 1 and feature point 2 respectively;

[0066] Assume that the coordinates of feature point 1 in the first frame are (x1, y1), and the coordinates of feature point 2 in the second frame are (x2, y2), then the offset is:

[0067] u=x2-x1

[0068] v=y2-y1

[0069] This offset can be regarded as the offset between two frames, because the machine sets up a camera above the crops to shoot the crops from a bird's-eye view. The machine's view can be approximately regarded as only a translation when it moves along the crops. The offset is calculated by taking only the point pair with the smallest feature instead of obtaining the homography matrix or motion transformation matrix between the two frames. This is because it can greatly reduce the amount of calculation and has extremely low requirements on the computing resources on the terminal side.

[0070] S6. After obtaining the inter-frame offset, add the offset to the leaf center position of the previous frame, which is equivalent to projecting it on the current frame. Calculate the pixel position distance between any two leaf centers projected on the current frame and the leaf centers detected by the current frame model to obtain a matrix C with the points of the previous frame and the points of the current frame as columns. The matrix content is the distance between the point pairs of the two frames. Arrange the distances in ascending order and take the point pairs starting from the minimum distance. The point pairs are not the feature point pairs of the previous step, but the point pairs of the positions of the leaf centers of the two frames. If the distance is less than the threshold, it is considered that the point pairs belong to the same point, and the point matching is successful and recorded as Mi;

[0071] The successfully matched point pairs no longer participate in the subsequent step of taking point pairs for distance comparison, that is, deleting all distances calculated by the matched point pairs in C, because the points in the previous frame correspond one-to-one with the points in the current frame, and there is no one-to-many or many-to-one situation;

[0072] S7, continue to take point pairs starting from the current minimum distance, repeat the steps until the distance taken is greater than the threshold, once it is greater than the threshold, it is considered that the remaining unmatched points of the previous frame have no matching relationship with the points of the current frame;

[0073] Among them, the points that were not matched in the previous frame are considered to be missed in the current frame, and the points that are not matched in the current frame are considered to be newly appeared leaf centers. New ids are assigned to the newly appeared leaf centers, and the leaf centers that are successfully matched use the previous leaf center ids and update the positions in the current frame. The corresponding directions of the leaf centers are also recorded.

[0074] S8. After the above steps, the position and corresponding orientation of each leaf center id in multiple frames are obtained. By taking out all orientations of a single leaf center in multiple frames, the degree of fluctuation of the leaf center orientation change can be determined.

[0075] If the current frame orientation fluctuates greatly relative to the historical orientation, it will not be used. Instead, the historical orientation will be used as output. The specific historical orientation value can be clustered and averaged, which is equivalent to taking the orientation that appears multiple times in the historical frame, thus obtaining a stable orientation of the leaf center.

[0076] S9, according to the output stable orientation and position of the leaf center, each leaf center is extended along the orientation by a straight line of a certain length, any two straight lines intersect to generate intersection points, and the intersection points within a certain range are averaged to obtain the final crop terminal bud position;

[0077] Because the leaves grow around the terminal bud, the leaves of the side buds will be covered by the leaves of the terminal bud and no intersection will be produced.

[0078] A crop root and stem top recognition device includes a cotton topping device, the cotton topping device includes a frame 100 and a visual recognition component 500, the visual recognition component 500 is fixedly installed on the side wall of the frame 100, and the visual recognition component 500 is used to recognize the spatial position of the cotton top;

[0079] The side wall of the frame 100 is fixedly connected with a mounting base 501 for mounting and supporting a binocular recognition camera 502 , and the top of the mounting base 501 is fixedly connected with a cable hub 503 , which is used for arranging and protecting cables on the binocular recognition camera 502 .

[0080] A storage medium for a crop root and stem top identification method comprises a master control machine, on which a computer program is designed and on which computer program instructions are stored. When the computer program instructions are executed, a crop root and stem top identification method is implemented.

[0081] In this embodiment, steps S1 and S2 involve data preparation and labeling, which are prerequisite steps for building a deep learning model and provide the model with the labeling information required for training; in step S3, deep learning technology is used to learn the labeling information, including the prediction of the leaf center position and the leaf orientation, which is a key step in building the model and ensures that the model can accurately predict this information; in step S4, a target detection model such as the YOLO series is selected, and angle prediction is added to make the model structure suitable for detecting the leaf center position and predicting the leaf orientation; in step S5, the SIFT / SURF / ORR algorithm is used to match feature points and calculate the offset between feature points, which is a key step for processing between frames and helps to track the continuity of the leaf center position. Steps S6 to S9 involve the calculation of inter-frame offsets, leaf center position matching, orientation stability judgment, and determination of the apical bud position. These steps combine feature point matching and historical information processing to ensure the accuracy and stability of the leaf center position and orientation.

Claims

1. A method for identifying the top of a crop root, characterized in that: The following steps are involved: S1. Using a camera to obtain images of crops with leaves, manually marking the images of crops with leaves, and marking the leaf centers using rectangular frames or dots; S2. Mark the leaf direction, and define the leaf direction as the extension line from the leaf center to the heart-shaped edge of the leaf base; S3, using deep learning technology to learn the annotation information in steps S1~S2; S4, through steps S1 to S3, establish a model for detecting the center and orientation of each visible leaf in the image; S5. Calculate the feature points and corresponding descriptors of two frames using sift / surf / orb feature points and descriptors between frames to obtain the offsets of u and v of the pixel positions of the images. S6. After obtaining the inter-frame offset, the leaf center position of the previous frame is added with the offset, which is equivalent to projecting it on the current frame. The pixel position distances of the leaf centers projected on the current frame and the leaf centers detected by the current frame model are calculated for any two of them to obtain a matrix C with the points of the previous frame and the points of the current frame as columns. The matrix content is the distance between the point pairs of the two frames; the distances are arranged in ascending order, and point pairs are taken from the minimum distance, where the point pair is not the feature point pair of the previous step, but the point pair of the position of the leaf centers of the two frames. If the distance is less than the threshold, it is considered that the point pair belongs to the same point, and the point matching is successful and recorded as Mi; S7, continue to take point pairs starting from the current minimum distance, and repeat the steps until the distance taken is greater than the threshold; S8. After the above steps, the position and corresponding orientation of each leaf center id in multiple frames are obtained. By taking out all orientations of a single leaf center in multiple frames, the degree of fluctuation of the leaf center orientation change can be determined. S9. According to the output stable orientation and position of the leaf center, each leaf center is extended along the orientation by a straight line of a certain length, any two straight lines intersect to generate intersection points, and the intersection points within a certain range are averaged to obtain the final crop terminal bud position.

2. A method for identifying the top of a crop root and stem according to claim 1, characterized in that: The S3 also uses the rectangular box detection in the target detection to enable the model to learn to predict the leaf center rectangular box and learn the angle information of the leaf orientation, so as to obtain a model that can output the leaf center position and the leaf orientation.

3. A method for identifying the top of a crop root and stem according to claim 1, characterized in that: The process of obtaining the offsets of u and v of the image pixel positions in S5 is as follows: after the features of the two frames are matched, the features are arranged in ascending order according to the matching distance, a pair of feature matching points with the shortest feature distance is taken out, and the upper positions of this pair of feature points in each image are subtracted to obtain the offsets of u and v of the image pixel positions, where uv represents the pixel position.

4. A method for identifying the top of a crop root and stem according to claim 1, characterized in that: The target detection method Yolo series is introduced in S3, specifically: Prepare images of crops with leaves and manually annotate them, using rectangular boxes or points to mark the position of the leaf center and the angle information of the leaf orientation; Select a deep learning model suitable for the target detection task and use the labeled crop image dataset to train the deep learning model. During the training process, the model needs to learn to recognize the rectangular box of the leaf center and predict the angle information of the leaf orientation.

5. A method for identifying the top of a crop root and stem according to claim 2, characterized in that: The process of marking the blade orientation in S3 is as follows: In the target detection task, the leaf center position is represented by the coordinates of the rectangular box. Assume that the leaf center position is the coordinates of the center point of the rectangular box (x_center, y_center), the width of the rectangular box is w, and the height is h. Leaf center position: (x_center, y_center) = (x_rect + 0.5 * w, y_rect + 0.5 * h) The blade orientation is expressed in degrees, Leaf angle: angle = arctan((y_leaf_base - y_leaf_tip) / (x_leaf_base -x_leaf_tip)) Among them, (x_leaf_base, y_leaf_base) is the coordinate of the heart-shaped edge of the leaf base, and (x_leaf_tip, y_leaf_tip) is the coordinate of the leaf center; When matching feature points between frames, the SIFT algorithm is used to extract feature points and descriptors, and then the offset between feature points is calculated by matching distance.

6. A method for identifying the top of a crop root and stem according to claim 1, characterized in that: The offset calculation process is as follows:

1. SIFT feature point extraction: The SIFT algorithm detects the key points of the image and generates a descriptor for each key point, which describes the characteristics of the area around the key point. a Feature descriptor calculation: For each extracted key point, the SIFT algorithm calculates a 128-dimensional descriptor that describes the characteristics of the area around the key point. The descriptor is calculated by the image gradient direction histogram around the key point. b Feature point matching: In the process of matching feature points between different frames, the nearest neighbor matching method is used, that is, for each feature point, the closest feature point in another frame is found as the matching point; c Calculate the offset between feature points: For successfully matched feature point pairs, the offset between them can be calculated using the pixel coordinates of the matching points in the two images.

7. A method for identifying the top of a crop root and stem according to claim 1, characterized in that: In S7, the points that were not matched in the previous frame are considered to be missed in the current frame, and the points that were not matched in the current frame are considered to be newly appeared leaf centers. A new id is assigned to the newly appeared leaf center, and the previous leaf center id is used for the successfully matched leaf center, and the position of the current frame is updated, and the corresponding direction of the leaf center is recorded together.

8. A crop root and stem top identification device, comprising a cotton topping device for implementing a crop root and stem top identification method according to any one of claims 1 to 7, characterized in that: The cotton topping device comprises a frame (100) and a visual recognition component (500), wherein the visual recognition component (500) is fixedly mounted on a side wall of the frame (100), and the visual recognition component (500) is used to recognize the spatial position of the cotton top; A mounting base (501) for mounting and supporting a binocular recognition camera (502) is fixedly connected to the side wall of the frame (100), and a cable hub (503) is fixedly connected to the top of the mounting base (501). The cable hub (503) is used for arranging and protecting cables on the binocular recognition camera (502).

9. A storage medium for a method for identifying the top of a crop root, characterized in that: It comprises a master control machine on which a computer program is designed, characterized in that computer program instructions are stored thereon, and when the computer program instructions are executed, a method for identifying the root and stem apex of a crop as described in any one of claims 1-7 is implemented.

Images