Method for identifying and measuring tomato axillary bud in real time and positioning picking point

Abstract

The invention discloses a tomato axillary bud real-time identification, measurement and removal point positioning method. The method comprises the following steps: acquiring RGB and depth images of a tomato plant by using a mobile phone or RGB-D sensor equipment; a lightweight model is adopted to carry out instance segmentation on the RGB image, and axillary buds and branches are identified and segmented. And then, aligning the binary mask of the axillary bud with the depth image, constructing corresponding three-dimensional point cloud data, and calculating the length of the axillary bud and judging the removability based on whether the point cloud and the axillary bud region contain axillary bud bifurcation. And after determining that the axillary buds can be removed, further determining axillary bud removal points. By means of the method, agricultural management equipment can rapidly, automatically and accurately obtain the axillary bud length and judge the removability of the axillary bud, and therefore tomato pruning operation is efficiently completed.

Description

Technical Field

[0001] The present invention belongs to the technical fields of target segmentation and three-dimensional reconstruction, and particularly relates to a method for real-time recognition, measurement, and removal point positioning of tomato axillary buds. Background Art

[0002] The production of tomatoes not only involves the healthy growth of plants, but also includes high yield, excellent quality, and improved labor efficiency. During the cultivation process of tomatoes, the growth control of axillary buds is one of the important factors affecting plant morphology, yield, and quality. The growth points of axillary buds are located between the main stem and side branches of the plant, and their growth may lead to excessive branching, thereby affecting the efficiency of photosynthesis, the distribution of nutrients, and the development of fruits. Therefore, timely removal of axillary buds is a key link in improving tomato yield and fruit quality.

[0003] Traditional methods for removing axillary buds usually rely on manual operation. Farmers need to regularly inspect plants during the growing season and manually remove unnecessary axillary buds. Although this method is effective, it has problems such as high labor intensity, low efficiency, and being easily affected by environmental factors. In current image segmentation methods, pruning mostly involves identifying and removing side branches, lacking an effective method for segmenting axillary buds and judging the position of the removal point. Since axillary buds grow between the main stem and side branches, it is necessary to effectively distinguish axillary buds from side branches while identifying them. At the same time, whether to remove axillary buds is also related to the length of axillary buds.

[0004] Therefore, how to intelligently and automatically accurately segment axillary buds, obtain the length of axillary buds, and judge the axillary bud removal point is an urgent problem to be solved. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide a method for real-time recognition, measurement, and removal point positioning of tomato axillary buds.

[0006] In a first aspect, an embodiment of the present invention provides a method for real-time recognition, measurement, and removal point positioning of tomato axillary buds, the method comprising:

[0007] Obtaining RGB and depth images of a tomato plant using an image acquisition module.

[0008] Specifically, the image acquisition is obtained through an RGB-D sensor, a mobile phone camera, and a vision unit for identifying the positions of axillary buds and bifurcations in the image, and axillary bud images of the tomato plant containing different morphologies, different shooting distances, and different shooting angles are collected.

[0009] Further, after obtaining the RGB image and depth image of the tomato plant, it further includes: annotating axillary bud and bifurcation categories in each sample image through labelme, each marked point includes position and category information, and generating a mask file.

[0010] The RGB image is subjected to instance segmentation by a segmentation model to identify and segment axillary buds and bifurcations.

[0011] Specifically, the segmentation model is the lightweight structure model SenYolact improved based on the Yolact++ (You Only Look At CoefficienTs) model.

[0012] In some embodiments, the improvement includes: the SenYolact model reduces the number of model parameters and computational complexity by replacing the backbone network of the Yolact++ model with ShuffleNetv2.

[0013] Specifically, using the pre-trained ShuffleNetV2 model as a basis, the input image passes through the initial convolution and downsampling structure to obtain a preliminary feature map, and multi-level feature extraction is performed through three stages of stage2, stage3, and stage4. Finally, it is input into the final convolution layer. After the end of each stage and the output of the final convolution layer, the corresponding feature maps are extracted as outputs at different levels to form multi-scale features, which are used as the input of the FPN; the number of channels is adjusted by applying 1×1 convolution on the feature map, and the spatial resolution of the feature map is adapted to ensure that the output of the backbone network is consistent with the expected input of the FPN.

[0014] In some embodiments, the improvement further includes: adding an ECA (Efficient Channel Attention) mechanism to the network structure of the Yolact++ model to help the model better distinguish the differences between axillary buds and surrounding leaves and plant backgrounds.

[0015] Specifically, the ECA module is inserted into the network module before the c3 and c4 feature maps enter the FPN; in the backbone.py file defining the model, the ECA module class is written, and the ECA is instantiated when the main model is constructed; in the forward propagation of the main model, first, the c2, c3, c4, and c5 feature maps are obtained through the backbone network; before the c3 and c4 feature maps are passed into the FPN, ECA processing is performed, and the ECA-processed c3 and c4 feature maps are input into the FPN together with the original c2 and c5 feature Figure 1 maps; where the c2, c3, and c4 feature maps are the feature maps extracted by stage2, stage3, and stage4 respectively, and c5 is the feature map output by the final convolution layer.

[0016] In some embodiments, the improvement further includes: replacing Fast NMS with Soft-NMS in the Yolact++ model to reduce the missed detection problem caused by high-overlap prediction boxes during the identification and segmentation of axillary buds.

[0017] Specifically, construct a function that conforms to the Yolact++ model to call the Soft-NMS algorithm, and ensure that the function can accept prediction boxes and confidence scores, and return the boxes, scores, and indices after Soft-NMS processing; in the yolact.py file, find the location where FastNMS is called and replace it with a Soft-NMS call; ensure that the Soft-NMS function can be correctly called during the forward propagation of the model, and select the corresponding boxes and labels according to the output of Soft-NMS.

[0018] Align the axillary bud and bifurcation binary masks with the depth image, map the binary masks into the depth image, construct a three-dimensional point cloud model of axillary buds and bifurcations, calculate the axillary bud length based on the point cloud, and filter axillary buds whose axillary bud lengths meet the removal criteria.

[0019] Specifically, the calculation of the axillary bud length based on the point cloud includes: filtering the noise of the axillary bud point cloud to remove abnormal points; applying a smoothing algorithm to reduce the discreteness of the point cloud to improve the accuracy of the main direction extraction; using PCA analysis to obtain the main growth direction of the axillary bud, and then projecting all points in the axillary bud point cloud onto the main direction vector to obtain the projected coordinates. In the projected coordinates, calculate the Euclidean distance between the points with the maximum and minimum projected values in the main growth direction of the axillary bud to obtain the axillary bud length.

[0020] Specifically, the filtering of axillary buds whose axillary bud lengths meet the removal criteria includes: comparing the calculated axillary bud length with the set threshold Bud best If the obtained axillary bud length meets the removal criteria, further locate the position of the removal point; if the length does not meet the removal criteria, mark this axillary bud as skipped.

[0021] Filter axillary buds with axillary bud bifurcation masks; calculate the bottom points of each axillary bud three-dimensional point cloud and the bifurcation centroids of each bifurcation three-dimensional point cloud, and filter the best matching relationship through the distance and direction between the bottom points and the bifurcation centroids. Select the bottom points that match the bifurcation centroids as the axillary bud removal points to obtain the spatial positions of the axillary bud removal points, and control the device to complete the tomato axillary bud removal operation.

[0022] Specifically, the filtering of axillary buds with axillary bud bifurcation masks includes: further determining whether there is a bifurcation mask for axillary buds that meet the removal criteria: if there is, it is considered that the axillary bud structure is complete and can be removed; if there is no axillary bud bifurcation mask, the axillary bud is incomplete and marked as skipped.

[0023] Specifically, calculating the bottom points of each axillary bud three-dimensional point cloud and the centroid of each bifurcation three-dimensional point cloud, and screening the best matching relationship through the distance and direction between the bottom points and the centroid of the bifurcation includes: for each axillary bud three-dimensional point cloud, traversing along the direction of the secondary principal component of the three-dimensional point cloud to find the point farthest from the centroid of the axillary bud as the bottom point, where the direction of the secondary principal component is the opposite direction of the main growth direction of the axillary bud, and the centroid of the axillary bud is obtained by the average value of the three-dimensional coordinates of the axillary bud; for each bifurcation three-dimensional point cloud, calculating the centroid of the bifurcation through the average value of the three-dimensional coordinates of the bifurcation; for each bottom point of the axillary bud, calculating the Euclidean distance between it and all the centroids of the bifurcations; calculating the direction vector between the bottom point and the centroid of the bifurcation as the growth direction vector of the centroid of the bifurcation, taking the opposite direction of the main growth direction of the axillary bud as the growth direction vector of the bottom point of the axillary bud, and calculating the included angle between the growth direction vectors of the bottom point of the axillary bud and the centroid of the bifurcation; based on the Euclidean distance between the bottom point of the axillary bud and the centroid of the bifurcation and the included angle between the growth direction vectors of the bottom point of the axillary bud and the centroid of the bifurcation, screening the centroid of the bifurcation with the smallest distance and included angle to match the bottom point of the axillary bud.

[0024] In a second aspect, an embodiment of the present invention provides a processor for running a program, where the program, when run, is used to execute the method for real-time identification, measurement, and removal point positioning of tomato axillary buds described above.

[0025] In a third aspect, an embodiment of the present invention provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to cause the machine to execute the method for real-time identification, measurement, and removal point positioning of tomato axillary buds described above.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] The present invention provides a method for real-time identification, measurement, and removal point positioning of tomato axillary buds. By introducing technologies such as 3D reconstruction technology and image segmentation models, the present invention realizes the real-time identification, measurement, and removal point positioning of tomato axillary buds. Through the improved SenYolact segmentation model, based on the traditional Yolact++ model, lightweight and performance optimization are carried out. By replacing the backbone network with ShuffleNetv2, the amount of calculation and the number of parameters are reduced, thereby improving the running efficiency, while ensuring high-precision output of the model under low hardware requirements. In addition, the ECA mechanism and Soft-NMS are introduced, effectively reducing the problems of missed detection and false detection caused by high-overlap prediction boxes, which makes the segmentation of axillary buds and bifurcations more accurate. At the same time, different from the prior art that mainly relies on two-dimensional images for axillary bud identification, the present invention combines 3D point cloud data, which can not only obtain the spatial position information of tomato plants, but also accurately analyze the main growth direction and length of axillary buds. By aligning the binary masks of axillary buds and bifurcations with the depth map, a 3D point cloud model is further constructed. This method can effectively identify the spatial position of axillary buds, providing a more accurate basis for the positioning of removal points. By calculating the length of axillary buds based on point clouds and comparing it with the set removal criteria, it can be automatically judged whether the axillary buds meet the removal conditions. Traditional tomato axillary bud removal methods rely on a large amount of manual intervention, which is both time-consuming and laborious, and is also prone to affecting the effect due to improper operation. In contrast, the present invention can automatically identify and accurately locate axillary bud removal points in a complex environment, significantly improving the removal efficiency. In addition, since the system can work stably for a long time, avoiding the high intensity and instability of manual labor, the production cost is further reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0029] Figure 1 is a flowchart of the method for real-time identification, measurement, and removal point positioning of tomato axillary buds provided by the embodiments of the present invention;

[0030] Figure 2 is a structural diagram of the Yolact++ model provided by the embodiments of the present invention;

[0031] Figure 3 is a schematic diagram of axillary bud mask and point cloud provided by the embodiments of the present invention; wherein Figure 3 in (a) is a schematic diagram of the axillary bud mask, Figure 3 in (b) is a schematic diagram of the axillary bud point cloud;

[0032] Figure 4 It is a schematic diagram for positioning axillary buds and picking points; among them Figure 4 (a) in it is a schematic diagram for positioning axillary buds and bifurcations, Figure 4 (b) in it is a schematic diagram for calculating the direction difference between the bottom endpoint and the bifurcation mass point;

[0033] Figure 5 It is a flowchart for judging axillary bud removal points provided by an embodiment of the present invention;

[0034] Figure 6 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. Specific embodiments

[0035] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0036] It should be noted that, without conflict, the features in the following embodiments and implementation manners can be combined with each other.

[0037] As Figure 1 shown, the present invention provides a method for real-time recognition, measurement and removal point positioning of tomato axillary buds, and the method includes the following steps:

[0038] Step S1, use an image acquisition module to obtain RGB and depth images of a tomato plant;

[0039] It should be noted that through the image acquisition module, axillary bud images of the tomato plant with different forms, different shooting distances, and different shooting angles are collected; labelme is used to label axillary buds and bifurcation categories in each sample image, and each marked point includes position and category information, and a mask file is generated. The summary data is divided into a training set, a test set and a validation set according to a ratio of 7:2:1.

[0040] Step S2, perform instance segmentation on the RGB image through a segmentation model to identify and segment axillary buds and bifurcations;

[0041] It should be noted that the segmentation model is the lightweight structural model SenYolact improved based on the Yolact++ (You Only Look At CoefficienTs) model; the SenYolact model reduces the model parameters and computational complexity by replacing the backbone network of the Yolact++ model with ShuffleNetv2; an ECA (Efficient Channel Attention) mechanism is added to distinguish the differences between axillary buds and surrounding leaves and plant backgrounds; Fast NMS is replaced with Soft-NMS to reduce the missed detection problems caused by high-overlap prediction boxes during the recognition and segmentation of axillary buds.

[0042] Step S3, align the binary masks of axillary buds and bifurcations with the depth image, construct a three-dimensional point cloud model of axillary buds and bifurcations, calculate the length of axillary buds based on the point cloud, and judge the removability.

[0043] It should be noted that the main growth direction of axillary buds is obtained by PCA analysis, and then all points in the axillary bud point cloud are projected onto the main direction vector to obtain the length of axillary buds; the calculated length of axillary buds is compared with the set threshold Bud best If the obtained length of axillary buds meets the removal standard, further locate the position of the removal point; if the length does not meet the removal standard, mark this axillary bud as skipped.

[0044] Step S4, further determine the removal point of the axillary bud based on the judged removable axillary bud point cloud, obtain the spatial position of the axillary bud removal point, and control the device to complete the tomato axillary bud removal operation.

[0045] It should be noted that for axillary buds that meet the removal standard, further judge whether there is a bifurcation mask: if it exists, it is considered that the axillary bud structure is complete and can be removed; if there is no axillary bud bifurcation mask, the segmentation of the axillary bud is incomplete, and this axillary bud is marked as skipped; for each axillary bud three-dimensional point cloud, calculate the bottom point P b for one-to-one matching with the bifurcation centroid to verify whether the axillary bud is completely segmented; for each bifurcation three-dimensional point cloud, calculate the centroid C Branc h for one-to-one matching with the bottom point of the axillary bud to verify whether the axillary bud is completely segmented; obtain the Euclidean distance d ij ; according to the bottom point P b of the axillary bud and the bifurcation centroid C Branc h, calculate the direction vector from the bottom point to the centroid Obtain the bottom point P b of the axillary bud and the bifurcation centroid C Branc h to obtain the direction difference; through the distance and direction information of the bottom point P b of the axillary bud and the bifurcation centroid C Branc h, jointly screen the best matching relationship, that is, the bifurcation corresponding to the axillary bud; if this axillary bud is associated with the best-matched bifurcation centroid jmatc If h, the axillary bud removal point is the bottom point P of the axillary bud b .

[0046] In step S2, as Figure 2 shown, during the training of the Yolact++ model, the backbone network module is replaced with ShuffleNetv2 to reduce the model's parameter quantity and computational complexity; the ECA (Efficient Channel Attention) mechanism is added to distinguish the differences between axillary buds and surrounding leaves and plant backgrounds; Fast NMS is replaced with Soft-NMS to reduce the missed detection problems caused by high-overlap prediction boxes during the recognition and segmentation of axillary buds.

[0047] Using the pre-trained ShuffleNetV2 model as a basis, it is decomposed into multiple feature extraction stages, including the initial convolution and downsampling structure, stage2, stage3, stage4, and the final convolution layer. The input image is first passed through the initial convolution and downsampling structure to obtain a preliminary feature map; then it is successively passed through stage2, stage3, and stage4 for multi-level feature extraction. After each stage, the corresponding feature map is extracted as the output of different levels to form multi-scale features (c2, c3, c4, c5), providing input for the subsequent construction of the Feature Pyramid Network (FPN). The number of channels is adjusted by applying 1×1 convolution on the feature map, and the spatial resolution of the feature map is adapted to ensure that the output of the backbone network is consistent with the input expectation of the FPN; in the original configuration of YOLACT++, the backbone network part is modified to the above-mentioned adapted ShuffleNetV2 backbone network, so that the generated multi-scale feature maps are directly passed into the FPN and subsequent original modules; the replaced YOLACT++ model is used for training and verification.

[0048] During the training of the Yolact++ model, the ECA (Efficient Channel Attention) mechanism is added to the network structure to help the model better distinguish the differences between axillary buds and surrounding leaves and plant backgrounds; the ECA module is inserted into the network module before the c3 and c4 feature maps enter the FPN; in the model definition backbone network file, the ECA module class is written, and the ECA is instantiated during the construction of the main model; the backbone network features are obtained. In the forward propagation of the main model, first, the c2, c3, c4, and c5 features are obtained through the backbone network. Before passing c3 and c4 into the FPN, ECA processing is performed on them, and finally, the processed c3 and c4 features are input into the FPN together with the original c2 and c5; the tuning during the training and verification process is observed for the axillary bud recognition effect.

[0049] During the training of the Yolact++ model, Fast NMS is replaced with Soft-NMS to reduce the problem of missed detections caused by high-overlap prediction boxes when identifying and segmenting axillary buds. First, construct a call to the Soft-NMS algorithm that conforms to the Yolact++ model. Modify the Yolact++ main model to use Soft-NMS. In the yolact.py file, find the location where Fast NMS is called and replace it with a call to Soft-NMS. Ensure that the model can correctly call the Soft-NMS function during the forward propagation process and select the corresponding boxes and labels based on the output of Soft-NMS. Name the modified Yolact++ as SenYlact.

[0050] As Figure 3 shown, use SenYlact to infer the RGB image and output the class label and segmentation mask of each instance. The masks are divided into axillary buds and axillary bud bifurcations. Separate the masks output by the SenYlact model by category to obtain the mask sets of axillary buds and bifurcations respectively. Align the masks with the depth map, and through mask mapping, map the segmented axillary bud masks to the depth map to generate axillary bud and bifurcation point clouds.

[0051] For axillary bud length measurement, first filter the noise of the axillary bud point cloud to remove abnormal points. Then apply a smoothing algorithm to reduce the discreteness of the point cloud to improve the accuracy of the main direction extraction. Use PCA analysis to obtain the main growth direction of the axillary bud. By constructing a data matrix, form a data matrix of N×3 with the three-dimensional coordinates (x, y, z) of the axillary bud point cloud, where N is the number of points. The first principal component direction of PCA is the main growth direction of the axillary bud, that is, the extension trend of the axillary bud from the bifurcation to the top. Use the main direction vector obtained in the PCA analysis to project all points in the axillary bud point cloud onto the main direction vector to obtain the projected coordinates. In the projected coordinates, find the maximum t max and minimum t min positions, and calculate the Euclidean distance between the two, which is the length Bud h of the axillary bud.

[0052] Construct a data matrix, represent the three-dimensional coordinates of each point in the axillary bud point cloud as a data matrix of N×3, where N is the number of points, and each row of the matrix represents the (x, y, z) coordinates of a point.

[0053]

[0054] Calculate the covariance matrix of the point cloud data:

[0055]

[0056] where P i =(x i ,yi , z i ) is the i-th point, is the mean vector of the point cloud.

[0057] Perform eigenvalue decomposition on the covariance matrix to obtain three eigenvalues and the corresponding eigenvectors. The first principal component direction is the main growth direction of the axillary bud.

[0058] Cov(X)v = αv

[0059] where v is the eigenvector, β is the eigenvalue, and the first principal component direction corresponds to the eigenvector of the largest eigenvalue.

[0060] Project all the points in the axillary bud point cloud onto the main direction vector of the axillary bud growth. By calculating the inner product of the coordinates of each point with the main direction vector, the projected coordinates are obtained:

[0061] The main direction vector of the axillary bud is v growth =(v x , v y , v z ), then the projected coordinates of each point can be calculated by the following formula:

[0062] t i = P i ·v growth

[0063] where t i is the projected value of the i-th point on the main direction, and P i is the three-dimensional coordinate of the i-th point.

[0064] Among the projected coordinates, find the maximum projected value t max and the minimum projected value t min , which correspond to the top and bottom positions of the axillary bud respectively.

[0065] The length Bud h of the axillary bud is equal to the distance between the maximum value t max and the minimum value t min of the projected coordinates:

[0066] Bud h = |t max - t min |

[0067] In step S4, as Figure 4 and Figure 5 shown, determine the position of the axillary bud removal point.

[0068] Compare the calculated length Bud h of the axillary bud with the set threshold Bud best , Budbest Set the range of the removable length according to the tomato variety and management method, usually set at 3 - 7 cm. The specific threshold can be determined through experiments or adjusted according to the actual situation of tomato growth;

[0069] Bud h ≥Bud best : The axillary bud reaches the removable length, and the position of the removal point can be further located;

[0070] Bud h <Bud best : Mark and skip this axillary bud. The length of the axillary bud does not meet the removal standard, the axillary bud is not completely segmented, or the growth length is insufficient;

[0071] Judge whether there is an axillary bud bifurcation mask. If it exists, the axillary bud structure is considered complete; if there is no axillary bud bifurcation mask, the axillary bud is incomplete, indicating that there is no bifurcation in the segmented axillary bud area (the detected image does not contain the bifurcation point of this axillary bud, missing), or the bifurcation is blocked by leaves or the stem and cannot be recognized. Mark and skip this axillary bud and change the viewing angle to re - detect.

[0072] For each axillary bud three - dimensional point cloud, calculate the bottom - end point P b To perform one - to - one matching with the bifurcation centroid to verify whether the axillary bud is completely segmented; find the point farthest from the axillary bud centroid along the direction of the secondary principal component. The direction of the secondary principal component is the opposite direction of the main direction of the axillary bud, that is, - V1. Define the bottom - end point as P b =(x b ,y b ,z b ).

[0073] Locate the axillary bud centroid C Bud , the centroid of the axillary bud is the geometric center representing the position of the axillary bud, and is obtained by calculating the mean value of the three - dimensional coordinates of the axillary bud:

[0074]

[0075] Among them, N’ is the number of points of this axillary bud, and the point coordinates in the axillary bud three - dimensional point cloud are (x i ′,y i ′,z i ′);

[0076] For each bifurcation three - dimensional point cloud, calculate the bifurcation centroid C Branch To perform one - to - one matching with the axillary bud bottom - end point to verify whether the axillary bud is completely segmented;

[0077] Locate the bifurcation centroid C Branc h, the centroid of the bifurcation is the geometric center representing the position of the bifurcation, and is obtained by calculating the mean value of the three - dimensional coordinates of the bifurcation:

[0078]

[0079] Among them, N” is the number of points of the fork, and the point coordinates in the forked three-dimensional point cloud are (x i ″, y i ″, z i ″);

[0080] Obtain the Euclidean distance d between the bottom point of the axillary bud and the centroid of the fork ij ; For each bottom point of the axillary bud, calculate the Euclidean distance between it and the centroids of all forks:

[0081]

[0082] Among them, d ij is the distance between the bottom point i of the axillary bud and the centroid of the j-th fork, (x, y, z) are the corresponding three-dimensional coordinates, the subscript base represents the bottom point, and the subscript centroid represents the centroid of the fork. According to the bottom point P of the axillary bud b and the centroid C of the fork Branch Calculate the direction vector from the bottom point to the centroid of the fork

[0083]

[0084] For each pair of the bottom point of the axillary bud and the centroid of the fork, calculate the included angle (direction difference) between their growth directions:

[0085]

[0086] Among them, the growth direction vector of the bottom point of the axillary bud, is the growth direction vector of the centroid of the fork. θ is the included angle between the growth directions of the bottom point of the axillary bud and the centroid of the fork, and is used as the direction difference between the bottom point P b and the centroid C of the fork Branch .

[0087] For the bottom point of the axillary bud, the growth direction vector should be the reverse of the growth direction of the axillary bud, that is:

[0088]

[0089] Among them, is the unit vector of the growth direction of the axillary bud.

[0090] If the direction difference is less than a certain set threshold (for example, θ < θ threshold , such as 90°), then it is considered that the bottom point of the axillary bud and the centroid of the fork are possible matching pairs. The threshold can be specifically defined according to the growth characteristics of tomato plants of different varieties and experimental data;

[0091] Through the bottom endpoint P b and the centroid C of the fork Branch to jointly screen the best matching relationship, that is, the fork corresponding to the axillary bud; when calculating the distance between the bottom endpoint of the axillary bud and the centroid of the fork, the geometric distance and direction consistency should be considered simultaneously. If two points are close in distance and similar in growth direction, they are more likely to be matched;

[0092] Design a comprehensive scoring mechanism to combine the distance and direction information and define the matching score:

[0093]

[0094] where d ij is the Euclidean distance between the bottom endpoint i of the axillary bud and the centroid j of the fork, and θ ij is the direction difference (the angle between the growth direction vectors of the bottom endpoint of the axillary bud and the centroid of the fork), and γ represents a hyperparameter used to adjust the influence of the direction information. The term is weighted according to the direction difference, and it is inclined to select a match with a smaller direction difference;

[0095] For each bottom endpoint of the axillary bud, select a centroid of the fork with the smallest distance and the highest direction consistency as its matching centroid of the fork:

[0096]

[0097] where j match is the centroid j of the fork that matches the bottom endpoint i of the axillary bud, θ threshold and d threshold are the thresholds of the distance and angle set according to the specific situation to control the matching accuracy.

[0098] If this axillary bud is associated with the centroid j of the fork with the best match match , then the axillary bud removal point is the bottom endpoint P of the axillary bud b , and the control device removes the axillary bud.

[0099] Correspondingly, the present application also provides a processor for running a program, and when the program is run, it is used to execute the method for real-time recognition, measurement, and removal point positioning of tomato axillary buds.

[0100] Correspondingly, the present application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method for real-time recognition, measurement, and removal point positioning of tomato axillary buds as described above. As Figure 6 shown, it is a hardware structure diagram of any device with data processing capabilities where the method for real-time recognition, measurement, and removal point positioning of tomato axillary buds provided by the embodiment of the present invention is located. Except forFigure 6 In addition to the processor, memory, and network interface shown, any device with data processing capabilities where the device in the embodiment is located usually includes other hardware according to the actual functions of the device with data processing capabilities, which will not be elaborated here.

[0101] Correspondingly, the present application also provides a machine-readable storage medium storing instructions for causing a machine to execute the method for real-time identification, measurement, and removal point positioning of tomato axillary buds. The machine-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The machine-readable storage medium may also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the machine-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The machine-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0102] The above embodiments are only used to illustrate the design concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made according to the principles and design concepts disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for real-time identification, measurement and removal point positioning of tomato axillary buds, characterized in that: The following steps are involved: Obtain RGB images and depth images of tomato plants through image acquisition; Perform instance segmentation on RGB images through segmentation models, identify and segment axillary buds and bifurcations; The binary masks of axillary buds and bifurcations are aligned with the depth image, and the binary masks are mapped to the depth image to construct a three-dimensional point cloud model of axillary buds and bifurcations. The length of axillary buds is calculated based on the point cloud, and the axillary buds whose length meets the removal criteria are selected; Screening axillary buds with the presence of axillary bud bifurcation mask; Calculate the bottom endpoint of each axillary bud three-dimensional point cloud and the bifurcation centroid of each bifurcation three-dimensional point cloud, select the best matching relationship through the distance and direction between the bottom endpoint and the bifurcation centroid, select the bottom endpoint with the bifurcation centroid as the axillary bud removal point, and control the device to complete the tomato axillary bud removal operation according to the spatial position of the axillary bud removal point.

2. A method for real-time identification, measurement and removal point positioning of tomato axillary buds according to claim 1, characterized in that: The image acquisition is obtained through an RGB-D sensor, a mobile phone camera and a visual unit for identifying the positions of axillary buds and forks in the image, and axillary bud images of the tomato plant with different forms, different shooting distances, and different shooting angles are collected; after obtaining the RGB image and depth image of the tomato plant, it also includes: marking the axillary bud and fork categories in each sample image through labelme, each marking point includes position and category information, and a mask file is generated.

3. The method for real-time identification, measurement and removal point positioning of tomato axillary buds according to claim 1, characterized in that: The segmentation model uses SenYolact, which is improved based on the Yolact++ model. The improvement includes: replacing the backbone network module of the Yolact++ model with ShuffleNetv2 to reduce the number of model parameters and the amount of calculation, specifically: Using the pre-trained ShuffleNetV2 model as the basis, the input image is passed through the initial convolution and downsampling structure to obtain the preliminary feature map, and multi-level feature extraction is performed through the three stages of stage2, stage3, and stage4. Finally, the final convolution layer is input. After each stage is completed and the final convolution layer is output, the corresponding feature map is extracted as the output of different levels to form multi-scale features as the input of FPN; By applying 1×1 convolution on the feature map, the number of channels is adjusted and the spatial resolution of the feature map is adapted to ensure that the output of the backbone network is consistent with the input expectations of the FPN.

4. The method for real-time identification, measurement and removal point positioning of tomato axillary buds according to claim 3, characterized in that: The improvements also include: adding an ECA mechanism to the network structure of the Yolact++ model to help the model distinguish the differences between axillary buds and surrounding leaves and plant background, specifically: Insert the ECA module into the network module before the c3 and c4 feature maps enter the FPN; In the model definition backbone.py file, write the ECA module class and instantiate ECA when the main model is built; In the forward propagation of the main model, the c2, c3, c4, and c5 feature maps are first obtained through the backbone network; before the c3 and c4 feature maps are input into the FPN, ECA processing is performed, and the ECA-processed c3 and c4 feature maps are input into the FPN together with the original c2 and c5 feature maps; Among them, the c2, c3, and c4 feature maps are feature maps extracted by stage2, stage3, and stage4 respectively, and c5 is the feature map output by the final convolutional layer.

5. The method for real-time identification, measurement and removal point positioning of tomato axillary buds according to claim 3, characterized in that: The improvements also include: replacing Fast NMS with Soft-NMS in the Yolact++ model to reduce the problem of missed detection caused by high-overlap prediction boxes when identifying and segmenting axillary buds, specifically: Build a Yolact++ model that calls the Soft-NMS algorithm and ensure that the function can accept the predicted box and confidence score, and return the box, score, and index processed by Soft-NMS; In the yolact.py file, find the location where Fast NMS is called and replace it with Soft-NMS. Make sure that the model can correctly call the Soft-NMS function during the forward propagation process and select the corresponding box and label based on the output of Soft-NMS.

6. The method for real-time identification, measurement and removal point positioning of tomato axillary buds according to claim 1, characterized in that: The point-based computing of axillary bud length includes: The noise of the axillary bud point cloud is filtered to remove abnormal points; the smoothing algorithm is applied to reduce the discreteness of the point cloud to improve the accuracy of main direction extraction; PCA analysis is used to obtain the main direction of axillary bud growth, and then all points in the axillary bud point cloud are projected onto the main direction vector to obtain the projected coordinates; In the projected coordinates, the Euclidean distance between the points with the maximum and minimum projection values ​​in the main direction of axillary bud growth was calculated as the axillary bud length.

7. The method for real-time identification, measurement and removal point positioning of tomato axillary buds according to claim 1, characterized in that: The step of calculating the bottom endpoint of each axillary bud three-dimensional point cloud and the bifurcation centroid of each bifurcation three-dimensional point cloud, and selecting the best matching relationship by the distance and direction between the bottom endpoint and the bifurcation centroid, includes: For each axillary bud three-dimensional point cloud, based on the three-dimensional point cloud, traverse along the PCA secondary principal component direction to find the point farthest from the axillary bud centroid as the bottom endpoint, wherein the secondary principal component direction is the opposite direction of the axillary bud growth main direction, and the axillary bud centroid is obtained by the average value of the axillary bud three-dimensional coordinates; For each bifurcation 3D point cloud, the bifurcation centroid is calculated by the mean of the bifurcation 3D coordinates; For each axillary bud bottom endpoint, the Euclidean distance between it and all bifurcation centroids is calculated; the direction vector between the bottom endpoint and the bifurcation centroid is calculated as the growth direction vector of the bifurcation centroid, the opposite direction of the main growth direction of the axillary bud is used as the growth direction vector of the axillary bud bottom endpoint, and the angle between the growth direction vectors of the axillary bud bottom endpoint and the bifurcation centroid is calculated; based on the Euclidean distance between the axillary bud bottom endpoint and the bifurcation centroid, and the angle between the growth direction vectors of the axillary bud bottom endpoint and the bifurcation centroid, the bifurcation centroid with the smallest distance and angle is selected to match the axillary bud bottom endpoint.

8. The method for real-time identification, measurement and removal point positioning of tomato axillary buds according to claim 7, characterized in that: The method of selecting the fork centroid with the smallest distance and angle to match the axillary bud bottom endpoint based on the Euclidean distance between the axillary bud bottom endpoint and the fork centroid and the angle between the growth direction vectors of the axillary bud bottom endpoint and the fork centroid includes: calculating a matching score, Among them, d ij is the Euclidean distance between the axillary bud end point i and the bifurcation centroid j, θ ij is the angle between the growth direction vector of the axillary bud bottom endpoint and the bifurcation centroid, γ represents a hyperparameter used to adjust the influence of directional information; Select the bifurcation centroid with the smallest distance and angle to match the axillary bud bottom endpoint: Among them, j match is the centroid j of the bifurcation matching the end point i of the axillary bud, θ threshold and d threshold is the threshold of the distance and angle.

9. A machine-readable storage medium having instructions stored thereon, characterized in that: The instruction is used to make the machine execute: the method for real-time identification, measurement and removal point positioning of tomato axillary buds as described in any one of claims 1-8.

10. A processor, characterized in that: Used to run a program, wherein the program, when run, is used to execute: the method for real-time identification, measurement and removal point positioning of tomato axillary buds as described in any one of claims 1-8.

Images