A method for identifying pruning points of tomato plant side branches

Abstract

The present invention discloses a method for identifying pruning points on tomato plant side branches, belonging to the technical field of branch pruning point identification. The method comprises the following steps: collecting images of tomato branches and establishing a library of tomato branch images in natural scenes; establishing and training a tomato plant side branch and main branch segmentation model based on a Mask R-CNN network; and pruning point identification: determining at which end of the side branch the pruning point is located in the image to be tested, and calculating the pruning point coordinates. The present invention utilizes the tomato plant side branch and main branch segmentation model based on a Mask R-CNN network to perform instance segmentation of tomato plant side branches and main branches. The method can separate the tomato plant side branches and main branches from leaves and backgrounds of similar colors. The method has the advantages of high segmentation accuracy, multiple applicable scenarios, and a small number of model parameters, laying the foundation for the application of tomato branch and leaf pruning robots.

Description

Technical Field

[0001] The invention belongs to the technical field of branch pruning point identification, in particular to a method for identifying pruning points of tomato plant side branches. Background Art

[0002] Tomato is one of the most common fruits and vegetables, and my country is a major producer and consumer of tomatoes in the world. Pruning branches and leaves is of great significance to the growth of tomato plants. Timely pruning of branches and leaves can improve the permeability of tomato plants, reduce the incidence of diseases and increase tomato yields. The leaves of tomato plants grow on the side branches. Currently, tomato branches and leaves are mainly pruned by manually cutting or breaking off the side branches. This not only wastes time and manpower, but may also cause damage to the tomato plants during the cutting or breaking process. Tomatoes are mainly grown on elevated platforms or in greenhouses. The wide row spacing between plants is conducive to the automated pruning of tomato plant branches and leaves. Therefore, the development of a tomato branch and leaf pruning robot is feasible and necessary. To realize the automated pruning of tomato branches and leaves, it is particularly important to identify the pruning points of the tomato plant side branches.

[0003] Currently, there is little research on identifying tomato pruning points. Chinese invention patent publication number CN109522901A discloses a method for identifying tomato plant stem edges based on edge duality. The method comprises segmenting a tomato plant color image, extracting continuous edges of the tomato plant, performing denoising, then sorting the tomato plant edges, performing edge type identification based on T1 neighborhood filtering, short edge segment filtering, and edge segmentation. Finally, based on the duality relationship between edges, tomato plant stem edge pairs are obtained. Chinese invention patent publication number CN112270708A discloses a method for identifying side branch points of fruit and vegetable plants based on different edge types. The method comprises first thresholding a fruit and vegetable plant color image, extracting horizontal and vertical edges from the binary image, and obtaining the branching points of the fruit and vegetable plants by obtaining the intersection of the processed horizontal and vertical edges. Finally, the side branch points are determined by the edge type of the branch point and the distance limit between the two points.

[0004] The main branches, side branches, leaves, and immature green fruit of a tomato plant are similar in color. Traditional image processing methods based on threshold features suffer from ineffective differentiation and segmentation, as well as poor applicability. However, instance segmentation models can be trained to detect and segment the main and side branches of a tomato plant. Currently, research on branch segmentation based on instance segmentation models is limited, and branch instance segmentation is a critical step in identifying pruning points for side branches.

[0005] Therefore, it is necessary to provide a method for identifying lateral branch pruning points of tomato plants. Summary of the Invention

[0006] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method for identifying pruning points of tomato plant side branches. This method uses a Mask R-CNN network to segment the main branches and side branches of tomato plants, solving the problem that traditional image processing methods based on threshold features cannot effectively segment the side branches and main branches of tomato plants from leaves and backgrounds of similar colors, resulting in difficulty in identifying pruning points.

[0007] The specific technical solutions adopted in the present invention are as follows:

[0008] The present invention provides a method for identifying lateral branch pruning points of a tomato plant, comprising the following steps:

[0009] S1. Collect images of tomato branches and build a tomato branch image library in natural scenes;

[0010] S2. Based on the tomato branch image library, establishing and training a tomato plant side branch and main branch segmentation model based on the Mask R-CNN network;

[0011] S3. Based on the tomato plant side branch and main branch segmentation model, determine at which end of the side branch the pruning point in the test image is located, and calculate the coordinates of the pruning point.

[0012] Preferably, the specific steps of step S1 are:

[0013] S11. Collect color images of tomato branches under different angles and illumination as a dataset;

[0014] S12, mirroring the image collected in step S11 and adding Gaussian noise to expand the data set;

[0015] S13. Based on the result of step S12, the tomato branch images in the dataset are processed, the images are adjusted to a unified standard format size, and then the dataset is divided into an image training set, a validation set, and a test set;

[0016] S14. Based on the result of step S13, perform instance segmentation and labeling on the main branches and side branches of the tomato plant.

[0017] Furthermore, the standard format size of the image in step S13 is 960×704.

[0018] Preferably, the specific steps of step S2 of establishing a Mask R-CNN network for segmenting tomato plant side branches and main branches are:

[0019] S21. Replace the backbone network ResNet50 / 100 of Mask R-CNN with MobileViT for feature extraction to obtain feature maps C1, C2, C3, C4, and C5.

[0020] S22, adding the ECA attention mechanism to the feature maps C3 and C4 to obtain a new feature map, and then inputting the obtained new feature map and the feature maps C2 and C5 into the feature pyramid network FPN for feature fusion;

[0021] S23. Based on the result of step S22, the feature map after feature fusion is sent to the region proposal network RPN to obtain the region of interest;

[0022] S24, mapping the region of interest to a feature map through a region of interest alignment layer to extract corresponding target features;

[0023] S25. Input the target features extracted in step S24 into the fully connected layer FC and the fully convolutional network layer FCN of the head network to obtain the target detection classification, predicted bounding box and mask.

[0024] Furthermore, in step S21, the MobileViT network has ten layers, the first layer is a 3×3 standard convolution, the second to sixth layers, the eighth layer and the tenth layer are all MV2 modules, the seventh and ninth layers are MVIT modules, MV2 is the module of the MobileNetV2 network, and MVIT is the module of the MobileViT network.

[0025] Furthermore, the feature maps C1, C2, C3, C4 and C5 in step S21 are the output results of the second layer, fifth layer, seventh layer, ninth layer and tenth layer of the MobileViT network respectively.

[0026] Furthermore, in step S22, the ECA attention mechanism directly performs global average pooling GAP, and then performs local cross-channel connection through adaptively selected fast one-dimensional convolution of size k. After that, the activation function Sigmoid is used to obtain the weight of each channel, and the original input feature is combined with the channel weight to obtain a feature with channel attention, where the value of k is expressed as:

[0027]

[0028] Where, |t| odd is the odd number closest to t, C is the current total number of channels, and γ and b are fixed to 2 and 1 respectively.

[0029] Furthermore, in step S25, the target detection classification is set to three categories: tomato plant side branches, main branches and background.

[0030] Preferably, in step S2, the Mask R-CNN network for segmenting tomato plant side branches and main branches is trained by loading the pre-trained weights of MobileViT for transfer learning.

[0031] Preferably, the specific steps of step S3 are:

[0032] S31. The image to be tested is input into the Mask R-CNN network for segmenting the side branches and main branches of the tomato plant. The range F near the main branch is calculated based on the output main branch bounding box:

[0033] F=[a1-70,a2+70]

[0034] Where a1 and a2 are the horizontal coordinates of the upper left corner and lower right corner of the main branch bounding box respectively;

[0035] S32. Perform edge extraction on each side branch mask binary image to obtain a side branch edge image E; obtain the coordinates of the side branch edge points in the image E by column, where the first edge point is the left end point L of the side branch, and the last edge point is the right end point R of the side branch. The coordinates (x, y) of the side branch edge points satisfy:

[0036] mask(x,y)=255

[0037] Where mask(x,y) represents the pixel value of the pixel with coordinates (x,y) in the side branch edge map E;

[0038] S33, determining whether the horizontal coordinate of the left end point L or the right end point R is within the range F obtained in S31, if the horizontal coordinate of the left end point L is within the range F, the side branch pruning point is at the left end of the side branch; if the horizontal coordinate of the right end point R is within the range F, the side branch pruning point is at the right end of the side branch;

[0039] S34. If the pruning point is at the right end of the side branch, count from the column where the right endpoint R is located in the edge graph E, and select two edge points B1 and B2 in the 15th column to the left to calculate the coordinates of the pruning point. If the pruning point is at the left end of the side branch, count from the column where the left endpoint L is located in the edge graph E, and select two edge points B1 and B2 in the 15th column to the right to calculate the coordinates of the pruning point. The side branch pruning point P(x0, y0) satisfies:

[0040]

[0041] Among them, x1 and y1 are the horizontal and vertical coordinates of edge point B1, and x2 and y2 are the horizontal and vertical coordinates of edge point B2.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1) The present invention uses the Mask R-CNN model to segment the main and side branches of tomato plants. It can separate the side branches and main branches of tomato plants from leaves and backgrounds of similar colors. It has the advantage of being applicable to many scenarios and facilitates the identification of side branch pruning points.

[0044] 2) This paper adopts MobileViT to replace the original Mask R-CNN model backbone network ResNet50 / 100, and adds the Efficient Channel Attention (ECA) mechanism to some feature maps, which reduces the number of model parameters while ensuring high segmentation accuracy, and the model has a faster running speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of the overall framework of a method for identifying lateral branch pruning points of a tomato plant according to the present invention.

[0046] Figure 2 It is a structural diagram of the Mask R-CNN model of the present invention.

[0047] Figure 3 This is a structural diagram of the ECA attention mechanism of the present invention.

[0048] Figure 4 This is a segmentation result diagram of a tomato plant main branch and side branch instance segmentation performed by the present invention based on the Mask R-CNN network.

[0049] Figure 5 This is a diagram showing the identification results of a method for identifying lateral branch pruning points of a tomato plant according to the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments. The technical features of each embodiment of the present invention may be combined accordingly, provided that there is no conflict between them.

[0051] The present invention provides a method for identifying lateral branch pruning points of tomato plants, the specific process is as follows: Figure 1 , including the following steps:

[0052] S1. Collect images of tomato branches and establish an image library of tomato branches in natural scenes.

[0053] In actual application, the steps are as follows:

[0054] S11. Data collection: Collect color images of tomato branches under different angles and illumination as a data set;

[0055] S12, data enhancement: mirror the images collected in S11 and add Gaussian noise to expand the dataset;

[0056] S13, establishing a database: processing the tomato branch images in the dataset, resizing the images to a unified standard format size of 960×704, and dividing the dataset into an image training set, a validation set, and a test set;

[0057] S14. Labeled data: Use the open source Labelme software to perform instance segmentation and labeling of the main branches and side branches of tomato plants.

[0058] S2. Build and train a tomato plant side branch and main branch segmentation model based on the Mask R-CNN network. The established Mask R-CNN network structure is shown in Figure 2 ,The training method is to load the pre-trained weights of MobileViT for transfer learning.

[0059] In practical application, the steps for establishing a tomato plant side branch and main branch segmentation model based on the Mask R-CNN network are as follows:

[0060] S21. Replace the Mask R-CNN backbone network ResNet50 / 100 with MobileViT for feature extraction. The MobileViT network has ten layers. The first layer is a 3×3 standard convolution. The second to sixth layers, the eighth layer, and the tenth layer are MV2 (MobileNetV2) modules. The seventh and ninth layers are MVIT (MobileViT) modules. The second, fifth, seventh, ninth, and tenth layers of the MobileViT network output feature maps C1, C2, C3, C4, and C5 respectively.

[0061] S22, add ECA (Efficient Channel Attention) attention mechanism to feature maps C3 and C4. The ECA structure is as follows Figure 3 , the ECA attention mechanism directly performs global average pooling (GAP), and then performs local cross-channel connection through adaptively selected fast one-dimensional convolution of size k. After that, the activation function Sigmoid is used to obtain the weight of each channel, and the original input features are combined with the channel weights to obtain features with channel attention, where the value of k is expressed as:

[0062]

[0063] Where, |t| odd is the odd number closest to t, C is the current total number of channels, and γ and b are fixed to 2 and 1 respectively.

[0064] Then the obtained new feature map and the remaining feature maps C2 and C5 are input into the feature pyramid network FPN (FeaturePyramid Network) for feature fusion;

[0065] S23, the feature map after feature fusion is sent to the region proposal network RPN (Region Proposal Network) to obtain the region of interest;

[0066] S24, mapping the region of interest to the feature map through the region of interest alignment layer to extract the corresponding target features;

[0067] S25. Input the target features extracted in S24 into the fully connected layer FC (Full Connection) and the fully convolutional network layer FCN (Full Convolution Network) of the head network to obtain the target detection classification, predicted bounding box and mask. The target detection classification is set to three categories: tomato plant side branches, main branches and background.

[0068] Figure 4 This is the segmentation result of a tomato plant main branch and side branch instance using the Mask R-CNN network. It can be seen that the segmentation effect of the main branch and side branch is good.

[0069] S3. Pruning point identification: The pruning point of a side branch can only be at the left or right end of the side branch, so it is necessary to first determine at which end of the side branch the pruning point is located, and then calculate the pruning point coordinates; if the pruning point is at the left end, calculate the pruning point coordinates at the left end of the side branch; if the pruning point is at the right end, calculate the pruning point coordinates at the right end of the side branch.

[0070] In actual application, the steps are as follows:

[0071] S31. The image to be tested is input into the Mask R-CNN network for tomato plant side branch and main branch segmentation. In order to identify more side branch pruning points, the range F near the main branch is calculated based on the output main branch bounding box:

[0072] F=[a1-70,a2+70]

[0073] Where a1 and a2 are the horizontal coordinates of the upper left corner and lower right corner of the main branch bounding box respectively;

[0074] S32. Use the canny edge operator to extract the edge of each side branch mask binary image to obtain a side branch edge image E; obtain the coordinates of the side branch edge points in the image E by column, where the first edge point is the side branch left end point L, the last edge point is the side branch right end point R, and the side branch edge point coordinates (x, y) satisfy:

[0075] mask(x,y)=255

[0076] Where mask(x,y) represents the pixel value of the pixel with coordinates (x,y) in the side branch edge map E;

[0077] S33. A side branch has a left endpoint and a right endpoint. The pruning point depends on which endpoint is near the main branch. Therefore, the pruning point can be determined by determining whether the horizontal coordinates of the side branch's left endpoint L and right endpoint R are within the range F obtained in S31. If the horizontal coordinate of the left endpoint is within the range F, the pruning point of the side branch is at the left end of the side branch; if the horizontal coordinate of the right endpoint is within the range F, the pruning point of the side branch is at the right end of the side branch.

[0078] S34. Calculate the coordinates of the pruning point: Considering the size of the end effector and the accuracy of pruning point recognition, if the pruning point is at the right end of the side branch, count from the column where the right end endpoint R is located in the edge graph E, and select two edge points B1 and B2 in the 15th column to the left to calculate the coordinates of the pruning point; if the pruning point is at the left end of the side branch, count from the column where the left end endpoint L is located in the edge graph E, and select two edge points B1 and B2 in the 15th column to the right to calculate the coordinates of the pruning point; the side branch pruning point P(x0, y0) satisfies:

[0079]

[0080] Where x1 and y1 are the horizontal and vertical coordinates of edge point B1, and x2 and y2 are the horizontal and vertical coordinates of edge point B2; Figure 5 This is the result of identifying the pruning points of the tomato plant's side branches. It can be seen that the pruning point identification is accurate and meets the actual operation requirements.

[0081] The present invention uses a tomato plant side branch and main branch segmentation model based on the Mask R-CNN network to perform instance segmentation of tomato plant side branches and main branches. It can separate the side branches and main branches of tomato plants from leaves and backgrounds of similar colors. It has the advantages of high segmentation accuracy, multiple applicable scenarios, and a small number of model parameters, laying the foundation for the application of tomato branch and leaf pruning robots.

[0082] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A method for identifying lateral branch pruning points of tomato plants, characterized in that: The following steps are involved: S1. Collect images of tomato branches and build a tomato branch image library in natural scenes; S2. Based on the tomato branch image library, establishing and training a tomato plant side branch and main branch segmentation model based on the Mask R-CNN network; S3. Based on the tomato plant side branch and main branch segmentation model, determine at which end of the side branch the pruning point in the image to be tested is located, and calculate the coordinates of the pruning point; The specific steps of step S2 to establish the Mask R-CNN network for segmenting the side branches and main branches of the tomato plant are: S21. Replace the backbone network ResNet50 / 100 of Mask R-CNN with MobileViT for feature extraction to obtain feature maps C1, C2, C3, C4, and C5. S22, adding the ECA attention mechanism to the feature maps C3 and C4 to obtain a new feature map, and then inputting the obtained new feature map and the feature maps C2 and C5 into the feature pyramid network FPN for feature fusion; S23. Based on the result of step S22, the feature map after feature fusion is sent to the region proposal network RPN to obtain the region of interest; S24, mapping the region of interest to a feature map through a region of interest alignment layer to extract corresponding target features; S25. Input the target features extracted in step S24 into the fully connected layer FC and the fully convolutional network layer FCN of the head network to obtain the target detection classification, predicted bounding box and mask.

2. A method for identifying lateral branch pruning points of a tomato plant according to claim 1, characterized in that: The specific steps of step S1 are: S11. Collect color images of tomato branches under different angles and illumination as a dataset; S12, mirroring the image collected in step S11 and adding Gaussian noise to expand the data set; S13. Based on the result of step S12, the tomato branch images in the dataset are processed, the images are adjusted to a unified standard format size, and then the dataset is divided into an image training set, a validation set, and a test set; S14. Based on the result of step S13, perform instance segmentation and labeling on the main branches and side branches of the tomato plant.

3. A method for identifying lateral branch pruning points of a tomato plant according to claim 2, characterized in that: The standard format size of the image in step S13 is 960×704.

4. A method for identifying lateral branch pruning points of a tomato plant according to claim 1, characterized in that: In step S21, the MobileViT network has ten layers, the first layer is a 3×3 standard convolution, the second to sixth layers, the eighth layer and the tenth layer are all MV2 modules, and the seventh and ninth layers are MVIT modules; the MV2 is a module of the MobileNetV2 network, and the MVIT is a module of the MobileViT network.

5. A method for identifying lateral branch pruning points of a tomato plant according to claim 4, characterized in that: The feature maps C1, C2, C3, C4 and C5 in step S21 are the output results of the second layer, fifth layer, seventh layer, ninth layer and tenth layer of the MobileViT network respectively.

6. A method for identifying lateral branch pruning points of a tomato plant according to claim 1, characterized in that: In step S22, the ECA attention mechanism directly performs global average pooling (GAP), and then performs local cross-channel connection through adaptively selected fast one-dimensional convolution of size k. After that, the activation function Sigmoid is used to obtain the weight of each channel, and the original input features are combined with the channel weights to obtain features with channel attention, where the value of k is expressed as: Where, |t| odd is the odd number closest to t, C is the current total number of channels, and γ and b are fixed to 2 and 1 respectively.

7. A method for identifying lateral branch pruning points of a tomato plant according to claim 1, characterized in that: In step S25, the target detection classification is set to three categories: tomato plant side branches, main branches and background.

8. The method for identifying lateral branch pruning points of a tomato plant according to claim 1, characterized in that: In step S2, the Mask R-CNN network for segmenting the side branches and main branches of the tomato plant is trained by loading the pre-trained weights of MobileViT for transfer learning.

9. The method for identifying lateral branch pruning points of a tomato plant according to claim 1, characterized in that: The specific steps of step S3 are: S31. The image to be tested is input into the Mask R-CNN network for segmenting the side branches and main branches of the tomato plant. The range F near the main branch is calculated based on the output main branch bounding box: F=[a1-70,a2+70] Where a1 and a2 are the horizontal coordinates of the upper left corner and lower right corner of the main branch bounding box respectively; S32. Perform edge extraction on each side branch mask binary image to obtain a side branch edge image E; obtain the coordinates of the side branch edge points in the image E by column, where the first edge point is the left end point L of the side branch, and the last edge point is the right end point R of the side branch. The coordinates (x, y) of the side branch edge points satisfy: mask(x,y)=255 Where mask(x,y) represents the pixel value of the pixel with coordinates (x,y) in the side branch edge map E; S33, determining whether the horizontal coordinate of the left end point L or the right end point R is within the range F obtained in S31, if the horizontal coordinate of the left end point L is within the range F, the side branch pruning point is at the left end of the side branch; if the horizontal coordinate of the right end point R is within the range F, the side branch pruning point is at the right end of the side branch; S34. If the pruning point is at the right end of the side branch, count from the column where the right endpoint R is located in the edge graph E, and select two edge points B1 and B2 in the 15th column to the left to calculate the coordinates of the pruning point. If the pruning point is at the left end of the side branch, count from the column where the left endpoint L is located in the edge graph E, and select two edge points B1 and B2 in the 15th column to the right to calculate the coordinates of the pruning point. The side branch pruning point P(x0, y0) satisfies: Among them, x1 and y1 are the horizontal and vertical coordinates of edge point B1, and x2 and y2 are the horizontal and vertical coordinates of edge point B2.

Images