Tomato plant segmentation method based on improved yolov8s-seg model

Abstract

The invention discloses a tomato plant segmentation method based on an improved yolov8s-seg model, and belongs to the field of crop machine vision detection methods. The method comprises the following steps: acquiring images of tomato plants, and establishing a tomato plant image data set in a natural scene; performing data enhancement on the tomato plant image data set, dividing a training set, a verification set and a test set, and labeling each tomato plant; the method comprises the following steps: generating an improved yolov8-seg model which comprises a Backbone network, a Neck network and a Head network; using the training set, the verification set and the test set to obtain a detection model with an optimal weight; and inputting a tomato plant image to be detected into the detection model, and segmenting a main branch, a side branch, a fruit branch, a mature fruit and an immature fruit from the tomato plant image to be detected by the detection model. According to the invention, each part of a tomato plant can be accurately segmented, and reliable technical support is provided for realizing integrated operation of intelligent tomato branch and leaf pruning and fruit picking.

Description

Technical Field

[0001] The invention belongs to the field of crop machine vision detection methods, and particularly relates to a tomato plant segmentation method based on an improved YOLOv8s-SEG model. Background Art

[0002] Tomatoes are widely cultivated worldwide and are a common fruit and vegetable in daily life. During the growing process, tomatoes require pruning and leaf removal, and harvesting after ripening. Excessive branches and leaves absorb moisture and block light from the fruits, hindering their growth. Timely pruning of tomato branches facilitates nutrient absorption and shortens the ripening period. However, harvesting and pruning tomatoes is a labor-intensive process. To reduce labor costs and alleviate the burden of manual harvesting, the use of smart devices can promote the sustainable development of tomato production.

[0003] In recent years, computer vision and deep learning technologies have made significant progress in the field of image recognition. The YOLO (You Only Look Once) algorithm, known for its real-time processing capabilities and single-stage detection process, can simultaneously predict the location and category of multiple objects directly from the entire image, leveraging global information to reduce false detections and improving small object detection through multi-scale prediction. Its end-to-end learning mechanism simplifies the training and inference processes. Combined with its continuously iteratively optimized model architecture and active open source community support, it has become a powerful tool for object detection that combines speed and accuracy. However, the traditional YOLO algorithm still has some issues with tomato plant recognition. Due to the close proximity of the fruit stem and lateral branches, and its poor robustness to environmental variations such as lighting and color, recognition accuracy is poor.

[0004] Therefore, it is necessary to provide a tomato plant segmentation method based on the improved yolov8-seg model to more accurately judge the maturity of side branches and fruits and reduce the occurrence of wrong cutting and picking. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects in the prior art and provide a tomato plant segmentation method based on an improved yolov8s-seg model.

[0006] The specific technical solutions adopted in the present invention are as follows:

[0007] The present invention provides a tomato plant segmentation method based on an improved yolov8s-seg model, which is as follows:

[0008] S1. Collect images of tomato plants and establish a tomato plant image dataset in natural scenes.

[0009] S2. Perform data augmentation on the tomato plant image dataset described in S1, divide it into a training set, a validation set, and a test set, and annotate each tomato plant image with the main branches, side branches, fruit branches, mature fruits, and immature fruits;

[0010] S3, generate an improved yolov8-seg model including Backbone network, Neck network and Head network;

[0011] S4. Inputting the training set described in S2 into the improved YOLOv8-SEG model described in S3 for multiple learning to train the model and determine the model parameters; during each training, the improved YOLOv8-SEG model identifies branches of different thicknesses, lengths, and shapes and fruits of different sizes and colors in the tomato plants from the input image, and uses the validation set described in S2 to adjust the hyperparameters of the model during training, and finally evaluates the generalization ability of the trained improved YOLOv8-SEG model through the test set described in S2, thereby obtaining a detection model with the optimal weight;

[0012] S5. Input the tomato plant image to be detected into the detection model with the optimal weight described in S4, and use the detection model to segment the main branches, side branches, fruit branches, mature fruits, and immature fruits from the tomato plant image to be detected.

[0013] Preferably, in S1, the image of the tomato plant includes fruits and branches, the fruits include mature fruits and immature fruits, and the branches include main branches, side branches and fruit branches.

[0014] Preferably, in S2, the K-fold cross-validation method is used to randomly divide the data set into 10 parts, and then selection is made from these 10 parts to construct the training set, validation set and test set for each training; each time a selection is made, 1 part is selected from the 10 parts as the test set, 1 part as the validation set, and the remaining 8 parts as the training set, that is, each time a selection is made, the data set is divided into the current training set, validation set and test set in a ratio of 7:2:1.

[0015] Preferably, in S2, the labelme tool is used to mark the main branches, side branches, fruit branches, mature fruits and immature fruits of each tomato plant, and finally a data set in json format is generated, which is then converted into a txt file format suitable for yolov8-seg training.

[0016] Preferably, in S3, the Backbone network includes a first convolutional layer, a second convolutional layer, a first c2f-PKI module, a third convolutional layer, a second c2f-PKI module, a fourth convolutional layer, a third c2f-PKI module, a fifth convolutional layer, a fourth c2f-PKI module and an SPPF module;

[0017] The PKINet module in the c2f-PKI module includes PKIMoudle and CAAMoudle. The PKIMoudle includes a 3×3 convolutional layer for capturing local information. This 3×3 convolutional layer is followed by a parallel set of depthwise separable convolutional layers with kernel sizes of 5, 7, 9, and 11, respectively, to capture multi-scale contextual information. Finally, 1×1 convolutions are used to fuse features of different receptive field sizes, capturing a wider range of contextual information without destroying local features.

[0018] The mathematical expression of this process is:

[0019]

[0020] Where, By k s ×k s Local features extracted by convolution, is the mth k (m) ×k (m) Context features extracted by depthwise separable convolution; n is the number of iterations, m is the index of depthwise separable convolution (corresponding to different kernel sizes of 5 / 7 / 9 / 11), N l-1 is the number of PKIMoudles;

[0021] Then, the local features and context features are fused through a 1×1 convolution to obtain

[0022]

[0023] Where, represents the output features;

[0024] At the same time, the contextual anchor attention CAAMoudle uses an average pooling operation and then performs a 1×1 convolution operation to obtain local area features. The mathematical expression of this process is:

[0025]

[0026] Where, P avg represents the average pooling operation, For the input of CAAMoudle, The output of the average pooling operation and the 1×1 convolution operation;

[0027] Using two depthwise separable convolutions, it can be expressed as:

[0028]

[0029] Among them, k b =11+2×l, It is the output after two depth-wise separable convolutions;

[0030] Subsequently, an attention weight is generated through CAAMoudle operation, and its mathematical expression is:

[0031]

[0032] Among them, Sigmoid is the activation function, A l-1,n It is the output after a 1×1 convolution and Sigmoid activation function;

[0033] Output features generated by PKIMoudle and CAAMoudle to generate attention weights Combining these two, we get The specific expression is:

[0034] Among them, the symbol ⊙ represents element-by-element multiplication, and the symbol ⊕ represents element-by-element addition. It is the enhanced feature;

[0035] After enhancement, the features Perform a 1×1 convolution operation, that is Get the output of PKINet.

[0036] Preferably, in S3, the Neck network includes a first upsampling Dysample layer, a first Concat fusion layer, a first C2f module, a second upsampling Dysample layer, a second Concat fusion layer, a second C2f module, a third Concat fusion layer, a third C2f module, a sixth convolutional layer, a fourth Concat fusion layer, a fourth C2f module, a seventh convolutional layer, a fifth Concat fusion layer, a fifth C2f module, an eighth convolutional layer, a sixth Concat fusion layer and a sixth C2f module;

[0037] Given an upsample s, and a feature map x of size C×H×W, the number of input and output channels are C and 2s respectively 2 The linear layer generates a size of 2s 2 ×H×W offset The expression is as follows:

[0038]

[0039] Where W is the weight matrix with size C×2s 2 , represents the mapping from input channel to output channel; b is the bias top size of 2s 2 , offset is the output feature map, size is 2s2 ×H×W; then use pixel shuffling PixelShuffl to set the offset The size is converted to 2s×sH×sW to achieve efficient spatial resolution improvement; finally, the offset and the original sampling grid Add up to get the sampling set The expression is as follows:

[0040]

[0041] However, in the above method, due to the upsampling of local s 2 The moving ranges of the sample positions may overlap significantly, which will affect the prediction near the boundary, and this error will propagate from stage to stage, so the offset Multiplying by 0.25 (just meeting the theoretical critical condition between overlap and non-overlap) can effectively alleviate the adverse effects of overlap on predictions near the boundary, and finally obtain the formula:

[0042]

[0043] Since the offset of the static factor is not flexible enough, we further generate a point-by-point dynamic range factor by linearly projecting the input feature x. Using the sigmoid function and a static factor of 0.5, we get the formula:

[0044]

[0045] The calculated value is centered at 0.25 and ranges from 0 to 0.5, the same range as the static factor.

[0046] Preferably, in S3, the Head network includes a first GT module, a first decoupling head detection layer, a second decoupling head detection layer, a third decoupling head detection layer and a fourth decoupling head detection layer;

[0047] The first GT module will fuse the three feature maps of different sizes output by the neck through the feature extractor conv_GN; the feature extractor uses N consecutive shared convolutional layers, group normalization and activation functions to calculate task interaction features:

[0048]

[0049] Among them, X fpn ∈R H×W×C represents FPN, H, W and C represent height, width and number of channels respectively; conv k , gn and δ refer to the kth shared convolutional layer, GroupNorm and SiLU functions respectively; represents the input eigenvalue, represents the output eigenvalue;

[0050] The calculated task interaction features are input into the two TaskDe modules to achieve the alignment of classification and localization, so that the two tasks can fully perceive each other's status. To solve the feature conflict problem caused by the difference in the prediction targets of the two tasks, specific features are calculated for each classification and localization task respectively:

[0051]

[0052] Among them, fc1 and fc2 represent two fully connected layers, σ is the sigmoid function, δ is the RELU function, and x inter Yes obtained by applying average pooling; yes The connection characteristics of for The kth processing result in; w represents x inter After two fully connected layers and the output of the sigmoid function and RELU function, Indicates that for 1 to N pairs One-to-one correspondence results of processing;

[0053] From each Predict classification or positioning results:

[0054] Z task =conv2(δ(gn(conv1(X task ))))

[0055] Among them, X task yes The connection characteristics, Z task Represents X task The output result after a series of feature processing, conv2 and conv1 both represent convolution operations;

[0056] The classification feature Z obtained above task With conv2(δ(conv1(X inter )))The classification features are obtained by element-by-element multiplication. For positioning the shape, Spatial Offset is added to generate an offset for each pixel in the feature map to adjust the position and shape of the target.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The YOLOv8-seg model suffers from large network size, low segmentation accuracy, and the risk of mis-cutting and mis-picking. This paper proposes an improved YOLOv8-seg model network that reduces the number of parameters and improves model detection speed and accuracy. The optimized structure can be applied to tomato plantations.

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart in an embodiment of the present invention;

[0061] Figure 2 It is a diagram of the existing yolov8s-seg model architecture in an embodiment of the present invention;

[0062] Figure 3 This is a diagram showing the improved yolov8s-seg model architecture in an embodiment of the present invention;

[0063] Figure 4 This is a diagram of the c2f-PKI model architecture in an embodiment of the present invention;

[0064] Figure 5 This is a diagram of the PKINet model architecture in an embodiment of the present invention;

[0065] Figure 6 This is a diagram of the Dysample model architecture in an embodiment of the present invention;

[0066] Figure 7 This is a diagram of the GT model architecture in an embodiment of the present invention;

[0067] Figure 8 This is a diagram of the Conv_GN model architecture in an embodiment of the present invention;

[0068] Figure 9 It is a diagram of the TaskDe model architecture in an embodiment of the present invention;

[0069] Figure 10 is the segmentation result predicted by the yolov8-seg model in an embodiment of the present invention;

[0070] Figure 11 It is the segmentation result predicted by the improved yolov8-seg model in the embodiment of the present invention. DETAILED DESCRIPTION

[0071] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments. The technical features of each embodiment of the present invention may be combined accordingly, provided that there is no conflict between them.

[0072] like Figure 1 As shown in the figure, a tomato plant segmentation method based on the improved yolov8s-seg model provided by the present invention is firstly collected in different growth stages, growing under various environmental conditions. Then, the collected data set is enhanced and each tomato plant is segmented and labeled, and the tomato plant is divided into main stem, side branch, fruit branch, mature fruit and immature fruit. The yolov8-seg segmentation model is optimized from the aspects of backbone network, middle layer and head structure. In the backbone network (Backbone), the c2f-PKI module is used to replace the original c2f module, which enhances the model's ability to extract feature information of targets of different scales. In the middle layer (Neck), Dysample is used to replace the traditional upsampling method, so that the model can improve the image resolution in a more concise and efficient way, and enhance the accuracy of the model for segmentation edge prediction. In the head (Head) structure, the independently developed GT module is introduced. This design not only successfully reduces the overall parameter amount of the model, but also realizes the efficient learning and deep fusion of task features between different convolutional layers, greatly improving the model recognition accuracy. This tomato plant segmentation technology can accurately segment various parts of a tomato plant, providing reliable technical support for the intelligent integrated operation of tomato branch and leaf pruning and fruit picking.

[0073] The present invention specifically comprises the following steps:

[0074] S1. Collect images of tomato plants and establish a tomato plant image dataset in natural scenes.

[0075] As a preferred embodiment of the present invention, the steps are as follows:

[0076] First, determine the size range of the tomato plant image. The image of the entire tomato plant should include the tomato fruits, including mature and immature fruits. The tomato branches need to include the main branch, side branches, and fruit branches. Ensure that the required data set is complete for subsequent model training.

[0077] At the same time, in order to improve the generalization ability of the model, the photos were taken in a variety of complex environments including daytime, nighttime, occlusion by branches and leaves, backlighting, and night flash photography.

[0078] S2. Perform data augmentation on the tomato plant image dataset obtained in S1, divide it into a training set, a validation set, and a test set, and label each tomato plant with the main branches, side branches, fruit branches, mature fruits, and immature fruits.

[0079] As a preferred embodiment of the present invention, in this step, data enhancement is mainly to improve the quality of the image and reduce the noise in the image. Common methods include histogram equalization and Gaussian blur. For example, the image can be enhanced by vertical mirroring, horizontal mirroring, random rotation, brightness adjustment, etc. In addition, this example uses the existing YOLOv8 model's own built-in online data enhancement Mosaic and HSVaugment two main enhancement methods for data enhancement.

[0080] In this embodiment, to fully utilize the data, the K-fold cross-validation method is used to randomly divide the data set into 10 parts. Then, selections are made from these 10 parts to construct the training set, validation set, and test set for each training session. Each time a selection is made, 1 part is selected from the 10 parts as the test set, 1 part as the validation set, and the remaining 8 parts as the training set. That is, each time a selection is made, the data set is divided into the current training set, validation set, and test set in a ratio of 7:2:1. The validation set and test set selected multiple times are not repeated, and the training set, validation set, and test set obtained from multiple training sessions are all non-repeated, thereby achieving full utilization of the data set.

[0081] Next, we need to label the preprocessed tomato plants. Using the labelme tool, we annotate each tomato plant with the main branches, side branches, fruiting branches, mature fruit, and immature fruit. This generates a dataset in JSON format, which we then convert to a txt file suitable for yolov8-seg training.

[0082] S3. Generate an improved yolov8-seg model. The improved yolov8-seg model is an improvement based on the existing yolov8s-seg model, which mainly includes Backbone network, Neck network and Head network.

[0083] like Figure 2 As shown in the figure, it is the existing yolov8s-seg model, which mainly includes Backbone network, Neck network and Head network. Figure 3 As shown, it is the improved yolov8-seg model of the present invention.

[0084] As a preferred embodiment of the present invention, in this step, the Backbone network mainly includes the first convolution layer, the second convolution layer, the first c2f-PKI module, the third convolution layer, the second c2f-PKI module, the fourth convolution layer, the third c2f-PKI module, the fifth convolution layer, the fourth c2f-PKI module and the SPPF module. The changes in this example are to change the original c2f to Figure 4 c2f-PKI shown.

[0085] Among them, the PKINet module in the c2f-PKI module is as follows Figure 5 As shown in Figure 1, the PKINet module mainly includes PKIMoudle and CAAMoudle. The structure of PKIMoudle is as follows: Figure 5 As shown in the figure, it contains a 3×3 convolutional layer, which is used to capture local information. The 3×3 convolutional layer is followed by a set of parallel depthwise separable convolutional layers with kernel sizes of 5, 7, 9, and 11, respectively. This setup captures multi-scale contextual information. Finally, a 1×1 convolution is used to fuse features of different receptive field sizes, capturing a wider range of contextual information without destroying local features.

[0086] The mathematical expression of this process is:

[0087]

[0088]

[0089] Where, By k s ×k s Local features extracted by convolution, is the mth k (m) ×k (m) Context features extracted by depthwise separable convolution (DWConv); n is the iteration round, m is the index of depthwise separable convolution (corresponding to different kernel sizes of 5 / 7 / 9 / 11), N l-1 is the number of PKIMoudles;

[0090] Then, the local features and context features are fused through a 1×1 convolution to obtain

[0091]

[0092] Where, Represents the output feature. In the process of fusing features of different receptive field sizes, the 1×1 convolution can capture a wider range of contextual feature information without destroying local features.

[0093] At the same time, in order to effectively capture longer-range context feature information, the PKINet module introduces contextual anchor attention CAAMoudle. Figure 5 The structure of CAAMoudle is shown in Figure 1. It uses average pooling and then performs 1×1 convolution to obtain local area features. The mathematical expression of this process is:

[0094]

[0095] Where, P avg represents the average pooling operation, For the input of CAAMoudle, The output of the average pooling operation and the 1×1 convolution operation;

[0096] Next, two depth-wise separable convolutions are used, which can be expressed as:

[0097]

[0098] Among them, k b =11+2×l, It is the output after two depth-wise separable convolutions;

[0099] Subsequently, an attention weight is generated through CAAMoudle operation, and its mathematical expression is:

[0100]

[0101] Among them, Sigmoid is the activation function, A l-1,n It is the output after a 1×1 convolution and Sigmoid activation function.

[0102] Output features generated by PKIMoudle and CAAMoudle to generate attention weights Combining these two, we get The specific expression is:

[0103] The Sigmoid function here ensures that the attention map A l-1,n The value of is in the range of (0,1), where the symbol ⊙ represents element-by-element multiplication and the symbol ⊕ represents element-by-element addition. It is the enhanced feature;

[0104] After enhancement, the features Perform a 1×1 convolution operation, that is Obtain the output of PKINet. Because tomato branches and fruits are easily obscured by leaves in natural environments, this improvement allows for more accurate and complete identification of tomato branches and main stems, effectively avoiding the problem of incorrect segmentation of the same branch.

[0105] As a preferred embodiment of the present invention, in this step, the Neck network includes a first upsampling Dysample layer, a first Concat fusion layer, a first C2f module, a second upsampling Dysample layer, a second Concat fusion layer, a second C2f module, a third Concat fusion layer, a third C2f module, a sixth convolutional layer, a fourth Concat fusion layer, a fourth C2f module, a seventh convolutional layer, a fifth Concat fusion layer, a fifth C2f module, an eighth convolutional layer, a sixth Concat fusion layer and a sixth C2f module.

[0106] Among them, the upsample in the existing model is replaced by Dysample.

[0107] DySample is a module with excellent performance. It can dynamically adjust the upsampling strategy according to the feature information of the input image, thereby improving the accuracy of the input content. This module not only has the advantages of being ultra-lightweight, with a small number of parameters and low computational complexity, but also performs well in terms of GPU memory usage and latency. In image segmentation tasks, DySample can more accurately predict the content inside the image, especially at the junction between different categories, and can clearly predict the boundaries. There are 5 segmentation categories in the tomato plant segmentation task, and the tomato fruit is closely connected to the fruit branches, the main branches and side branches, and the fruit branches. In this case, DySample can significantly improve the accuracy of the prediction of the boundary edges. The structure of the DySample module is as follows: Figure 6 , given an upsample s, and a feature map "x" of size "C×H×W", the number of input and output channels are C and 2s respectively 2 The linear layer can generate a size of 2s 2 ×H×W offset The expression is as follows:

[0108]

[0109] Where W is the weight matrix with size C×2s 2 , represents the mapping from input channel to output channel; b is the bias top size of 2s 2 , offset is the output feature map, size is 2s 2 ×H×W; then use pixel shuffling PixelShuffl to set the offset The size is converted to 2s×sH×sW to achieve efficient spatial resolution improvement; finally, the offset and the original sampling grid Add up to get the sampling set The expression is as follows:

[0110]

[0111] However, in the above method, due to the upsampling of local s 2 The moving ranges of the sample positions may overlap significantly, which will affect the prediction near the boundary, and this error will propagate from stage to stage, so the offset Multiplying by 0.25 (which just meets the theoretical critical condition between overlap and non-overlap) can effectively alleviate the adverse effects of overlap on predictions near the boundary, and finally obtain the formula:

[0112]

[0113] Since the offset of the static factor is not flexible enough, we can further generate a point-by-point "dynamic range factor" by linearly projecting the input feature x. Using the sigmoid function and a static factor of 0.5, we get the formula:

[0114]

[0115] The calculated value is centered at 0.25 and ranges from 0 to 0.5, the same range as the static factor. This approach enhances the model's flexibility in handling complex boundaries and improves prediction accuracy.

[0116] As a preferred embodiment of the present invention, in this step, the head network includes a first GT module, a first decoupling head detection layer, a second decoupling head detection layer, a third decoupling head detection layer, and a fourth decoupling head detection layer. The GT module is an additional module on the original basis.

[0117] In the tomato plant segmentation task, since some tomato fruit branches and tomato side branches are very similar in shape, it is very easy to make mistakes during recognition. When faced with this situation, it is not enough to judge based on shape alone. It is also necessary to further determine whether there are fruits near the two branches, or to combine other surrounding environmental factors for comprehensive judgment. The GT module, with its unique performance, can not only achieve more diverse feature interactions, but also use positioning features to further improve the recognition accuracy. Figure 7 As shown in the figure, the structure of the first GT module is shown. The first GT module will fuse the three feature maps with different scales output by the neck through the feature extractor conv_GN; Figure 8 As shown, the feature extractor uses N consecutive shared convolutional layers, group normalization and activation functions to calculate task interaction features:

[0118]

[0119] Among them, X fpn ∈RH×W×C represents FPN, H, W and C represent height, width and number of channels respectively; conv k , gn and δ refer to the kth convolutional layer, GroupNorm and SiLU functions respectively; represents the input eigenvalue, Represents the output eigenvalue.

[0120] Therefore, conv_GN can efficiently extract rich and diverse multi-scale features from FPN features. Subsequently, the calculated task interaction features will be input into the two TaskDe modules to achieve the alignment operation of classification and positioning. Based on the calculated task interaction features, the two tasks of target classification and positioning are carried out simultaneously, so that the two tasks can fully and well perceive each other's status. In order to solve the feature conflict problem caused by the difference in the prediction targets of the two tasks, for each classification and positioning task, their respective specific features are calculated separately. The specific structure is as follows: Figure 9 The TaskDe module shown in:

[0121] w=σ(fc2(δ(fc1(x inter ))))

[0122]

[0123] Among them, fc1 and fc2 represent two fully connected layers, σ is the sigmoid function, δ is the RELU function, and x inter Yes Applying average pooling, yes ( The connection feature of the kth processing result in x inter After two fully connected layers and the output of the sigmoid function and RELU function, Indicates that for 1 to N pairs The one-to-one correspondence results of the processing.

[0124] From each Predict classification or positioning results:

[0125] Z task =conv2(δ(gn(conv1(X task ))))

[0126] where X task yes The connection characteristics, Z task Represents X task The output result after a series of feature processing, conv2 represents the convolution operation, and conv1 represents the convolution operation;

[0127] The classification feature Z obtained above task With conv2(δ(conv1(X inter )))The classification features are obtained by element-by-element multiplication. For positioning the shape, Spatial Offset is added to generate an offset for each pixel in the feature map to adjust the position and shape of the target.

[0128] S4. Input the training set obtained in S2 into the improved yolov8-seg model obtained in S3 for multiple learning to train the model and determine the model parameters; during each training, the improved yolov8-seg model identifies branches of different thicknesses, lengths, and shapes and fruits of different sizes and colors in tomato plants from the input images, and uses the validation set obtained in S2 to adjust the model's hyperparameters during training. Finally, the generalization ability of the trained improved yolov8-seg model is evaluated through the test set obtained in S2, thereby obtaining a detection model with the optimal weight.

[0129] S5. Input the tomato plant image to be detected into the detection model with the optimal weight obtained in S4, and use the detection model to segment the main branches, side branches, fruit branches, mature fruits, and immature fruits from the tomato image to be detected.

[0130] like Figure 10 is the segmentation result predicted by the yolov8-seg model in the embodiment of the present invention, Figure 11 It is the predicted segmentation result of the improved yolov8-seg model in the embodiment of the present invention. It can be clearly seen by comparison that the improved model has been significantly improved in performance. Not only has the recognition accuracy been greatly improved, but the complete recognition of the tomato side branches and main branches can be achieved more outstandingly. Specifically, in the original model, the fruit branch in the upper left corner could not be identified due to being blocked by branches and leaves, but the improved model successfully identified the fruit branch with its better performance. In addition, the original model also failed to identify the upper half of the main branch, and when only one main branch in the figure was blocked by the fruit, the original model mistakenly identified the main trunk of the upper and lower parts of the fruit into two sections, while the improved model can accurately identify the complete form of the main branch. In terms of the recognition of the fruit in the lower left corner, the original model had the error of identifying the same fruit as two results, while the improved model solved this problem well, and the recognition result was more accurate and reliable.

[0131] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A tomato plant segmentation method based on an improved yolov8s-seg model, characterized in that: The details are as follows: S1. Collect images of tomato plants and establish a tomato plant image dataset in natural scenes. S2. Perform data augmentation on the tomato plant image dataset described in S1, divide it into a training set, a validation set, and a test set, and annotate each tomato plant image with the main branches, side branches, fruit branches, mature fruits, and immature fruits; S3, generate an improved yolov8-seg model including Backbone network, Neck network and Head network; S4. Inputting the training set described in S2 into the improved YOLOv8-SEG model described in S3 for multiple learning to train the model and determine the model parameters; during each training, the improved YOLOv8-SEG model identifies branches of different thicknesses, lengths, and shapes and fruits of different sizes and colors in the tomato plants from the input image, and uses the validation set described in S2 to adjust the hyperparameters of the model during training, and finally evaluates the generalization ability of the trained improved YOLOv8-SEG model through the test set described in S2, thereby obtaining a detection model with the optimal weight; S5. Input the tomato plant image to be detected into the detection model with the optimal weight described in S4, and use the detection model to segment the main branches, side branches, fruit branches, mature fruits, and immature fruits from the tomato plant image to be detected.

2. A tomato plant segmentation method based on the improved yolov8s-seg model according to claim 1, characterized in that: In S1, the image of the tomato plant includes fruits and branches, the fruits include mature fruits and immature fruits, and the branches include main branches, side branches and fruit branches.

3. The tomato plant segmentation method based on the improved yolov8s-seg model according to claim 1, characterized in that: In S2, the K-fold cross-validation method is used to randomly divide the data set into 10 parts, and then selection is made from these 10 parts to construct the training set, validation set and test set for each training; each time a selection is made, 1 part is selected from the 10 parts as the test set, 1 part as the validation set, and the remaining 8 parts as the training set, that is, each time a selection is made, the data set is divided into the current training set, validation set and test set in a ratio of 7:2:

1.

4. The tomato plant segmentation method based on the improved yolov8s-seg model according to claim 1, characterized in that: In S2, the labelme tool is used to mark the main branches, side branches, fruit branches, mature fruits and immature fruits of each tomato plant, and finally a data set in json format is generated, which is then converted into a txt file format suitable for yolov8-seg training.

5. The tomato plant segmentation method based on the improved yolov8s-seg model according to claim 1, characterized in that: In S3, the Backbone network includes a first convolutional layer, a second convolutional layer, a first c2f-PKI module, a third convolutional layer, a second c2f-PKI module, a fourth convolutional layer, a third c2f-PKI module, a fifth convolutional layer, a fourth c2f-PKI module and an SPPF module; The PKINet module in the c2f-PKI module includes PKIMoudle and CAAMoudle. The PKIMoudle includes a 3×3 convolutional layer for capturing local information. This 3×3 convolutional layer is followed by a parallel set of depthwise separable convolutional layers with kernel sizes of 5, 7, 9, and 11, respectively, to capture multi-scale contextual information. Finally, 1×1 convolutions are used to fuse features of different receptive field sizes, capturing a wider range of contextual information without destroying local features. The mathematical expression of this process is: Where, By k s ×k s Local features extracted by convolution, is the mth k (m) ×k (m) Context features extracted by depthwise separable convolution; n is the iteration round, m is the index of depthwise separable convolution, N l-1 is the number of PKIMoudles; Then, the local features and context features are fused through a 1×1 convolution to obtain Where, represents the output features; At the same time, the contextual anchor attention CAAMoudle uses an average pooling operation and then performs a 1×1 convolution operation to obtain local area features. The mathematical expression of this process is: Where, P avg represents the average pooling operation, For the input of CAAMoudle, The output of the average pooling operation and the 1×1 convolution operation; Using two depthwise separable convolutions, it can be expressed as: Among them, k b =11+2×l, It is the output after two depth-wise separable convolutions; Subsequently, an attention weight is generated through CAAMoudle operation, and its mathematical expression is: Among them, Sigmoid is the activation function, A l-1,n It is the output after a 1×1 convolution and Sigmoid activation function; Output features generated by PKIMoudle and CAAMoudle to generate attention weights Combining these two, we get The specific expression is: Among them, the symbol ⊙ represents element-by-element multiplication, and the symbol represents element-by-element addition, It is the enhanced feature; After enhancement, the features Perform a 1×1 convolution operation, that is Get the output of PKINet.

6. The tomato plant segmentation method based on the improved yolov8s-seg model according to claim 1, characterized in that: In S3, the Neck network includes a first upsampling Dysample layer, a first Concat fusion layer, a first C2f module, a second upsampling Dysample layer, a second Concat fusion layer, a second C2f module, a third Concat fusion layer, a third C2f module, a sixth convolutional layer, a fourth Concat fusion layer, a fourth C2f module, a seventh convolutional layer, a fifth Concat fusion layer, a fifth C2f module, an eighth convolutional layer, a sixth Concat fusion layer and a sixth C2f module; Given an upsample s, and a feature map x of size C×H×W, the number of input and output channels are C and 2s respectively 2 The linear layer generates a size of 2s 2 ×H×W offset The expression is as follows: Where W is the weight matrix with size C×2s 2 , represents the mapping from input channels to output channels; b is the offset top size of 2s 2 , offset is the output feature map, size is 2s 2 ×H×W; then use pixel shuffling PixelShuffl to set the offset The size is converted to 2s×sH×sW to achieve efficient spatial resolution improvement; finally, the offset and the original sampling grid Add up to get the sampling set The expression is as follows: However, in the above method, due to the upsampling of local s 2 The moving ranges of the sample positions may overlap significantly, which will affect the prediction near the boundary, and this error will propagate from stage to stage, so the offset Multiply by 0.25 to effectively alleviate the adverse effects of overlapping problems on predictions near the boundary, and finally get the formula: Since the offset of the static factor is not flexible enough, we further generate a point-by-point dynamic range factor by linearly projecting the input feature x. Using the sigmoid function and a static factor of 0.5, we get the formula: The calculated value is centered at 0.25 and ranges from 0 to 0.5, the same range as the static factor.

7. The tomato plant segmentation method based on the improved yolov8s-seg model according to claim 1, characterized in that: In S3, the Head network includes a first GT module, a first decoupling head detection layer, a second decoupling head detection layer, a third decoupling head detection layer, and a fourth decoupling head detection layer; The first GT module will fuse the three feature maps of different sizes output by the neck through the feature extractor conv_GN; the feature extractor uses N consecutive shared convolutional layers, group normalization and activation functions to calculate task interaction features: Among them, X fpn ∈R H×W×C represents FPN, H, W and C represent height, width and number of channels respectively; conv k , gn and δ refer to the kth shared convolutional layer, GroupNorm and SiLU functions respectively; represents the input eigenvalue, represents the output eigenvalue; The calculated task interaction features are input into the two TaskDe modules to achieve the alignment of classification and localization, so that the two tasks can fully perceive each other's status. To solve the feature conflict problem caused by the difference in the prediction targets of the two tasks, specific features are calculated for each classification and localization task respectively: w=σ(fc2(δ(fc1(x inter )))) Among them, fc1 and fc2 represent two fully connected layers, σ is the sigmoid function, δ is the RELU function, and x inter Yes obtained by applying average pooling; yes The connection characteristics of for The kth processing result in; w represents x inter After two fully connected layers and the output of the sigmoid function and RELU function, Indicates that for 1 to N pairs One-to-one correspondence results of processing; From each Predict classification or positioning results: Z task =conv2(δ(gn(conv1(X task )))) Among them, X task yes The connection characteristics, Z task Represents X task The output result after a series of feature processing, conv2 and conv1 both represent convolution operations; The classification feature Z obtained above task With conv2(δ(conv1(X inter )))The classification features are obtained by element-by-element multiplication. For positioning the shape, Spatial Offset is added to generate an offset for each pixel in the feature map to adjust the position and shape of the target.

Images