Lemon fruit cyst counting method and system based on deep learning

Abstract

The invention is suitable for the technical field of computer vision and deep learning, and provides a lemon fruit cyst counting method and system based on deep learning, and the method comprises the following steps: 1, obtaining a cross section image of a lemon fruit, and carrying out the preprocessing of the image; step 2, manually annotating the preprocessed image by using an annotation tool, generating an annotation file, and constructing a training data set; 3, performing data enhancement processing on the training data set; 4, an instance segmentation model is selected, ResNet50 or ResNet101 is taken as a main network, and model training is carried out on the constructed training data set; 5, segmenting the cyst in the cross section image of the lemon fruit by using the trained model; 6, counting the lemon fruit cysts according to the segmentation result; according to the method, the accuracy and efficiency of lemon fruit cyst segment segmentation and counting are improved, and powerful technical support is provided for lemon quality evaluation and variety characteristic research.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, in particular to a lemon fruit segment counting method and system based on deep learning. BACKGROUND

[0002] The number of lemon fruit segments is one of the important indicators for evaluating the quality of lemons. Traditional segment counting methods mainly rely on manual visual inspection, which is not only inefficient but also easily affected by subjective factors, resulting in inconsistent counting results. With the development of computer vision technology, some image processing-based methods for analyzing the internal structure of fruits have been proposed, but these methods often have unsatisfactory segmentation and counting results when faced with complex situations such as tightly packed and clustered lemon segments and unclear boundaries.

[0003] Existing instance segmentation networks such as Mask R-CNN and YOLACT perform well in general object segmentation tasks, but when directly applied to lemon fruit segment segmentation, the following problems exist: (1) the fuzzy septum between segments leads to inaccurate segmentation boundaries; (2) the internal reflective region of the fruit and the seed can easily cause misidentification; (3) the morphological differences of lemon segments of different varieties and maturity are large, and the model generalization ability is insufficient.

[0004] Therefore, in view of the above status, it is urgent to provide a lemon fruit segment counting method and system based on deep learning to overcome the shortcomings in current practical applications. SUMMARY

[0005] The present application aims to provide a lemon fruit segment counting method and system based on deep learning, effectively solving the problems in the background art.

[0006] The present application is implemented as follows: a lemon fruit segment counting method based on deep learning, the method comprising the following steps: Step 1: Obtain a lemon fruit cross-sectional image and pre-process the image; Step 2: Use a labeling tool to manually label the pre-processed image to generate a label file and build a training data set; Step 3: Perform data augmentation processing on the training data set; Step 4: Select an instance segmentation model with ResNet50 or ResNet101 as the backbone network, and train the model on the constructed training data set; Step 5: Use the trained model to segment the segments in the lemon fruit cross-sectional image; Step 6: Count the lemon fruit segments based on the segmentation results.

[0007] As a further scheme of the present application: the preprocessing in step 1 comprises image cropping, scaling and normalization processing.

[0008] As a further scheme of the present application: in step 2, the artificial labeling is performed using the Labelme labeling tool, the fruit locule contour is enclosed with an irregular polygon, the labeling box is uniformly represented as “fruitlocule”, a json format labeling file is generated, and the labeling file contains the picture name, storage path, labeling box category name and position information.

[0009] As a further scheme of the present application: in step 3, the data augmentation processing adopts online augmentation technology, including geometric transformation, brightness and contrast adjustment and noise addition. The geometric transformation includes one or more combinations of random rotation, translation and flipping. The noise addition adopts the methods of Gaussian filtering and salt and pepper noise.

[0010] As a further scheme of the present application: in step 4, the instance segmentation model is a Mask R-CNN model or a YOLACT model.

[0011] As a further scheme of the present application: when the instance segmentation model is a Mask R-CNN model, the instance segmentation model is composed of a backbone network, a region candidate network, a region of interest alignment and an output network, and the backbone network is composed of a ResNet and a feature pyramid network.

[0012] As a further scheme of the present application: when the instance segmentation model is a YOLACT model, the instance segmentation model is composed of a feature extraction structure, a prediction module, a mask template generation branch, an aggregation branch and a cropping and threshold segmentation module, and the feature extraction structure is composed of a backbone network and a feature pyramid network.

[0013] As a further scheme of the present application: in step 4, the hyperparameter settings for model training are as follows: IMAGE_SIZE is 5496x3672, LEARNING_RATE is 0.0025, WEIGHT_DECAY is 0.0001, MOMENTUM is 0.9, BATCH_SIZE is 2, EPOCH is 60, and the optimizer is SGD.

[0014] As a further scheme of the present application: in step 5, the evaluation of the model segmentation effect adopts the mean average precision (mAP) and the average detection time consumption, and the calculation of the mean average precision is based on the intersection over union (IoU), the confusion matrix, the precision, the recall and the average precision (AP).

[0015] The lemon fruit locule counting system based on deep learning runs the method as described above, and the system comprises: an image acquisition module, configured to acquire a cross-section image of a lemon fruit; a data preprocessing module, configured to preprocess the acquired image; a data set construction module, configured to manually label the preprocessed image using a labeling tool, generate a labeling file, and construct a training data set; a data enhancement module, configured to perform data enhancement processing on the training data set; a model training module, configured to select an instance segmentation model, take ResNet50 or ResNet101 as a backbone network, and perform model training on the constructed training data set; a locule segmentation module, configured to segment locules in the cross-section image of the lemon fruit by using the trained model; a locule counting module, configured to count the locules of the lemon fruit according to the segmentation result.

[0016] Compared with the prior art, the present application has the following advantages: The two-stage instance segmentation model Mask R-CNN is combined with the ResNet50 backbone network to optimize the calculation efficiency while maintaining high segmentation accuracy; A data enhancement strategy specifically designed for the characteristics of lemon locules is designed, including random rotation, brightness contrast adjustment and noise addition, which improves the generalization ability of the model for different varieties and maturity lemons; A lemon fruit locule special data set containing 24 varieties and 454 high-quality labeled images is constructed; A locule automatic counting algorithm based on the segmentation result is proposed, which achieves a counting accuracy of 91.1%. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0018] Figure 1 Mask R-CNN network framework diagram; Figure 2 Yolact network framework diagram; Figure 3 Residual structure diagram; Figure 4 Part of the original image of lemon fruit collection; Figure 5 Labelme operation interface and labeling process schematic diagram; Figure 6 for annotation file schematic diagram; Figure 7 for data enhancement example; Figure 8 for mAP solving flowchart; Figure 9 for IoU principle schematic diagram; Figure 10 for loss curve schematic diagram; Figure 11 for Mask R-CNN and YOLACT segmentation result visualization schematic diagram. DETAILED DESCRIPTION

[0019] The technical solutions of the present application will be described clearly and completely below in conjunction with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0020] The present application utilizes an instance segmentation network in deep learning to classify target pixels through a deep learning network, thereby achieving segmentation effect. The instance segmentation network can not only identify and classify lemon locules, but also frame and segment each lemon fruit locule individual. It can effectively obtain individual information under complex conditions such as tight aggregation of locules and unclear boundary lines, and is suitable for precise quantitative determination of lemon locules. Therefore, in combination with research targets, network segmentation accuracy, segmentation rate, and network environment configuration factors, the two-stage instance segmentation model Mask R-CNN with high segmentation accuracy and strong universality and the single-stage instance segmentation model YOLACT with simple structure and fast speed are finally selected as the technical method for lemon fruit locule segmentation task.

[0021] The present application will be further explained and described below in conjunction with specific implementation manners.

[0022] Please refer to Figures 1-11 The lemon fruit locule counting method based on deep learning provided by the embodiments of the present application comprises the following steps: Step 1: Obtain a lemon fruit cross-sectional image and pre-process the image; the pre-processing includes cropping, scaling and normalizing the image; Step 2: manually label the pre-processed images using a labeling tool to generate a label file and build a training data set; specifically, manually label using the Labelme labeling tool, enclose the fruit locule outline with an irregular polygon, label the frame as "fruitlocule", generate a json format label file, and the label file contains the picture name, storage path, label frame class name and position information; Step 3: data augmentation processing is performed on the training data set; the data augmentation processing adopts online enhancement technology, including geometric transformation, brightness and contrast adjustment, and adding noise; The geometric transformation includes one or more combinations of random rotation, translation and flipping; The method of adding noise adopts Gaussian filtering and salt and pepper noise; Step 4: select an instance segmentation model, take ResNet50 or ResNet101 as the backbone network, and train the model on the built training data set; the instance segmentation model is a Mask R-CNN model or a YOLACT model; When the instance segmentation model is a Mask R-CNN model, the instance segmentation model is composed of a backbone network, a region candidate network, a region of interest alignment and an output network, and the backbone network is composed of a ResNet and a feature pyramid network; when the instance segmentation model is a YOLACT model, the instance segmentation model is composed of a feature extraction structure, a prediction module, a mask template generation branch, an aggregation branch, and a cropping and threshold segmentation module, and the feature extraction structure is composed of a backbone network and a feature pyramid network; Step 5: use the trained model to segment the locule in the lemon fruit transverse section image; the evaluation of the model segmentation effect adopts the mean average precision (mAP) and the average detection time consumption, and the calculation of the mean average precision is based on the intersection over union (IoU), the confusion matrix, the precision, the recall and the average precision (AP); Step 6: count the lemon fruit locule according to the segmentation result.

[0023] Please refer to Figures 1-11 The lemon fruit locule counting system based on deep learning provided by the embodiment of the application runs the method as described above, and the system comprises: An image acquisition module for acquiring lemon fruit transverse section images; A data preprocessing module for preprocessing the acquired images; A data set construction module for manually labeling the pre-processed images using a labeling tool to generate a label file and build a training data set; A data enhancement module for performing data augmentation processing on the training data set; The model training module is configured to select an instance segmentation model, take ResNet50 or ResNet101 as a backbone network, and perform model training on the constructed training data set. The capsule segmentation module is configured to segment the capsules in the lemon fruit transverse section image by using the trained model. The capsule counting module is configured to count the capsules of the lemon fruit according to the segmentation result.

[0024] The following further describes the above steps: The Mask R-CNN structure is shown in Figure 1 The principle can be divided into two parts, which are an improved Faster R-CNN target detection structure and a full convolutional neural network structure for segmenting a mask. The network is composed of a backbone network, a region proposal network (RPN), a region of interest alignment (Ro I Align), and an output network (Head). The backbone network is composed of a deep residual network (Resnet) and a feature pyramid network (FPN).

[0025] The steps of implementing the Mask R-CNN algorithm are as follows: (1) First, input the image to be processed, perform the corresponding preprocessing operation, and then input the preprocessed image into the ResNet network to obtain the corresponding feature map (feature map). By setting a predetermined ROI for each point of the feature map, multiple candidate ROIs are obtained; (2) The obtained multiple candidate ROIs are sent into the RPN, and bbox (bounding box) regression and binary classification (foreground and background) are performed on them, while part of the candidate ROIs are removed through non-maximum suppression; (3) Perform ROI Align regression operation on all the remaining candidate ROIs; (4) Perform multi-classification, bounding box regression, and mask generation operations on the normalized candidate ROIs.

[0026] The YOLACT structure is shown in Figure 2 The network is composed of five parts: a feature extraction structure, a prediction module, a mask template generation branch (Protonet), an assembly branch, and a cropping and threshold segmentation module. The feature extraction structure is composed of a backbone network and a feature pyramid network (FPN).

[0027] The algorithm works as follows: It uses a ResNet backbone to extract features and generate feature maps, and FPN to optimize multi-scale features. The two types of features generated through different convolutions are then input into the Protonet segmentation module and the Prediction Head module, respectively, and the two modules run in parallel. The first branch generates the mask template, and the second branch outputs the object category, predicted bounding box, and the confidence scores (mask coefficients) of K masks. Finally, for each instance after Non-Maximum Suppression (NMS) processing, the predicted mask coefficients and the prototype mask are linearly combined through an aggregation branch, and the final mask for each instance is constructed using a Sigmoid nonlinear function.

[0028] ResNet's residual structure effectively solves the problems of loss of global feature information and vanishing or exploding gradients caused by increasing the number of layers in convolutional neural networks. The ResNet residual structure is as follows: Figure 3 As shown, direct connection channels are added to the network, allowing the target feature information of the original input to be directly passed to deeper layers, preserving the integrity of the features. The residual structures of ResNet can be stacked arbitrarily as needed; common networks include ResNet18, ResNet34, ResNet50, and ResNet101. The entire network only needs to learn the differences between the input and output, reducing the difficulty of feature learning for the model. This experiment selects ResNet50 and ResNet101 as the two backbone networks, which, compared to ResNet18 and ResNet34, can learn more and better image features without losing information.

[0029] The following experiment further illustrates this point: 1. Data Acquisition and Preprocessing 1.1 Experimental Data Acquisition Lemon samples were collected from the Lemon Germplasm Resource Nursery of Chongqing Academy of Agricultural Sciences between August and October 2024. To reduce overfitting of the network model due to insufficient diversity of lemon training samples, mature and immature fruits of 24 varieties, including Eureka, Beijing Lemon, and Werner (see Table 5-1), were collected to ensure sufficient sample images of lemon fruits of different varieties and maturity levels.

[0030] After collecting samples in the field, laboratory images of the lemon fruits were quickly taken, including cross-sectional images. Images with issues such as overexposure and blurriness were manually removed, resulting in 454 original, valid images. Some original images of lemon varieties are shown below. Figure 4 As shown.

[0031] 1.2 Constructing training data set The quality of deep learning data determines the performance of the model to some extent, but when the image and annotation information are obtained, it is particularly important to choose the data storage method. The present application uses the open source image annotation software Labelme to manually annotate the lemon fruit pictures. This software is a visual image annotation tool written in Python and using Qt for its graphical interface. The operation interface and annotation process are as follows Figure 5 As shown. The fruit locule outline is surrounded by an irregular polygon through the Create Polygons option in the Labelme software, because the present application only detects one class of fruit locule, so the annotation box is uniformly represented by fruit locule, that is, the picture is only divided into two categories of fruit locule and background labels, and finally saved and generated json file to ensure that the annotation file and the picture name correspond one by one.

[0032] The content of the generated.json file is as shown Figure 6 The storage information of the annotation mainly includes the name of the picture, the storage path of the picture, the information of the annotation box part mainly includes the class name of the annotation box and the position information of the box, and "points" records the coordinate position information of all points of the irregular polygon annotation box.

[0033] In this experiment, COCO2017 is used as the data set format. COCO2017 mainly contains two parts: Images and Annotations. Among them, Images contains all image information, including training set, validation set and test set; Annotations stores the corresponding mark files of all images, which are composed of json files.

[0034] 1.3 Data augmentation In deep learning, a large number of sample data is very important, because it can provide enough information to train the model. However, in practice, the size of the data set may be limited by factors such as time, resources and cost, which may have a negative impact on the performance of the model. Therefore, data augmentation is particularly important. Data augmentation is widely used in deep learning, which increases the size of the data set by transforming and expanding the existing data, thereby improving the generalization ability and robustness of the model. Generalization ability refers to the ability of the model to handle new data, while robustness refers to the ability of the model to resist noise and perturbations. In addition, data augmentation can also reduce the risk of overfitting. Overfitting is when the model learns too much about the characteristics of the training data set, resulting in a decline in performance on new data. Since the number of lemon fruit locule images in the self-made data set is relatively small, it may cause the model to overfit during training.

[0035] Data augmentation methods can be divided into offline augmentation techniques and online augmentation techniques. Offline augmentation techniques involve using image processing algorithms to process the dataset images before loading the dataset for network training. This method expands and saves the dataset. Online augmentation techniques embed fixed data augmentation methods into the model framework. During model training, random geometric transformations and noise addition are applied to the input images before they are input into the network for training. After training, the original dataset image information does not change. Online augmentation techniques can reduce GPU consumption while improving the model's feature analysis of small targets.

[0036] Therefore, to enhance the generalization ability and robustness of the model and avoid over-reliance on certain specific attributes, this section selects online data augmentation for the lemon fruit petal instance segmentation dataset during the data processing stage. Various data augmentation methods are used to increase the diversity and complexity of the training samples.

[0037] (1) Geometric transformation To simulate the camera's perspective in real-world scenarios, random rotation transformation is used to obtain images taken from different directions, including translation, flipping, and combinations of these methods.

[0038] (2) Brightness and contrast adjustment Adjusting the brightness and contrast of the original image simulates different lighting conditions.

[0039] (3) Adding noise Adding noise introduces irregularities or randomness, simulating the uncertainty and complexity of the real world, allowing the model to better handle unknown data. By adding noise, the model can better learn the essential features of the data rather than just memorizing certain patterns in the training data. This section chooses Gaussian filtering and salt and pepper noise to add noise.

[0040] Although data augmentation can improve the generalization ability and robustness of deep learning models, there are still some problems in practical applications. For example, excessive data augmentation may lead to overfitting, affecting the performance of the model. In addition, when using data augmentation, the quality and diversity of the data samples need to be considered to avoid overfitting and underfitting. Therefore, when choosing data augmentation methods, the characteristics of the task, the quality and diversity of the dataset, and other factors need to be considered, and the specific application needs to be combined. The original dataset is randomly divided into training, validation, and test sets in the ratio of 7:2:1.

[0041] 2. Experimental design and model training 2.1 Experimental design Due to the different ideas and implementation methods of segmentation methods, the two-stage instance segmentation model Mask R-CNN usually has higher segmentation accuracy, while the single-stage instance segmentation model YOLACT is faster. To explore whether the two instance segmentation methods based on deep learning are suitable for lemon fruit segment segmentation tasks and the influence of different network depths on model performance, ResNet50 and ResNet101 were used as the backbone feature extraction network, and Mask R-CNN and YOLACT two instance segmentation models were combined to build four experimental models. Experiments were conducted on the self-built lemon fruit segment segmentation dataset, and the experimental results were analyzed to evaluate the performance of each model. By comparing the performance and characteristics of different models, the optimal model for segmenting lemon fruit segments and counting segments was selected.

[0042] 2.2 Experimental environment The main hardware configuration of the test platform is Intel(R) Xeon(R) Platinum 8362 CPU with a main frequency of 2.80 GHz, NVIDIA GeForce RTX 3090 GPU, and 24 GB of memory. The operating system is Windows 11. The relevant server environment and application configuration are shown in Table 1-1.

[0043] Table 1-1 Relevant server environment and application configuration

[0044] 2.3 Model training The setting of hyperparameters is one of the prerequisites for training lemon fruit segment instance segmentation models, and has a great influence on the effect of model training. Unlike convolution kernel parameters that can be automatically updated with model iteration training, hyperparameters need to be manually set before model training. In addition to relying on experience to estimate model training parameters, timely adjusting hyperparameters according to model training results is also a key part of the experimental process, which can speed up the training process and improve the robustness of the model.

[0045] The designed Mask R-CNN and YOLACT instance segmentation models were trained on the self-built lemon fruit segment segmentation dataset. The experimental hyperparameter settings are shown in Table 1-2. To reduce the difficulty of model convergence, the idea of transfer learning was used to use the parameter model weight of the pre-trained backbone network on the large dataset COCO2017, so that the parameters of the feature extraction network can be relatively optimal, avoiding the problem of model overfitting. Then, the backbone network along with other structures was fine-tuned on the self-built dataset to achieve better prediction and segmentation of lemon fruit segments.

[0046] Table 1-2 Parameter settings

[0047] 3. Results and Analysis 3.1 Evaluation Index To analyze the effectiveness of the algorithm in the instance segmentation of fruit locules, the segmentation results obtained by the algorithm need to be evaluated. The manually labeled label map is used as the standard, and the more similar the label map and the network segmentation result, the better the accuracy of the network model. Therefore, the invention selects the mean average precision (mAP) and the average detection time t as the general evaluation index of instance segmentation model to evaluate the effect of the prediction box and instance segmentation mask. The solving steps and calculation formula of mAP are as follows: (1-1) (1-2) (1-3) (1-4) (1-5) (1) Intersection over Union The Intersection over Union (IoU) represents the consistency of the segmentation result and the actual one. As shown in Figure 9 , A represents the detection box, and B represents the true box. The IOU calculates the probability of the overlap between the predicted box and the true box, that is, the proportion of the intersection (A∩B) in the union (A∪B), which takes a value in the range of [0, 1]. The larger the IoU value, the better the segmentation effect. Usually, IoU is compared with a threshold value. If the IoU is greater than or equal to the threshold value, the detection box is considered correct, otherwise it is considered incorrect.

[0048] (2) Confusion Matrix The confusion matrix is a matrix for evaluating the performance of a classification model. It is based on the true label manually labeled and the model prediction result. The test samples are divided into four categories according to their classification, which are true positive (TP), false positive (FP), true negative (TN), and false negative (FN). The specific judgment depends on the comparison of the IoU of the prediction box and the true box with the set threshold value. The confusion matrix can intuitively show the classification effect of the model, thereby helping the inventor to analyze and adjust the performance of the model. Its matrix form is as follows.

[0049] Table 1-3 Evaluation Samples

[0050] TP represents the number of correctly predicted fruit locules, FN represents the number of targets belonging to fruit locules but misjudged as background, and FP represents the number of targets belonging to background but misjudged as fruit locules. The confusion matrix can play a key role in model evaluation and improvement, helping us better understand the classification performance of the model and adjust the model parameters and improve the model structure to improve the accuracy and robustness of the model.

[0051] (3) Precision (P) and recall (R) Precision and recall are two important indicators for measuring model performance. Precision is the proportion of true samples among the pixels segmented as fruit locules, reflecting the model's ability to identify relevant targets. High precision indicates that the model has a high correct rate in predicting positive samples, and the prediction result is reliable. Recall is the proportion of true fruit locule pixels segmented in the segmentation result, reflecting the model's ability to find all relevant targets, i.e., the coverage rate of the model. High recall indicates that the model can detect more true positive examples, but may also lead to an increase in false positives. Accuracy and recall reflect the performance of the lemon fruit locule segmentation algorithm from different aspects.

[0052] (4) Average precision (AP) and mean average precision (mAP) As new samples are continuously added in the fruit locule prediction process, precision and recall change constantly. Average precision is an indicator that combines the changes in precision and recall to analyze model performance, and can express the classifier performance of the image segmentation model. AP value is the area surrounded by the P-R curve and coordinate axes, and mAP is the average of AP values of all target categories. Since this invention only contains one category of fruit locules, mAP is equal to AP value.

[0053] When calculating mAP, a threshold needs to be determined, i.e., when the model's confidence score is higher than the threshold, it is judged as the target category. This invention selects two different IoU thresholds, i.e., 0.50 and 0.75, represented as AP@[0.50] and AP@[0.75]. AP@[0.50] and AP@[0.75] are the average precision calculated by fixing the IoU threshold to 0.5 and 0.75, respectively.

[0054] 3.2 Results and analysis (1) Loss curve analysis In this invention, ResNet50 and ResNet101 are used as the backbone feature extraction network and YOLACT model for experiments on the lemon fruit locule segmentation dataset. The loss curve of each model during iteration is shown in Figure 10 .

[0055] The loss function gradually decreases with the increase of the number of iterations, and the loss function of the four models gradually tends to be stable at the end of training, indicating that at the end of training, the four models do not appear overfitting or underfitting phenomenon. It can be seen from the figure that the loss value of the YOLACT model with ResNet50 as the backbone feature extraction network is larger, and when the model tends to be stable, the loss value is about 2.2, and the loss curve fluctuates greatly. When ResNet101 is used as the backbone feature extraction network, the loss value is about 2.0 when training converges. By observing the loss curve of Mask R-CNN, it can be seen that whether ResNet50 or ResNet101 is used as the backbone feature extraction network, the model is steadily declining without large fluctuations, and the loss value is small, which is more stable than YOLACT. The loss value of the MackR-CNN model with ResNet50 as the backbone feature extraction network is about 0.6 when it converges, and the loss value of the MackR-CNN model with ResNet101 as the backbone feature extraction network is about 0.5 when it converges. Compared with YOLACT and Mask R-CNN models, Mask R-CNN model performs better, and the bounding box loss, classification loss and mask loss are also smaller, which indicates that the positioning accuracy of Mask R-CNN model is higher and the convergence speed is faster when identifying different varieties and different maturity of lemon fruit locules.

[0056] (2) Performance evaluation of instance segmentation model The Mask R-CNN with ResNet50 and ResNet101 as the backbone feature extraction network was compared with the YOLACT model with ResNet50 and ResNet101 as the backbone feature extraction network, and the evaluation indexes were the average precision of bounding box and mask when IoU took different values and the inference time. The segmentation performance of the two models is shown in Tables 1-4.

[0057] Table 1-4 Comparison of segmentation performance of different network models

[0058] As can be seen from the table, the model segmentation indicators Bbox AP and Mask AP of the four models are the highest when the IoU threshold is 0.5. When ResNet50 and ResNet101 are used as the backbone feature extraction network, the average precision of the bounding box of the single-stage instance segmentation model YOLACT is 87.0% and 88.0%, and the average precision of the mask is 83.9% and 84.8%, respectively. The average precision of the bounding box of the two-stage instance segmentation model Mask R-CNN is 98.9% and 99.0%, and the average precision of the mask is 97.5% and 97.7%, respectively. In terms of model segmentation accuracy, the segmentation performance of the two Mask R-CNN models is higher than that of the two YOLACT models, and the model segmentation performance of the model with ResNet101 as the backbone network is higher than that of the model with ResNet50. However, in terms of image inference speed of the model, the two YOLACT models are superior to the two Mask R-CNN models, and among them, the average processing time of YOLACT for each image is the fastest, which can reach 0.0241s. These data show that the segmentation accuracy of the Mask R-CNN-ResNet101 model is relatively accurate, and the segmentation speed of the YOLACT-ResNet101 model is relatively fast.

[0059] Figure 11 The segmentation effects of the Mask R-CNN and YOLACT models with different backbone networks on the test set images of lemon fruit locules were tested, and the segmentation results of each model were predicted according to the models with the highest mAP selected from Tables 1-4.

[0060] When the fruit locules in the image are clear and the interval features are obvious, any of the four models can identify the locules. Compared with the two YOLACT models, the lemon fruit locule contours are segmented more completely in the Mask R-CNN, and the segmentation effect of YOLACT at the junction of the flesh and inner pericarp is relatively poor. In the locules with blurred layers, YOLACT has the situation of missing detection or false detection, and cannot accurately segment the contour of the lemon fruit locule. The fruit color, size, locule characteristics and seeds have little effect on the segmentation performance of the two Mask R-CNN models, and the segmentation effect of the junction of the adjacent locules of the two Mask R-CNN models has no obvious difference compared with the backbone network ResNet50 and ResNet101. The results show that the Mask R-CNN model can accurately segment the locule region from the transverse image of the lemon fruit after training.

[0061] (3) Fruit locule counting results In order to verify the locule counting ability of the four models, the original 45 test set images were detected by Mask R-CNN and YOLACT network to obtain the segmentation images of lemon fruit locules. The number of locules of each fruit was counted as the total number of "fruit locule" detection boxes. The difference between the manual counting true value and the model prediction value of the number of fruit locules is shown in Tables 1-5, wherein the difference of "0" represents correct locule counting. The difference of "-2" and "-1" represents that the predicted number of locules is greater than the actual number of locules, which may be due to the model detecting the reflection or seeds in the same fruit locule as another locule. The values of "1" and "2" indicate that the predicted number of locules is less than the actual number of locules, which may be due to the existence of fuzzy septum between the locules. As can be seen from Tables 1-5, the correct counting rate of the two Mask R-CNN models is much higher than that of YOLACT, and the absolute value of the difference between the predicted value and the true value of the two Mask R-CNN models is within 1. Among them, the counting accuracy of Mask R-CNN-ResNet50 model is the highest, which is 91.1%, which is 4.4 percentage points higher than Mask R-CNN-ResNet101 model and 42.2 percentage points higher than YOLACT-ResNet50 model.

[0062] Table 1-5 Difference distribution of fruit locule counting

[0063] In summary, compared with other models, Mask R-CNN-ResNet50 can learn more features of lemon fruit locules, has the best comprehensive performance, and is more suitable for lemon fruit locule segmentation and counting.

[0064] The present application respectively carries out lemon fruit locule segmentation and counting experiments in Mask R-CNN and YOLACT models with ResNet50 and ResNet101 as the backbone feature extraction network. First, according to the lemon fruit locule segmentation and counting task, the selection idea and principle of the four models are introduced, and then the construction method of the lemon fruit locule dataset is elaborated in detail, including image acquisition, data labeling and data enhancement. Finally, the four models on the self-made lemon fruit locule dataset are compared, and through the comparative analysis of the experimental results, it is found that the Mask R-CNN-ResNet101 model has high segmentation accuracy, but due to its two-stage structure, the running speed is slow. In terms of segmentation effect, there is no obvious difference between the two Mask R-CNN models, and the segmentation effect is good. In terms of locule counting, the Mask R-CNN-ResNet50 model has the highest accuracy, reaching 91.1%. Therefore, the Mask R-CNN-ResNet50 model has the best comprehensive performance and is more suitable for lemon fruit locule segmentation and counting.

[0065] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for counting the locules of lemon fruits based on deep learning, characterized by, The method comprises the following steps: Step 1: Obtain a lemon fruit cross-section image and pre-process the image; Step 2: Manually label the pre-processed image using a labeling tool to generate a label file and build a training data set; Step 3: Perform data enhancement processing on the training data set; Step 4: Select an instance segmentation model, take ResNet50 or ResNet101 as the backbone network, and train the model on the built training data set; Step 5: Use the trained model to segment the locule in the lemon fruit cross-section image; Step 6: Count the locule of the lemon fruit according to the segmentation result.

2. The deep learning-based method for counting the number of segments of lemon fruits according to claim 1, wherein, The pre-processing in step 1 includes cropping, scaling and normalizing the image. 3.The deep learning-based lemon fruit segment counting method of claim 1, wherein, In step 2, the Labelme labeling tool is used for manual labeling, and the locule contour is enclosed with an irregular polygon. The label box is uniformly represented as "fruitlocule", and a json format label file is generated. The label file contains the picture name, storage path, label box category name and position information. 4.The deep learning-based lemon fruit segment counting method of claim 1, wherein, In step 3, the data enhancement processing adopts online enhancement technology, including geometric transformation, brightness and contrast adjustment, and adding noise. The geometric transformation includes one or more combinations of random rotation, translation and flipping. The method of adding noise adopts Gaussian filtering and salt and pepper noise. 5.The deep learning-based lemon fruit segment counting method according to claim 1, wherein, The instance segmentation model in step 4 is Mask R-CNN model or YOLACT model.

6. The deep learning-based method for counting the number of segments of lemon fruits according to claim 5, wherein, When the instance segmentation model is Mask R-CNN model, the instance segmentation model is composed of a backbone network, a region candidate network, a region of interest alignment and an output network. The backbone network is composed of ResNet and a feature pyramid network. 7.The deep learning-based lemon fruit segment counting method according to claim 5, wherein, When the instance segmentation model is YOLACT model, the instance segmentation model is composed of a feature extraction structure, a prediction module, a mask template generation branch, an aggregation branch, and a cropping and threshold segmentation module. The feature extraction structure is composed of a backbone network and a feature pyramid network.

8. The deep learning-based method of counting the locules of lemon fruit according to claim 1, wherein, The hyperparameter settings for model training in step 4 are as follows: IMAGE_SIZE is 5496x3672, LEARNING_RATE is 0.0025, WEIGHT_DECAY is 0.0001, MOMENTUM is 0.9, BATCH_SIZE is 2, EPOCH is 60, and the optimizer is SGD. 9.The deep learning-based lemon fruit segment counting method of claim 1, wherein, In step 5, the evaluation of the segmentation effect of the model adopts mean average precision and average detection time. The calculation of mean average precision is based on intersection over union, confusion matrix, precision, recall and average precision.

10. A deep learning based lemon fruit locule counting system, operating the method of any one of claims 1-9, characterized in that, The system comprises: An image acquisition module for acquiring a lemon fruit cross-section image; A data preprocessing module for preprocessing the acquired image; A data set construction module for manually labeling the pre-processed image using a labeling tool to generate a label file and build a training data set; A data enhancement module for performing data enhancement processing on the training data set; A model training module for selecting an instance segmentation model, taking ResNet50 or ResNet101 as the backbone network, and training the model on the built training data set; The capsule petal segmentation module is configured to segment the capsule petal in the lemon fruit transverse section image by using the trained model. The capsule petal counting module is configured to count the capsule petal of the lemon fruit according to the segmentation result.

Images