A method of grading apples
By combining computer vision and machine learning technologies, an apple grading system was constructed, incorporating features such as fruit stem, color, diameter, and shape. Lightweight detection and grading algorithms were used to solve the problems of low efficiency and high damage in existing apple grading methods, achieving efficient and accurate apple grading.
Patent Information
- Application Number
- CN202411698458.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2044-11-26
AI Technical Summary
Existing apple grading methods suffer from problems such as fruit collisions, surface dents, low grading efficiency, significant human factors, and high labor costs. Furthermore, automatic grading technology is not yet fully mature in terms of fruit grading and non-destructive testing.
Combining computer vision, image segmentation, and machine learning techniques, an apple grading system is constructed. Features such as stem, color, diameter, and shape are introduced. The FDNet lightweight stem detection algorithm, MBFilter algorithm, and GBDT algorithm are used to design a lightweight stem detection and grading model. Multiple features of apples are extracted through image processing and graded.
It achieves efficient and accurate apple grading, avoids secondary damage, improves grading efficiency, reduces resource consumption, and enhances grading accuracy.
Smart Images
Figure CN119649106B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of apple grading evaluation, in particular to an apple grading evaluation method. BACKGROUND
[0002] The problems such as fruit body collision, fruit surface indentation, low grading efficiency, human factor influence and huge labor consumption caused by artificial sorting of apples are paid more and more attention. Apple automatic grading technology has been gradually widely applied in apple industry and is a crucial link in apple industry and market circulation. Under this background, feature extraction of apple appearance, nondestructive testing of fruit skin, classification model design and optimization of fruit grade have become research hotspots.
[0003] With the rapid development of deep learning, nondestructive testing and intelligent sorting technology are more and more widely used in the process of apple grading. Fan et al. combined color and fruit diameter features to detect apple tree fruits and realize size grading, which can effectively detect small targets and unevenly illuminated fruits, and the grading accuracy reaches 90%. Shi et al. proposed a new multi-view spatial network to solve the apple grading task. The model takes the size information of apple as one of the grading standards. First, a well-pretrained lightweight CNN is used to extract low-level features of apples. Second, a spatial feature aggregation module is constructed by bidirectional LSTM and mean pooling to mine the relevant information of apples from multiple angles. The multi-view spatial network has good application prospect in apple grading task. Fan et al. proposed a real-time apple defect detection method based on YOLOV4 deep learning algorithm. The input image is composed of three continuous infrared images. Channel pruning and layer pruning are used to simplify the YOLOV4 network and speed up the detection. At the same time, a non-maximum suppression (NMS) method based on L1 norm is proposed to remove redundant prediction boxes after fine-tuning the pruned network, improving the detection speed of the model. SUMMARY
[0004] In view of the problems in the above background art, based on the research basis of the above apple automatic classification and grading, the present application combines advanced computer vision technology, image segmentation and machine learning technology, constructs an apple grading system, and integrates efficient, accurate and secondary damage-free apple appearance quality grading technology according to the apple grading standard;
[0005] Among them, the present application extracts four important indexes of fruit stem feature, color feature, fruit diameter feature and fruit shape feature, and combines the local grading standard of Red Fuji apple to revise a set of simplified apple grading standard. Three different grade fruits need to carry fruit stem, and the fruit diameter and fruit shape features are further refined,
[0006] In addition, the priority of the four features in the apple grading task of the present application is stem, color, fruit diameter and fruit shape in turn, and if the relevant features do not meet the requirements of the corresponding grade, they are directly classified into the next grade.
[0007] Therefore, in order to achieve the above-mentioned application purposes, the present application adopts the following technical solutions:
[0008] An apple grading evaluation method comprises the following steps:
[0009] S1: For the apple stem detection task, a network crawler and an artificial collection method are proposed to collect high-quality images, and a data set for the fruit stem detection task is constructed. The X-AnyLabeling2.3.0 labeling tool is used to manually label the fruit stem data, and a fruit stem detection data set is constructed;
[0010] S2: For fruit stem detection and fruit body auxiliary positioning, a FDNet lightweight fruit stem detection algorithm is proposed. A high-efficiency FasterNet is used in combination with an SPPF module as a feature extraction network. A multi-scale structure re-parameterization feature fusion network is used to enhance the model's perception ability. A model compression strategy based on LAMP score is used to realize an extremely lightweight detection algorithm.
[0011] S3: For apple feature extraction, an MBFilter algorithm structure is proposed to ensure the clarity of apple details. A preprocessing algorithm is used to accurately segment the fruit body from the background. A DPC-AKNN algorithm is used for apple segmentation. An AdaptiveKNN is used for adaptive calculation of local density to dynamically process apple images of different densities and shapes.
[0012] S4: For the apple grading model, an image processing method is used to extract the coloring ratio, fruit diameter and fruit shape of the apple, which are three important features. The apple grading data set is constructed by grade calibration. A GBDT algorithm is used to realize an efficient apple grading model.
[0013] Preferably, the step S1 specifically comprises an apple grading pipeline device, a FasterNet feature extraction network and a DBB-PANet feature fusion network. The apple grading pipeline device comprises a transmission control system, an image acquisition end, a fruit stem detection, a fruit body image preprocessing, a multi-feature extraction, a grading model evaluation and an automatic sorting, and realizes efficient grading of apples.
[0014] Preferably, the FasterNet feature extraction network is a FasterBackbone composed of three modules of Embedding, Merging and FasterBlock. The visual feature information of the fruit stem is extracted in the shallow level of the network, and the semantic feature information of the fruit stem is extracted in the deep level,
[0015] Among them, the Embedding and Merging two modules are used to convert the dimensions and sizes of the feature map, and the FasterBlock module is constructed based on the partial convolution operator, and the feature information across the channels is fused.
[0016] Preferably, the DBB-PANet feature fusion network introduces a DBB module to reconstruct the C3 module and form a C3-DBB feature fusion module. The C3-DBB module uses a multi-scale idea to capture rich feature information during the training stage. The specific scheme is to extract feature information using four strategies of 1x1conv-bn, 1x1conv-bn+AVG-bn, 1x1conv-bn+3x3conv-bn and 3x3conv-bn, and to combine the four different multi-branches into a single branch structure.
[0017] Preferably, for the combination of extracting feature information using the 1x1conv-bn+AVG-bn and 1x1conv-bn+3x3conv-bn strategies, the two inputs are The output is obtained as shown in the formula:
[0018] wherein represents a convolution operation, represents a padding operation on the bn layer,
[0019] Since represents a 1x1 convolution, the parameters are combined into the convolution kernel of using a linear restructuring strategy, as shown in the formula,
[0020] wherein represents a dimension conversion operation;
[0021] The partial bias term is constructed as shown in the formula:
[0022]
[0023] The total bias term is combined as shown in the formula:
[0024] The AVG operation therein is equivalent to a convolution with equal size and step, and the specific operation is shown in the formula:
[0025]
[0026] Finally, it is combined into a new convolution operation within the branch, as shown in the formula:
[0027] .
[0028] Preferably, in the step S2, the LAMP score-based channel pruning is adopted, the L2 norm of all weights of each input channel is calculated and sorted, the index mapping is established according to the newly generated sequence, the weights and the channels with smaller weights are removed, so as to realize the compression of the model, and the LAMP score is defined as the formula:
[0029]
[0030] wherein, represents a target channel, represents the square sum of the weight of the target channel, represents the square sum of the weight of the remaining channel other than the target channel and having an index greater than that of the target channel.
[0031] Preferably, in the step S3, the calculation formula of the MBFilter algorithm is:
[0032]
[0033]
[0034] wherein, respectively represent the median filtering and the bilateral filtering, respectively represent the Gaussian functions in the spatial domain and the color domain, is a normalization factor, represents the neighborhood of the pixel p.
[0035] Preferably, in the step S3, based on the segmented apple image, the RGB color space is converted into the HSV color space, and the formula for calculating the apple red color ratio is as follows:
[0036]
[0037] wherein, represents the number of apple red pixels, represents the number of all pixels of the apple;
[0038] For the apple shape index , the specific calculation process is as follows:
[0039]
[0040] wherein, represents the vertical diameter, represents the horizontal diameter.
[0041] Preferably, in the step S4, the GBDT algorithm is for the input apple feature data set:
[0042] ,
[0043] wherein represents the color feature, represents the diameter feature, represents the shape feature, represents the grade to which the apple belongs.
[0044] Compared with the prior art, the above one or more technical solutions have the following beneficial effects:
[0045] 1. The fruit stem index is introduced in the apple grading task for the first time, a high-precision lightweight fruit stem detection algorithm FDNet-p is proposed by combining computer vision technology and the current mainstream model compression technology, and the superior performance of the algorithm is proved through a large number of experiments. In the apple grading process, the FDNet-p algorithm is used to screen out apples without carrying fruit stems, and the auxiliary positioning idea is proposed to prevent the camera from triggering the collection operation too early or too late.
[0046] 2. In the apple feature extraction process, an improved DPC-AKNN segmentation algorithm is proposed, and an adaptive brightness equalization strategy is introduced to improve the effect of fruit body segmentation, then image processing technology is used to extract three features of the apple, including color, diameter and shape, and a mapping is formed with the real artificial grading scene.
[0047] 3. Based on the multi-feature information of the apple, multiple performance indicators are used to fully evaluate the performance of the machine learning classification algorithm, and finally the GBDT algorithm is used to realize the intelligent grading of the apple. DETAILED DESCRIPTION
[0048] The drawings constituting a part of the specification of the present application are used to provide a further understanding of the present application, the illustrative embodiments of the present application and the description thereof serve to explain the present application, and do not constitute an improper limitation on the present application.
[0049] Figure 1 is a structural schematic diagram of the whole structure of the apple grading of the present application;
[0050] Figure 2 is a structural schematic diagram of the FDNet network structure and auxiliary positioning of the present application;
[0051] Figure 3 is a structural schematic diagram of the FasterBlock module structure of the present application;
[0052] Figure 4 is a structural schematic diagram of the C3-DBB module of the present application;
[0053] Figure 5 is a structural schematic diagram of the apple feature extraction scheme of the present application;
[0054] Figure 6It is the effect comparison chart of different noise removal algorithms of apples of the present application.
[0055] Figure 7 It is the processing process of the apple brightness equalization algorithm of the present application.
[0056] Figure 8 It is the segmentation effect chart of the DPC-FKNN algorithm of the present application.
[0057] Figure 9 It is the apple color feature extraction schematic diagram of the present application.
[0058] Figure 10 It is the apple fruit diameter feature extraction schematic diagram of the present application.
[0059] Figure 11 It is the apple fruit shape feature extraction schematic diagram of the present application.
[0060] Figure 12 It is the different evaluation index comparison chart of FDNet and YOLOv5-s of the present application.
[0061] Figure 13 It is the performance comparison chart of the lightweight detection model of the present application.
[0062] Figure 14 It is the comparison effect chart of different apple detection algorithms of the present application. DETAILED DESCRIPTION
[0063] In order to enable the person skilled in the art to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.
[0064] With reference to Figures 1-11 , the present application proposes an apple grading evaluation method, comprising the following steps:
[0065] (1) For the apple stem detection task, a network crawler and an artificial collection method are proposed to collect high-quality images, and a set of data set for the fruit stem detection task is constructed. The fruit stem data is manually annotated using X-AnyLabeling2.3.0 annotation tool, and the fruit stem detection data set is constructed, which can enhance the diversity and anti-interference of the data set, and further improve the robustness and generalization ability of the model.
[0066] (2) For fruit stem detection and fruit body assisted positioning, a FDNet lightweight fruit stem detection algorithm is proposed, which uses an efficient FasterNet combined with an SPPF module as a feature extraction network, adopts a multi-scale structure reparameterization feature fusion network to enhance the model's perception ability, and uses a model compression strategy based on LAMP score to achieve an extremely lightweight detection algorithm, which can further improve the efficiency of apple grading in the pipeline and reduce unnecessary resource consumption.
[0067] (3) For apple feature extraction, the MBFilter algorithm structure is proposed to ensure the clarity of apple details, and a preprocessing algorithm is used to accurately segment the fruit body from the background. The DPC-AKNN algorithm is used for apple segmentation, and the AdaptiveKNN is used for adaptive calculation of local density to dynamically process apple images of different densities and shapes.
[0068] (4) For apple grading model, image processing methods are used to extract the color ratio, fruit diameter and fruit shape of apples, and an apple grading dataset is constructed. The GBDT algorithm is used to realize an efficient apple grading model.
[0069] Specifically as follows:
[0070] 1. Available fruit stem detection dataset:
[0071] For apple fruit stem detection tasks, there is currently a lack of publicly available fruit stem detection datasets. In the introduction of complex scene apple dataset, which contains multiple target, small target fruit stem image in different scenes, the model has poor anti-interference performance. For apple images with various fruit stem targets in complex backgrounds, use image rotation (15°, 45°, 60°) technology to randomly enhance part of the dataset, so that the model cannot fully learn the different poses of apples in the grading pipeline.
[0072] Table 1: Apple grading custom standards
[0073] 2. Fruit stem detection and fruit body assisted positioning:
[0074] In order to screen out apples with missing fruit stems, a lightweight apple fruit stem detection algorithm is designed to prevent it from entering subsequent grading operations, further improving the efficiency of apple grading in the pipeline and reducing unnecessary resource consumption. Another task of fruit stem detection is to participate in fruit body positioning, that is, to prevent the front and rear acquisition cameras from triggering too early or late, resulting in the inability to accurately capture the complete front and rear fruit body view. The FDNet lightweight fruit stem detection algorithm is used, and the method is as follows:
[0075] 1) The apple grading pipeline device is divided into seven parts: conveying control system, image acquisition end, fruit stem detection, fruit body image preprocessing, multi-feature extraction, grading model evaluation and automatic sorting. The efficient grading of apples is realized.
[0076] The conveying control system includes a standard conveyor belt and a driving motor. The driving motor can dynamically adjust the running speed of the pipeline according to the needs. The image acquisition end includes a high-performance computer, an image acquisition device, a high-definition camera, etc. Three high-definition wide-angle distortion-free industrial cameras are placed in the image acquisition device. When shooting images, the ping-pong ball auxiliary calibration method is used first. By calculating the error between the diameter of the ping-pong ball in the collected image and the true diameter, the shooting distance of the camera and the apple is determined. When the apple is detected to reach the top camera field of view, the camera of the apple top view acquisition area will detect whether the apple carries a fruit stem. The apple without fruit stem will not trigger the front view and rear view acquisition action and will not enter the subsequent apple grading process, which reduces the consumption of resources to a certain extent.
[0077] At the same time, the fruit stem is used for auxiliary positioning. When the fruit stem detection frame completely enters the set area, the apple with fruit stem will trigger the computer to control the other two cameras to shoot, in turn for the front view and rear view of the apple. The front view and rear view acquisition is based on the maximum cross-sectional area. The apple can be approximated as a sphere. The high-definition camera uses the equidistant view acquisition method, so that the front and rear two acquisition angles are the entire side view of the apple. Then the image processing operation is used to extract the key feature information of the apple, and then the model comparison, training and evaluation are carried out. The model with the best performance is selected for apple grade determination. Finally, the mechanical arm is used to sort the apples into the designated grade area.
[0078] 2) The FasterNet feature extraction network (represented as FasterBackbone) is composed of Embedding, Merging and FasterBlock three modules. The visual feature information of the fruit stem is extracted in the shallow layer of the network, and the semantic feature information of the fruit stem is extracted in the deep layer. Among them, the Embedding and Merging two modules are used to convert the dimensions and sizes of the feature maps, and the FasterBlock module is constructed based on the partial convolution operator, which fuses the feature information across channels;
[0079] Another task of fruit stem detection is to participate in fruit body positioning, that is, to prevent premature or delayed triggering of front and rear acquisition cameras to shoot, which leads to the inability to accurately shoot the complete front and rear fruit body view. In image acquisition, the fruit stem detection positioning frame is combined with the shooting trigger operation. When the fruit stem completely appears in the specified position within the specified area, shooting is performed to ensure that completely clear multi-angle apple pictures are collected.
[0080] The DBB-PANet feature fusion network is adopted to introduce the DBB module to reconfigure the C3 module, and a C3-DBB feature fusion module is formed. The C3-DBB module uses the multi-scale idea to capture rich feature information in the training stage. The specific scheme is to extract feature information by using four strategies of 1x1conv-bn, 1x1conv-bn+AVG-bn, 1x1conv-bn+3x3conv-bn and 3x3conv-bn, and to combine the four different multi-branches into a single-branch structure,
[0081] For the combination of the 1x1conv-bn+AVG-bn and 1x1conv-bn+3x3conv-bn strategies for extracting feature information, the two inputs are The output is obtained as shown in the formula:
[0082] Among them, represents a convolution operation, represents a padding operation on the bn layer,
[0083] Since represents a 1x1 convolution, the parameters are combined into the convolution kernel of using a linear reorganization strategy, as shown in the formula,
[0084] Among them, represents a dimension conversion operation;
[0085] The partial bias term is constructed as shown in the formula:
[0086]
[0087] The total bias term is combined as shown in the formula:
[0088] The AVG operation in it is equivalent to a convolution with the same size and step length. The specific operation is shown in the formula:
[0089]
[0090] Finally, it is combined into a new convolution operation within the branch, as shown in the formula:
[0091] .
[0092] 3. For apple feature extraction:
[0093] In the image acquisition device, the overhead view collected by the top camera is detected by the FDNet-p algorithm, and when the apple is in the best shooting area, the remaining two cameras are triggered to shoot the front view and the rear view of the apple, and then the apple images of the three perspectives are batch processed using image processing methods. In the apple feature extraction stage, first, the influence of foreground image noise is suppressed, then the brightness is balanced, the apple contour is segmented, and finally the color, fruit diameter and fruit shape are extracted as three important features.
[0094] 1) When the high-definition camera collects apple images, it ensures the clarity of apple details, but it is also susceptible to small noise in the natural environment, which is not conducive to the extraction of apple features in the subsequent process. To solve the above problem, the present application proposes a noise processing method (MBFilter) that combines median filtering and bilateral filtering. The MBFilter algorithm rearranges each pixel and its neighborhood in the image, selects the median value for replacement, and combines the spatial domain and color domain Gaussian weighted filtering method. The calculation formula is as follows:
[0095]
[0096]
[0097] wherein median filtering and bilateral filtering, spatial domain and color domain Gaussian function, is a normalization factor, indicates the neighborhood of pixel p.
[0098] The MBFilter filter first uses median filtering to suppress noise to the greatest extent, then uses bilateral filtering to adaptively smooth the noise area in the image, further processes small noise, and ensures that the apple contour is clear and important detail information is not lost. The specific effect is shown in Figure 6 As shown in the figure, three different intensity salt and pepper noises are simulated in the experiment, and it can be seen from the figure that a small amount of small noise remains on the surface of the apple after median filtering. The MBFilter algorithm in the experiment has good effect on processing small noise, and the image detail information is well preserved.
[0099] 2) The background information has a greater impact on the result of apple feature extraction, so it is necessary to accurately segment the fruit body from the background. In order to ensure more accurate apple segmentation, the present application designs a set of preprocessing algorithms for the task scenario of apple grading pipeline to deal with the problem of uneven brightness on the surface of apples. First, the image in RGB space is converted to LAB space, the L component representing brightness is separated, adaptive histogram equalization is used for brightness processing, then the processed L component, A component and B component are merged, and finally the image in LAB space is converted to RGB space again. The specific effect is shown in Figure 7 As shown in the figure, the surface brightness of the processed image tends to be uniform, which partially solves the problem of uneven surface illumination of apples after camera acquisition.
[0100] 3) In the experiment, the preprocessed apples are segmented using clustering algorithm. DPC algorithm is a density-based clustering algorithm that can automatically determine the clustering center and the number of clusters, but the density calculation method of local density of sample points is based on the cutoff distance, which fails to consider the distribution of samples in the neighborhood, and the value of cutoff distance has a greater impact on the apple segmentation result. In view of the above problems, the present application proposes an improved DPC clustering algorithm (DPC-AKNN) for apple segmentation, which uses Adaptive KNN to adaptively calculate local density, does not depend on global distance threshold, automatically adjusts to the local density of points, and more effectively identifies the peak value of density, so that the DPC algorithm dynamically processes apple images of different densities and shapes, reducing the sensitivity of the DPC algorithm to the cutoff distance parameter. DPC-AKNN first calculates the Euclidean distance of each pixel point to other points in the apple image segmentation task, then uses AKNN algorithm to calculate the local density, and obtains the minimum distance, finally obtains the cluster center of clustering, and assigns the class label of the remaining pixel points, realizing the segmentation of apples and background.
[0101] To prove the effectiveness of the DPC-KNN algorithm proposed in the experiment in the apple segmentation task, the segmentation effect of apples under different resolutions is verified, the clustering, density decision graph and clustering distribution graph are drawn, and the segmentation effect is compared with the mainstream KMeans clustering algorithm, and the specific visualization result is shown in Figure 8 As shown in the decision graph and clustering graph, under different resolutions, DPC-AKNN can accurately find the optimal decision boundary; in the segmentation effect comparison part, KMeans clustering algorithm fails to accurately segment the weak light part, DPC-KNN algorithm accurately segments the complete fruit body, and the segmentation effect of the weak light edge is better, the algorithm has strong anti-interference ability, and can meet the demand of fruit body segmentation task in apple grading pipeline.
[0102] 4) Color feature of apple is a key indicator in grading task. Based on the segmented apple image, the RGB color space is converted to HSV color space, and the red region is extracted according to the threshold range in HSV color space representing the approximate red color of apple. The specific extraction process is visualized, and the red color ratio of apple is calculated , and the specific calculation process is as follows:
[0103]
[0104] Among them, represents the number of red pixels of apple, represents the number of all pixels of apple.
[0105] 5) Based on the front view, the fruit diameter feature is extracted. Firstly, the minimum circumscribed rectangle is calculated. In order to improve the accuracy of fruit diameter measurement, the minimum circumscribed circle is introduced to assist the measurement. The results measured by the above two methods are arithmetically averaged, and finally the pixel diameter of apple image is obtained. Taking table tennis ball (actual diameter is 40mm) as reference object, the actual fruit diameter size is indirectly calculated by table tennis ball diameter, and the fruit diameter feature is extracted by the method of weighted minimum circumscribed rectangle and minimum circumscribed circle.
[0106] 6) Apple is a kind of spherical fruit. The similarity to standard circle is used as a grading indicator, that is, fruit shape index. Canny is an algorithm suitable for edge detection task. Gaussian filter is used to convolve the image to suppress the influence of noise, and then Sobel operator is used to calculate the horizontal and vertical gradients of the image. The pixels along the gradient direction are checked, and the non-maximum value pixels are suppressed, and only the local maximum value is reserved. Strong edges and edges are further distinguished, and strong edges and weak edges are connected to form continuous edges. In this paper, Canny algorithm and minimum circumscribed rectangle are used to calculate apple fruit shape index. In this paper, the segmented RGB image is converted to grayscale image, and the edge of fruit body is detected by Canny algorithm. The visual operation process is shown in Figure 11 . The edge image of apple is used to calculate the vertical diameter and horizontal diameter of apple by minimum circumscribed rectangle, and then the apple fruit shape index is calculated. The specific calculation process is as follows:
[0107]
[0108] Among them, represents the vertical diameter, represents the horizontal diameter.
[0109] 4. In the research of apple grading model:
[0110] The shooting rate of the shooting camera is 30 frames per second, which ensures the fluency of the apple image acquisition action; meanwhile, the IR-CUT function is equipped, the problem of color deviation is corrected, the brightness of the low light environment is improved, the IR-CUT double filter switcher starts to work under the condition of sufficient light, the CCD restores the true color of the apple, when the photosensitive sensor senses insufficient light, the automatic LED infrared fill light (the light source is invisible to the naked eye), the infrared cutoff filter is automatically removed, the infrared filter starts to work, so that the camera makes full use of the LED infrared light, greatly improves the image acquisition performance in low light environment, and ensures the clarity and accuracy of the collected apple image. The industrial camera shoots three perspective photos of each apple and transmits them to the local computer, and then the computer saves the information as an image file. The present application takes 126 apples as the research object of the grading model, and uses the image processing method to extract the corresponding color feature, fruit diameter feature and fruit shape feature, and finally divides the corresponding grade fruit according to the apple grading standard, and divides the training set and the test set according to the ratio of 7:3.
[0111] The image processing method is used to extract the coloring ratio, fruit diameter and fruit shape of the apple, grade calibration is carried out, and an apple grading data set is constructed, and a gradient boosting tree (GBDT) algorithm is selected to realize an efficient apple grading model. GBDT is an integrated learning algorithm, using CART tree as a weak learner, using the residual of the sample in the current learner as the label of the next learner for training, fitting the negative gradient of the current model, minimizing the loss function, and constructing the GBDT model through multiple rounds of iterative training, which has good generalization. The GBDT algorithm is used for the input apple feature data set:
[0112] ,
[0113] Among them, the coloring ratio feature is represented by C, the fruit diameter feature is represented by D, the fruit shape feature is represented by S, and the grade of the apple is represented by Y. Since the apple grading is a classification problem, the log-likelihood loss function is selected to evaluate the model.
[0114] For the above method, refer to Figures 12-14 The present application provides the experimental results and analysis:
[0115] 1. Experimental environment and parameter setting
[0116] The experiment of the present application is based on an operating system with version number CentOS7.9.2009, and the processor is a 12vCPU Intel (R) Xeon (R) Platinum8255C CPU@2.50GHz. Considering that different size models have different requirements for video memory, a high-performance graphics card with model NVIDIA GeForce RTX3090 is selected, the video memory size is 24G, the integrated development environment is Miniconda3, the development language is Python (version number is 3.8.0), two mainstream deep learning frameworks, Pytorch (version number is 1.11.0) and MMDetection (version number is 3.2.0), are used to build a fruit stem detection model, the training process uses a CUDA acceleration model with version number 11.3, the input image resolution size is 640*640, the Adam method combining momentum optimization and adaptive learning rate strategy is used to optimize the training process of the model, the initial learning rate is 1e-3, the batch size of the image transmitted into the network for training is 64, the baseline model and the improved model are both trained for 200 rounds according to the above parameter settings, and the last 10 rounds are closed mosaic data enhancement. The scikit-learn machine learning library is used to build an apple grading model.
[0117] 2. Fruit stem detection model evaluation index
[0118] The present application uses common evaluation indexes of detection tasks to evaluate the performance of the fruit stem detection model, which are recall R (Recall), precision P (Precision), (mean Average Precision), (Average Precision), the computational complexity (GFLOPS) and FPS of the model, and the specific calculation formula is shown as follows:
[0119] ;
[0120] ;
[0121] ;
[0122]
[0123] Where n refers to the number of categories, only one category of fruit stem in the experiment, mAP refers to the average precision calculated on the fruit stem category, P refers to the proportion of actual fruit stems among all instances detected as fruit stems, which measures the false detection degree of the model, R refers to the proportion of instances correctly detected as fruit stems among all actual fruit stems, which measures the missed detection degree of the model; TP represents the number of correctly predicted fruit stems, FP represents the number of background content misdetected as fruit stems, and FN represents the number of actual fruit stems that the model fails to detect.
[0124] 3. Fruit stem detection experiment analysis
[0125] 3.1 Comparison experiment of different feature extraction networks
[0126] To prove the effectiveness of FasterNet-t0 feature extraction network, a comparison experiment with different feature extraction networks is set up, mainly including mainstream lightweight networks (MobileNetv3 and GhostNet), and three different sizes of FasterNet-t in the FasterNet network series. To ensure the fairness of the experimental comparison, the feature fusion network is the PANet of YOLOv5s, and the specific comparison experiment results of different feature extraction networks in the fruit stem detection task are shown in Table 2.
[0127] Table 2: Comparison experiment of different feature extraction networks
[0128] According to the experimental results in Table 2, for the fruit stem detection task, the comprehensive performance of FasterNet-t0 network is optimal, the model is reduced by 2.8MB, GFLOPS is reduced by 4.6, and the precision is almost unchanged, which shows that the PConv operator reduces the redundant calculation of the model to a certain extent and reduces the complexity of the model; Although MobileNetv3 and GhostNet networks greatly reduce the parameter quantity and computational quantity of the network, the mAP@0.5:0.95 value is severely lost, which is difficult to meet the high precision requirement in the actual fruit stem task. In addition, the precision values of FasterNet-t1 and FasterNet-t2 are improved to varying degrees, but the model calculation and size are large, and compared with different models, FasterNet-t0 is the best choice as the feature extraction network.
[0129] 3.2 Ablation experiment of improvement process
[0130] The application proves the improvement effect of the proposed improvement strategy on the fruit stem detection task through a series of ablation experiments, and is based on a YOLOv5s detection network, and specifically includes: using FasterNet-t0 to improve the feature extraction network, proposing DBB-PANet to improve the feature fusion network, and combining the two methods for improvement. The ablation experiment results of different improvement strategies in the fruit stem detection task are shown in Table 3.
[0131] Table 3: Ablation experiment of improvement process
[0132] According to the ablation experiment results in Table 3, after using FasterNet-t0 as the feature extraction network, the model's calculation amount GFLOPS is reduced by 4.6, and the model size is reduced by 2.8, realizing the lightweight of the model, which shows that the PConv operator in FasterNet effectively reduces the memory access frequency, and the accuracy is almost the same as the baseline model; the P value of the model using DBB-PANet to improve the feature fusion network is improved by 0.3%, and the R value is improved by 2.3%, and the accuracy is slightly increased, which shows that DBB-PANet has an improvement effect on the performance of the fruit stem detection task, and the multi-scale structure reparameterization strategy effectively enhances the recognition ability of different size fruit stems; compared with the baseline model, the P value of the YOLOv5-s model improved by combining the two strategies is improved by 2.7%, the mAP@0.5 value is improved by 0.8%, and the model's calculation amount GFLOPS is reduced by 1.8. The DBB-PANet fully fuses the feature information extracted by the FasterNet-t0 network, realizes the increase of detection accuracy, and reduces the calculation amount of the model.
[0133] Figure 12 For comparison of different evaluation indexes of FDNet and YOLOv5-s. (a) is the bboxloss training process curve. (b) is the mAP@0.5 training process curve. (c) is the obj_loss training process curve. (d) is the PRCurves of YOLOv5-s on the validation set. (e) is the PRCurves of FDNet on the validation set
[0134] In order to more intuitively show that the FDNet proposed in the application is better than the baseline model YOLOv5-s, the bbox_loss, obj_loss and mAP@0.5 in the training process are visualized and compared, as shown in Figure 12 From (a) and (c) in Figure 12 It can be seen that the two different loss curves in the training process of FDNet are better than YOLOV5-s, which shows that the convergence effect of the model is good, and the mAP@0.5 value is also higher than YOLOv5-s, and the specific effect is as shown in Figure 12(b) is shown. Precision-Recall Curve is used to further analyze the detection accuracy of FDNet and YOLOv5-s on the validation set, Figure 12 (d) shows the detection results of FDNet, Figure 12 (e) shows the detection results of YOLOv5-s, wherein the mAP@0.5 value of FDNet is 0.969, which is higher than the mAP@0.5 value of YOLOv5-s.
[0135] 3.3 Model compression comparison experiment
[0136] In order to fully prove the influence of the pruning strategy on the performance of the model, the present application selects different model compression strategies to carry out sufficient comparison tests, which are three mainstream pruning algorithms of Slim, L1 and Lamp, and at the same time evaluates two key indicators of inference time and FPS in low resource GPU devices. The specific comparison experiments of different pruning strategies are shown in Table 4.
[0137] Table 4: Model compression comparison experiment
[0138] Note: The values of Latency and FPS are tested in the environment of 1080Ti GPU.
[0139] According to the model compression comparison experiment in Table 4, when the compression ratio is set to 2.0, in the comparison of various evaluation indicators, the Lamp pruning algorithm has the best comprehensive effect, compared with the Slim and L1 pruning algorithms, the model size of the Lamp algorithm after pruning is the smallest, the mAP@0.5:0.95 value decreases the least, and the inference speed on low resource devices is the fastest, the Latency value is 0.268±0.032, and the FPS value is 363.0, which fully illustrates the effectiveness of the Lamp pruning algorithm. For the adaptive structured pruning on the channel level, the original convolution structure of the pruned model is still retained, which is more friendly to hardware resources. In addition, in order to explore the optimal strategy of model compression, based on the Lamp pruning algorithm, comparison tests with different compression ratios are added, and it is found that when the compression ratio is 4.0, the compression effect of the model is the best, the mAP@0.5 value is 96.6, the GFLOPs value is 3.4, the model size is 2.5, the Latency is 0.200±0.113, and the FPS is 499.5. In order to more intuitively visualize the pruning effect, the channel number change of each layer of the model before and after using the Lamp pruning (compression ratio 4.0) is plotted.
[0140] 3.4 Model comparison experiment
[0141] The superiority of the FDNet proposed in the application in the fruit stalk detection task is proved by setting a series of comparative experiments, and the front-line mainstream target detection model is fully compared and verified in multiple dimensions. The FDNet proposed in the application is a lightweight single-stage target detection algorithm, therefore, the YOLO series lightweight single-stage detection algorithm is mainly selected for comparison, mainly including YOLOv3, YOLOv5, YOLOv6, YOLOv8, YOLOv9, YOLOv10 and YOLOv11. In addition, in order to fully verify the performance of the FDNet, the current mainstream non-YOLO series target detection algorithm is also added, mainly including FasterRCNN, GFL, RTMDet, YOLOx and TOOD, and the specific comparative experimental results are shown in Table 5.
[0142] Table 5: Comparative experiments of different detection models
[0143] According to the quantitative analysis results of different detection models in Table 5, it can be seen that the fruit stalk detection model of the FDNet (not compressed) proposed in the experiment has better detection performance than other detection models, and the mAP@0.5 value reaches the highest, and the specific value is 96.9%. Among the YOLO series target detection algorithms, the detection performance of YOLOv6s is the best, but the parameter quantity and the calculation quantity are large, and the device resource requirement is high. Compared with YOLOv6s, the mAP@0.5 value of the FDNet algorithm is 0.09 higher, the mAP@0.5:0.95 value is almost the same, but the GFLOPS and the model size are much smaller than YOLOv6s. Compared with the remaining models with equivalent calculation quantity, the detection accuracy of the FDNet model is higher. Among the non-YOLO series target detection algorithms, the FDNet model is in the leading level in terms of detection accuracy, calculation quantity and model size. After using the compression strategy to process the model, the P value of the FDNet-p model is 96.3%, the R value is 89.4%, the mAP@0.5 value is 96.6%, the mAP@0.5:0.95 value is 67.9%, the GFLOPS value is only 3.4, and the model size is only 2.5MB. Compared with the index values of the FDNet, the FDNet-p implementation model is slightly compressed, which fully illustrates the high efficiency of the LAMP pruning algorithm. Among the many target detection models, the resource consumption of the FDNet-p model is the smallest, the calculation quantity of the model after LAMP pruning is greatly reduced, the GFLOPS value is the smallest, only 3.4, and the accuracy of the FDNet-prune remains at a high level.
[0144] In order to more intuitively show the excellent performance of the compressed FDNet model (FDNet-p), the lightweight detection models of the YOLO series (v5, v6, v8, v9, v10 and v11) are selected for comparative analysis, and the specific visualization effect is as shown in Figure 13 In the fruit stem detection task, compared with the other lightweight models, the proposed FDNet-p requires the least GFLOPS, has the smallest model size, is easy to deploy in devices with insufficient resources in the pipeline, and at the same time maintains the highest mAP@0.5 value, meeting the accuracy requirements of the fruit stem detection task.
[0145] The detection effect diagram set in the real scene further shows the strong robustness and high detection accuracy of the proposed FDNet-p. In order to ensure the fairness of the detection effect, the lightweight detection models of the YOLO series (v5, v6, v8, v9, v10 and v11) are still selected for comparative analysis of detection effect, and the specific detection result comparison is as shown in Figure 14 Among them, four different scenes are selected, 1. Multi-target + complex background (Sample1), the detection accuracy of FDNet-p in three different apples is the highest, and the detection accuracy of other models is low; 2. Single target + complex background (Sample2), the YOLOv9-n model incorrectly identifies the background as a fruit stem; 3. Simple background + calyx interference (Sample3), other models have different degrees of false detection, and the calyx is identified as a fruit stem, and FDNet-p accurately identifies the fruit stem. 4. Aerial view perspective (Sample4), it can be seen that FDNet-p has the best detection effect.
[0146] 4. Apple grading experiment analysis
[0147] 4.1 Evaluation index of apple grading model
[0148] In the model research of apple grading, a variety of evaluation indexes are set to comprehensively analyze the performance of the apple grading model. These indexes include P (Precision), R (Recall), Jaccard Score and F1 Score, among which F1 Score and Jaccard Score are used to measure the overall performance and accuracy of the model in each class. In order to comprehensively analyze the performance of the model and better understand the performance of the model in multi-class problems, three different average methods, micro, macro and weighted, are introduced. Micro average (micro) evaluates the global performance, reflecting the equal proportion of each sample; macro average (macro) emphasizes the importance of each class; and weighted average (weighted) focuses on the class imbalance problem; these indexes reflect the sensitivity, correctness and applicability of the model from different angles, and the specific calculation formula is as shown in Table 6.
[0149] Table 6: Evaluation indexes of hierarchical model
[0150] Note: C represents the number of categories, N represents the total number of samples, and ni represents the number of samples in the ith category.
[0151] 4.2 Comparative experiment of apple grading algorithm
[0152] The hierarchical algorithm research of the present application is based on three characteristics of color, fruit diameter and fruit shape. A series of comparative experiments of mainstream machine learning algorithms are set up, and different algorithms in the performance of apple grading algorithm are evaluated using multiple evaluation indexes. In order to determine the best apple grading model, six apple grading models are established, and the algorithm performance is evaluated using 10-fold cross-validation algorithm, which effectively reduces the bias in the model evaluation process. Among them, the integrated learning algorithm models are RF (RandomForest), BC (BaggingClassification) and GBDT (GradientBoosting), and the non-integrated learning algorithm is KNN (KNearestNeighbors), SVM and MLP. The specific experimental results of different apple grading algorithms are shown in Table 7.
[0153] Table 7: Comparison of 10-fold cross-validation experimental results of different apple grading algorithms
[0154] According to the cross-validation experimental results of the hierarchical algorithm in Table 7, the overall performance of the non-integrated learning algorithm is weaker than that of the integrated learning algorithm, and the GBDT apple grading algorithm has the best comprehensive performance in the integrated learning algorithm in terms of various evaluation indexes. Specifically, compared with the remaining integrated learning algorithms, the F1Score values of the GBDT algorithm in three different average calculation methods are the best, which are 0.9513, 0.9476 and 0.9506 respectively, and the highest values of JaccardScore, P and R evaluation indexes are also achieved, which reflects the excellent classification ability of the GBDT algorithm, strong robustness and generalization ability, and is suitable for apple grading model.
[0155] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.
Claims
1. A method of grading apples, characterized in that, The method comprises the following steps: S1: For the apple stem detection task, a network crawler and manual collection method are used to collect high-quality images, and a data set for the fruit stem detection task is constructed. The fruit stem data is manually annotated using X-AnyLabeling2.3.0 annotation tool to construct the fruit stem detection data set; S2: For fruit stem detection and fruit body assisted positioning, a FDNet lightweight fruit stem detection algorithm is proposed. FasterNet is used in combination with SPPF module as a feature extraction network, a multi-scale structure reparameterization feature fusion network is used to enhance the model perception ability, and a model compression strategy based on LAMP score is used to realize an extremely lightweight detection algorithm; S3: For apple feature extraction, an MBFilter algorithm structure is proposed to ensure the clarity of apple details. A preprocessing algorithm is used to accurately segment the fruit body and background. A DPC-AKNN algorithm is used for apple segmentation, and an AdaptiveKNN is used for adaptive calculation of local density to dynamically process apple images of different densities and shapes; S4: For the apple grading model, image processing method is used to extract the coloring ratio, fruit diameter and fruit shape of the apple, and an apple grading data set is constructed. GBDT algorithm is used to realize the apple grading model. In step S3, the calculation formula of the MBFilter algorithm is: ； ； wherein respectively denote the median filter and the bilateral filter, respectively denote the spatial and color domain Gaussian functions, is a normalization factor, denotes the neighborhood of pixel p.
2. The method of claim 1, wherein: In step S2, the apple grading pipeline device, FasterNet feature extraction network and DBB-PANet feature fusion network are specifically included. The apple grading pipeline device includes a transmission control system, an image acquisition end, a fruit stem detection, a fruit body image preprocessing, a multi-feature extraction, a grading model evaluation and an automatic sorting, and realizes the grading of apples.
3. A method of grading apples according to claim 2, wherein: The FasterNet feature extraction network is represented as FasterBackbone, which is composed of Embedding, Merging and FasterBlock three modules. The visual feature information of the fruit stem is extracted in the shallow level of the network, and the semantic feature information of the fruit stem is extracted in the deep layer, Wherein, the two modules of Embedding and Merging are used to convert the dimension and size of the feature map, and the FasterBlock module is constructed based on the partial convolution operator, which fuses the feature information between the cross channels.
4. The method of claim 2, wherein: The DBB-PANet feature fusion network introduces the DBB module to reconstruct the C3 module and form the C3-DBB feature fusion module. The C3-DBB module uses the multi-scale idea to capture rich feature information in the training stage. Four strategies of 1x1conv-bn, 1x1conv-bn+AVG-bn, 1x1conv-bn+3x3conv-bn and 3x3conv-bn are used to extract feature information, and four different multi-branches are combined into a single branch structure.
5. A method of grading apples according to claim 4, characterized in that: For the combination of feature information extracted by 1x1conv-bn+AVG-bn and 1x1conv-bn+3x3conv-bn strategies, two inputs are The output is obtained as follows: ； wherein denotes a convolution operation, denotes a padding operation on the bn layer, Due to denotes a 1x1 convolution, the parameters are merged into the convolutional kernel using a linear recombination strategy, as shown in the equation, ； wherein represents a dimension conversion operation; Constructing partial bias terms As shown in the equation: ； All bias terms are combined, as shown in the formula: ； The AVG operation is equivalent to a convolution with the same size and step length, as shown in the formula: ； Finally, it is combined into a new convolution operation in the branch, as shown in the formula: 。 6. The method of claim 1, wherein: In the step S2, the channel pruning based on the LAMP score is adopted, the L2 norm of all weights of each input channel is calculated and sorted, the index mapping is established according to the newly generated sequence, the channel with smaller weights is removed, so that the compression of the model is realized, and the LAMP score is defined as formula: ； wherein, denotes the target channel, denotes the sum of the square of the target channel weight, denotes the sum of the square of the weight of all remaining channels except the target channel which are not cropped and have an index greater than the target channel index.
7. The method of claim 1, wherein: In the step S3, based on the segmented apple image, the RGB color space is converted into the HSV color space, and the apple red coloration ratio is calculated The formula is: ； wherein represents the number of apple red pixels, represents the number of all pixels of the apple; For apple fruit shape index The specific calculation process is as follows: ； wherein represents a longitudinal diameter, represents a transverse diameter.
8. The method of claim 1, wherein: In the step S4, the GBDT algorithm is used for the input apple feature data set: ， wherein represents the color characteristic, represents the diameter characteristic, represents the shape characteristic, represents the grade to which the apple belongs.