Innovative apple grading evaluation method

Abstract

The invention discloses an innovative apple grading evaluation method, and relates to the technical field of apple grading evaluation, and the method specifically comprises the following steps: S1, providing a web crawler and manual collection mode to collect a high-quality image for an apple stem detection task; s2, proposing an FDNet lightweight fruit stem detection algorithm for fruit stem detection and fruit body auxiliary positioning; s3, for apple feature extraction, two algorithm structures of MBFilter and DPC-AKNN are provided; and S4, aiming at the apple grading model, adopting a GBDT algorithm. According to the invention, a high-precision lightweight fruit stem detection algorithm FDNet-p is provided by combining a computer vision technology with a current mainstream model compression technology, and superior performance of the algorithm is proved through a large number of experiments. In the apple grading process, the FDNet-p algorithm is preliminarily used for screening out apples without fruit stems, the idea of auxiliary positioning is put forward, camera collection operation is prevented from being triggered in advance or lagging, then the image processing technology is used for extracting the characteristics such as the apple color, the apple shape and the apple diameter, and finally the GBDT algorithm is used for completing the intelligent grading model design of the apples. And intelligent grading of apples is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of apple grading and evaluation, and in particular to an innovative apple grading and evaluation method. Background Art

[0002] The problems caused by the manual sorting of apples, such as fruit collision, fruit surface dents, low grading efficiency, human factors, and huge manpower consumption, have attracted more and more attention. Automatic apple grading technology has gradually been widely used in the apple industry and is a vital link in the apple industry and market circulation. In this context, feature extraction of apple appearance, non-destructive testing of peels, and design and optimization of fruit grade classification models have become research hotspots.

[0003] With the rapid development of deep learning, the application of nondestructive testing and intelligent sorting technology in the apple grading process is becoming more and more extensive. Fan et al. combined color and fruit diameter features to detect apple tree fruits and realize size grading. They can effectively detect small objects and unevenly illuminated fruits, and the grading accuracy rate reaches 90%. Shi et al. proposed a new multi-view spatial network to solve the apple grading task. The model uses apple size information as one of the grading criteria. First, a well-pre-trained lightweight CNN is used to extract low-level features of apples. Secondly, a spatial feature aggregation module is constructed by bidirectional LSTM and mean pooling to mine relevant information of apples from multiple angles. The multi-view spatial network has a good application prospect in the apple grading task. Fan et al. proposed a real-time detection method for apple defects based on the YOLOV4 deep learning algorithm. The input image is synthesized from three consecutive infrared images. The YOLOV4 network is simplified by using channel pruning and layer pruning methods to speed up the detection speed. At the same time, a non-maximum suppression (NMS) method based on the L1 norm is proposed to remove redundant prediction boxes after fine-tuning the pruned network to improve the detection speed of the model. Summary of the invention

[0004] In view of the problems raised in the above background technology, based on the above research foundation of automatic classification and grading of apples, the present invention combines advanced computer vision technology, image segmentation and machine learning technology to construct an apple grading system, and integrates efficient, accurate and secondary damage-avoiding apple appearance quality grading technology with reference to apple grading standards;

[0005] The present invention extracts four important indicators, namely, fruit stalk characteristics, color characteristics, fruit diameter characteristics and fruit shape characteristics, and combines the local grading standards of Red Fuji apples to revise a set of streamlined apple grading standards, in which fruits of three different grades are required to carry fruit stalks, and the two characteristics of apple diameter and fruit shape are further refined.

[0006] In addition, the priorities of the four features in the apple grading task of the present invention are fruit stalk, color, fruit diameter, and fruit shape. If the relevant features do not meet the requirements of the corresponding level, they will be directly classified as the next level.

[0007] Therefore, in order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solution:

[0008] An innovative apple grading and evaluation method comprises the following steps:

[0009] S1: For the apple stalk detection task, we proposed web crawlers and manual collection methods to collect high-quality images, and built a dataset for the stalk detection task. We used the X-AnyLabeling2.3.0 annotation tool to manually annotate the stalk data and build a stalk detection dataset.

[0010] S2: For fruit stem detection and fruit body auxiliary positioning, the FDNet lightweight fruit stem detection algorithm is proposed. The efficient FasterNet is combined with the SPPF module as the feature extraction network, and a multi-scale structural re-parameterized feature fusion network is adopted to enhance the model perception ability. The model compression strategy based on the LAMP score is adopted to achieve an extremely lightweight detection algorithm.

[0011] S3: For apple feature extraction, the MBFilter algorithm structure is proposed to ensure the clarity of apple details. At the same time, the pre-processing algorithm is used to accurately segment the fruit body and the background. The DPC-AKNN algorithm is used for apple segmentation. AdaptiveKNN is used to adaptively calculate local density and dynamically process apple images of different densities and shapes.

[0012] S4. For the apple grading model, the image processing method is used to extract the three important features of apple color ratio, fruit diameter and fruit shape, and the grade is calibrated and an apple grading dataset is constructed. At the same time, the GBDT algorithm is used to realize an efficient apple grading model.

[0013] Preferably, the step S1 specifically includes an apple grading pipeline device, a FasterNet feature extraction network and a DBB-PANet feature fusion network; the apple grading pipeline device includes a transmission control system, an image acquisition terminal, fruit stalk detection, fruit body image preprocessing, multi-feature extraction, grading model evaluation and automatic sorting, so as to achieve efficient grading of apples.

[0014] Preferably, the FasterNet feature extraction network is represented by FasterBackbone, which is composed of three modules: Embedding, Merging and FasterBlock. The visual feature information of the fruit stalk is extracted at the shallow level of the network, and the semantic feature information of the fruit stalk is extracted at the deep level.

[0015] Among them, the Embedding and Merging modules are used to convert the dimension and size of the feature map. The FasterBlock module is constructed based on the partial convolution operator to fuse the feature information across channels.

[0016] Preferably, the DBB-PANet feature fusion network introduces the DBB module to reconstruct the C3 module and constitute a C3-DBB feature fusion module. The C3-DBB module uses a multi-scale idea to capture rich feature information during the training phase. The specific scheme is to extract feature information using four strategies: 1x1conv-bn, 1x1conv-bn+AVG-bn, 1x1conv-bn+3x3conv-bn and 3x3conv-bn, and merge four different multi-branches into a single-branch structure.

[0017] Preferably, for the 1x1conv-bn+AVG-bn and 1x1conv-bn+3x3conv-bn strategies, the two inputs are The output formula is:

[0018] O'=(Inpute F (1) +REP(b (1) ))e F (2) +REP(b (2) )

[0019] Where e represents the convolution operation and REP(·) represents the padding operation on the bn layer.

[0020] Because F (1) Represents a 1x1 convolution, using a linear recombination strategy to merge parameters into F (2) In the convolution kernel, as shown in the formula,

[0021]

[0022] Where T represents the dimension conversion operation;

[0023] Constructing partial bias terms As shown in the formula:

[0024]

[0025] Combine all bias terms, as shown in the formula:

[0026]

[0027] The AVG operation of branch (2) can be equivalent to a convolution with the same size and stride. The specific operation is shown in the formula:

[0028]

[0029] Finally, they are merged into a new convolution operation within the branch, as shown in the formula:

[0030] Concat(·)=Inpute F'+REP(b').

[0031] Preferably, in step S2, channel pruning based on LAMP score is adopted, the L2 norm of all weights of each input channel is calculated and sorted, an index mapping is established according to the newly generated sequence, and weights and smaller channels are removed, so as to achieve model compression. The LAMP score is defined as follows:

[0032]

[0033] Where u represents the target channel, (C W (u)) 2 represents the sum of squares of target channel weights, ∑ u≤v (C W (v)) 2 Represents the sum of the squared weights of all remaining channels except the target channel that have not been clipped and whose index is greater than the target channel index.

[0034] Preferably, in step S3, the calculation formula of the MBFilter algorithm is:

[0035] I med (p) = med({I(q)|q∈N(p)})

[0036]

[0037] Among them I med , G med Respectively represent median filtering and bilateral filtering, f s 、f r Represents the Gaussian function of the spatial domain and the color domain, W p is the normalization factor, and N(p) represents the neighborhood of pixel p.

[0038] Preferably, in step S3, based on the segmented apple image, the RGB color space is converted into the HSV color space, and the formula for calculating the apple red coloring ratio ratio is:

[0039]

[0040] Among them, pixel red Indicates the number of red pixels in the apple, pixel all Represents the number of all pixels of the apple; for the apple shape index E, the specific calculation process is as follows:

[0041]

[0042] Wherein h represents the longitudinal diameter and w represents the transverse diameter.

[0043] Preferably, in step S4, the GBDT algorithm for the input apple feature data set is:

[0044] D={((x c ,x d ,x s ) i ,(y g ) i )|i=1,2,L,N},

[0045] where x c represents the coloring ratio feature, x d represents the fruit diameter characteristics, x s Indicates the fruit shape characteristics, y g Indicates the grade of the apple.

[0046] Compared with the prior art, one or more of the above technical solutions have the following beneficial effects:

[0047] 1. For the first time, the fruit stem indicator was introduced into the apple grading task. A high-precision lightweight fruit stem detection algorithm FDNet-p was proposed by combining computer vision technology with the current mainstream model compression technology, and its superior performance was proved through a large number of experiments. In the apple grading process, the FDNet-p algorithm was initially used to screen out apples without fruit stems, and the idea of ​​auxiliary positioning was proposed to prevent the camera acquisition operation from being triggered in advance or delayed.

[0048] 2. In the process of apple feature extraction, an improved DPC-AKNN segmentation algorithm is proposed, and an adaptive brightness balance strategy is introduced to improve the effect of fruit segmentation. Then, image processing technology is used to extract the three features of apple color, fruit diameter, and fruit shape, which are mapped with the real manual grading scene;

[0049] 3. Based on the multi-feature information of apples, a variety of performance indicators are used to fully evaluate the performance of the machine learning classification algorithm, and finally the GBDT algorithm is selected to realize intelligent grading of apples. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0051] Figure 1 It is a schematic diagram of the overall structure of apple grading of the present invention;

[0052] Figure 2It is a schematic diagram of the structure of the FDNet network structure and auxiliary positioning of the present invention;

[0053] Figure 3 is a schematic structural diagram of the FasterBlock module structure of the present invention;

[0054] Figure 4 It is a schematic diagram of the structure of the C3-DBB module of the present invention;

[0055] Figure 5 It is a structural schematic diagram of the apple feature extraction solution of the present invention;

[0056] Figure 6 This is a comparison chart of the effects of different noise removal algorithms for apples of the present invention;

[0057] Figure 7 It is the apple brightness equalization algorithm processing process of the present invention;

[0058] Figure 8 This is a segmentation effect diagram of the DPC-FKNN algorithm of the present invention;

[0059] Fig. 9 It is a schematic diagram of apple color feature extraction of the present invention;

[0060] Fig.10 It is a schematic diagram of apple diameter feature extraction of the present invention;

[0061] Fig.11 It is a schematic diagram of apple shape feature extraction of the present invention;

[0062] Fig.12 It is a comparison chart of different evaluation indicators of FDNet and YOLOv5-s of the present invention;

[0063] Fig.13 It is a performance comparison chart of the lightweight detection model of the present invention;

[0064] Fig.14 It is a comparison effect diagram of different apple detection algorithms of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0066] Specific reference Figure 1-Figure 11The present invention proposes an innovative apple grading and evaluation method, comprising the following steps:

[0067] (1) Aiming at the task of apple stalk detection, a web crawler and manual collection method are proposed to collect high-quality images, and a dataset for the stalk detection task is constructed. The stalk data are manually labeled using the X-AnyLabeling2.3.0 annotation tool to construct a stalk detection dataset, which can enhance the diversity and anti-interference of the dataset and further improve the robustness and generalization ability of the model.

[0068] (2) Aiming at fruit stalk detection and fruit body auxiliary positioning, a lightweight fruit stalk detection algorithm called FDNet is proposed. The efficient FasterNet is combined with the SPPF module as the feature extraction network, and a multi-scale structural re-parameterized feature fusion network is adopted to enhance the model perception ability. A model compression strategy based on the LAMP score is adopted to achieve an extremely lightweight detection algorithm, which can further improve the efficiency of apple grading in the pipeline and reduce unnecessary resource consumption.

[0069] (3) For apple feature extraction, the MBFilter algorithm structure is proposed to ensure the clarity of apple details. At the same time, a preprocessing algorithm is used to accurately segment the fruit body and the background. The DPC-AKNN algorithm is used for apple segmentation, and the AdaptiveKNN algorithm is used to adaptively calculate the local density and dynamically process apple images of different densities and shapes.

[0070] (4) For the apple grading model, the image processing method is used to extract the three important features of apples, namely color ratio, fruit diameter and fruit shape, to perform grade calibration and construct an apple grading dataset. At the same time, the GBDT algorithm is used to realize an efficient apple grading model.

[0071] The details are as follows:

[0072] 1. Available fruit stem detection datasets:

[0073] There is currently a lack of publicly available datasets for apple stalk detection. When introducing a complex apple dataset that contains multiple and small stalk images in different scenes, the model has poor anti-interference performance. The image rotation (15°, 45°, 60°) technique is used to randomly enhance some datasets of various apple images with stalk targets in complex backgrounds, making it impossible for the model to fully learn the different postures of apples in the grading pipeline.

[0074] Table 1: Customized apple grading standards

[0075]

[0076] 2. Fruit stem detection and fruit body auxiliary positioning:

[0077] In order to screen out apples with missing stalks, a lightweight apple stalk detection algorithm needs to be designed to prevent them from entering the subsequent grading operation, further improve the efficiency of apple grading in the pipeline, and reduce unnecessary resource consumption. Another task of stalk detection is to participate in fruit body positioning, that is, to prevent the front and back acquisition cameras from being triggered in advance or delayed, resulting in the inability to accurately capture the complete front and back body view. The FDNet lightweight stalk detection algorithm is used, and the specific method is as follows:

[0078] 1) The apple grading line device is divided into seven parts: transmission control system, image acquisition end, fruit stem detection, fruit body image preprocessing, multi-feature extraction, grading model evaluation and automatic sorting, which can achieve efficient grading of apples;

[0079] The conveying control system includes a standard conveyor belt, a drive motor, etc. The drive motor can dynamically adjust the running speed of the assembly line according to demand; the image acquisition end includes a high-performance computer, an image acquisition device, a high-definition camera, etc. Three high-definition wide-angle distortion-free industrial cameras are placed in the image acquisition device. When shooting images, the table tennis ball-assisted calibration method is first used to determine the shooting distance between the camera and the apple by calculating the error between the diameter of the table tennis ball in the acquired image and the actual diameter; when it is detected that the apple reaches the field of view of the top camera, the camera in the acquisition area of ​​the apple's overhead view will detect whether the apple has a stalk. Apples without stalks will not trigger the acquisition actions of the front view and the back view, and will not enter the subsequent apple grading process, thereby reducing resource consumption to a certain extent.

[0080] At the same time, the fruit stem is used for auxiliary positioning. When the fruit stem detection frame completely enters the set area, the apple with the fruit stem will trigger the computer to control the other two cameras to shoot, which are the front view and the back view of the apple respectively; the collection of the front view and the back view is based on the maximum cross-sectional area. The apple can be approximated as a sphere. The high-definition camera uses the isometric view collection method, so that the front and back two collection angles are the entire side image of the apple. Then use image processing operations to extract the key feature information of the apple, and then perform model comparison, training and evaluation, select the model with the best performance to determine the apple level, and finally use a robotic arm to sort the apples to the designated level area.

[0081] 2) The FasterNet feature extraction network (expressed as FasterBackbone) is used, which consists of three modules: Embedding, Merging, and FasterBlock. The visual feature information of the fruit stalk is extracted at the shallow level of the network, and the semantic feature information of the fruit stalk is extracted at the deep level. Among them, the Embedding and Merging modules are used to convert the dimension and size of the feature map. The FasterBlock module is constructed based on the partial convolution operator to fuse the feature information across channels.

[0082] Another task of fruit stem detection is to participate in fruit body positioning, that is, to prevent the front and rear acquisition cameras from being triggered early or late, resulting in the inability to accurately capture the complete front and rear body view. During image acquisition, the fruit stem detection positioning frame is combined with the shooting trigger operation. When the fruit stem completely appears at the set position within the specified area, the shot is taken to ensure that completely clear multi-view apple images are captured.

[0083] 3) The DBB module is introduced into the DBB-PANet feature fusion network to reconstruct the C3 module and form the C3-DBB feature fusion module. The C3-DBB module uses the multi-scale idea to capture rich feature information during the training phase. The specific schemes are 1x1conv-bn, 1x1conv-bn+AVG-bn,

[0084] The four strategies of 1x1conv-bn+3x3conv-bn and 3x3conv-bn extract feature information and merge four different multi-branch structures into a single branch structure.

[0085] For the 1x1conv-bn+AVG-bn and 1x1conv-bn+3x3conv-bn strategies, the two inputs are The output formula is:

[0086] O'=(Inpute F (1) +REP(b (1) ))e F (2) +REP(b (2) )

[0087] Where e represents the convolution operation and REP(·) represents the padding operation on the bn layer.

[0088] Because F (1) Represents a 1x1 convolution, using a linear recombination strategy to merge parameters into F (2) In the convolution kernel, as shown in the formula,

[0089]

[0090] Where T represents the dimension conversion operation;

[0091] Constructing partial bias terms As shown in the formula:

[0092]

[0093] Combine all bias terms, as shown in the formula:

[0094]

[0095] The AVG operation of branch (2) can be equivalent to a convolution with the same size and stride. The specific operation is shown in the formula:

[0096]

[0097] Finally, they are merged into a new convolution operation within the branch, as shown in the formula:

[0098] Concat(·)=Inpute F'+REP(b').

[0099] 3. Extraction of apple features:

[0100] In the image acquisition device, the top view captured by the top camera is used to detect the fruit stem using the FDNet-p algorithm. When the apple is in the best shooting area, the remaining two cameras are triggered to shoot the front and back views of the apple, and then the apple images from the three perspectives are batch processed using the image processing method. In the apple feature extraction stage, the influence of foreground image noise is first suppressed, and then brightness equalization is performed to segment the outline of the apple, and finally the three important features of color, fruit diameter, and fruit shape are extracted.

[0101] 1) When capturing apple images, the high-definition camera ensures the clarity of the apple details, but it is also easily affected by tiny noises in the natural environment, which is not conducive to the extraction of apple features in the subsequent process. In view of the above problems, the present invention proposes a noise processing method (MBFilter) that mixes median filtering and bilateral filtering. The MBFilter algorithm rearranges each pixel and its neighborhood in the image, selects the pixel median for replacement, and combines the Gaussian weighted filtering method in the spatial domain and color domain. The calculation formula is as follows:

[0102] I med (p) = med({I(q)|q∈N(p)})

[0103]

[0104] Among them I med , G med Respectively represent median filtering and bilateral filtering, f s 、f r Represents the Gaussian function of the spatial domain and the color domain, W p is the normalization factor, and N(p) represents the neighborhood of pixel p.

[0105] MBFilter first uses median filtering to suppress noise to the greatest extent, and then uses bilateral filtering to adaptively smooth the noise area in the image to further process tiny noise, while ensuring that the outline of the apple is clear and important details are not lost. The specific effect is as follows: Figure 6As shown in the figure, three different intensities of salt and pepper noise were simulated in the experiment. It can be seen from the figure that after the median filter processing, a small amount of tiny noise still remains on the surface of the apple. The experimental MBFilter algorithm has a better effect in processing tiny noise, and the image detail information is better preserved.

[0106] 2) Background information has a great influence on the results of apple feature extraction, so it is necessary to accurately segment the fruit body from the background. In order to ensure that the apple segmentation is more accurate, the present invention designs a set of preprocessing algorithms for the task scenario of the apple grading pipeline to deal with the problem of uneven brightness on the apple surface. First, the image in the RGB space is converted to the LAB space, the L component representing the brightness is separated, and the brightness is processed using adaptive histogram equalization. Then, the processed L component, A component, and B component are merged, and finally the image in the LAB space is converted to the RGB space again. The specific effect is as follows: Figure 7 As shown, the surface brightness of the processed image tends to be uniform, which partially solves the problem of uneven illumination on the apple surface after being captured by the camera.

[0107] 3) In the experiment, the pre-processed apples are segmented using a clustering algorithm. The DPC algorithm is a density-based clustering algorithm that can automatically determine the cluster center and the number of clusters. However, the density calculation method of the local density of the sample points is based on the cutoff distance, which fails to take into account the distribution of samples in the neighborhood. The value of the cutoff distance has a great influence on the apple segmentation result. In view of the above problems, the present invention proposes an improved DPC clustering algorithm (DPC-AKNN) for apple segmentation, which uses AdaptiveKNN to adaptively calculate the local density, does not rely on the global distance threshold, and automatically adjusts the local density of the point, more effectively identifies the density peak, so that the DPC algorithm dynamically processes apple images of different densities and shapes, and reduces the sensitivity of the DPC algorithm to the cutoff distance parameter. In the apple image segmentation task, DPC-AKNN first calculates the Euclidean distance from each pixel to other points, then uses the AKNN algorithm to calculate the local density and obtain the minimum distance, and finally obtains the cluster center of the cluster, and assigns the class labels to the remaining pixels to achieve the segmentation of apples and backgrounds.

[0108] In order to prove the effectiveness of the DPC-KNN algorithm proposed in the experiment in the apple segmentation task, the segmentation effect of apples at different resolutions was verified, the clustering, density decision diagram and cluster distribution diagram were drawn, and the segmentation effect was compared with the current mainstream KMeans clustering algorithm. The specific visualization results are shown in the figure. Figure 8As shown in the decision diagram and cluster diagram, it can be seen that at different resolutions, DPC-AKNN can accurately find the optimal decision boundary; in the segmentation effect comparison part, the KMeans clustering algorithm failed to accurately segment the part with weaker light, and the DPC-KNN algorithm accurately segmented the complete fruit body, with better segmentation effect for weak light edges. The algorithm has strong anti-interference ability and can meet the needs of fruit body segmentation tasks in the apple grading pipeline.

[0109] 4) The color feature of apples is a key indicator in the grading task. Based on the segmented apple image, the present invention converts the RGB color space into the HSV color space, extracts the red area according to the threshold range that is close to the red color of apples in the HSV color space, visualizes the specific extraction process, and calculates the red coloring ratio of the apple. The specific calculation process is as follows:

[0110]

[0111] Among them, pixel red Indicates the number of red pixels in the apple, pixel all Represents the number of all pixels in the apple.

[0112] 5) Extract fruit diameter features based on the front view. First, calculate the minimum circumscribed rectangle. In order to improve the accuracy of fruit diameter measurement, introduce the minimum circumscribed circle to assist measurement. The results measured by the above two methods are arithmetic averaged to finally obtain the pixel diameter of the apple image. Take a ping-pong ball (actual diameter is 40 mm) as a reference, and indirectly calculate the actual fruit diameter through the diameter of the ping-pong ball. Specifically, the minimum circumscribed rectangle and the minimum circumscribed circle weighted method are used to extract fruit diameter features.

[0113] 6) Apple is a kind of spherical fruit. The similarity with the standard circle is used as the grading index, namely the fruit shape index. Canny is an algorithm suitable for edge detection tasks. It uses a Gaussian filter to convolve the image to suppress the influence of noise, and then uses the Sobel operator to calculate the horizontal and vertical gradients of the image. It checks the pixels along the gradient direction, suppresses the pixels of non-maximum values, and only retains the local maximum value. It further distinguishes between strong edges and edges, connects strong edges and weak edges, and forms continuous edges. This paper calculates the apple fruit shape index based on the Canny algorithm and the minimum enclosing rectangle. This paper converts the segmented RGB image into a grayscale image, detects the edge of the fruit body through the Canny algorithm, and visualizes the operation process as follows. Fig.11 As shown. The edge image of the apple is used to calculate the vertical and horizontal diameters of the apple using the minimum enclosing rectangle, and then the apple shape index E is calculated. The specific calculation process is as follows:

[0114]

[0115] Wherein h represents the longitudinal diameter and w represents the transverse diameter.

[0116] 4. In the study of apple grading model:

[0117] The shooting rate of the camera is 30 frames per second, which ensures the smoothness of the apple image acquisition action; it is also equipped with an IR-CUT function to correct the color cast problem and improve the brightness in low-light environments. When the IR-CUT dual filter switcher is in sufficient light, the infrared cutoff filter starts to work, and the CCD restores the true color of the apple. When the light sensor senses insufficient light, the automatic LED infrared fill light (the light source is invisible to the naked eye) automatically moves away, and the infrared filter starts to work, so that the camera can make full use of the LED infrared light, greatly improving the image acquisition performance in low-light environments, and ensuring the clarity and accuracy of the collected apple images. The industrial camera takes three perspective photos of each apple and transmits them to the local computer, and then the computer saves the information as an image file. The present invention uses 126 apples as the research object of the grading model, and uses image processing methods to extract the corresponding color features, fruit diameter features and fruit shape features. Finally, it is divided into corresponding grade fruits according to the apple grading standard, and divided into training sets and test sets according to the ratio of 7:3.

[0118] The three important features of apple color ratio, fruit diameter and fruit shape are extracted by image processing methods, and the grade is calibrated to construct an apple grading data set. The gradient boosting tree (GBDT) algorithm is used to implement an efficient apple grading model. GBDT is an ensemble learning algorithm that uses CART tree as a weak learner. Each time a new learner is constructed, the residual of the sample in the current learner is used as the label of the next learner for training. By fitting the negative gradient of the current model and minimizing the loss function, the GBDT model is constructed after multiple rounds of iterative training, which has good generalization. The GBDT algorithm has the following apple feature data set as the input:

[0119] D={((x c ,x d ,x s ) i ,(y g ) i )|i=1,2,L,N},

[0120] where x c represents the coloring ratio feature, x d represents the fruit diameter characteristics, x s Indicates the fruit shape characteristics, y g Indicates the grade of the apple. Since apple grading is a classification problem, the log-likelihood loss function is selected to evaluate the model.

[0121] For the above methods, please refer to Figure 12-14 The present invention provides the above experimental results and analysis:

[0122] 1. Experimental environment and parameter settings

[0123] The experiment of the present invention is based on the operating system with version number CentOS7.9.2009, the processor model is 12vCPUIntel(R)Xeon(R)Platinum8255CCPU@2.50GHz, considering the different requirements of video memory for models of different sizes, the high-performance graphics card model NVIDIAGeForceRTX3090 is selected, the video memory size is 24G, the integrated development environment is Miniconda3, the development language is Python (version number is 3.8.0), and Pytorch (version number is 1 The fruit stem detection model was built using two mainstream deep learning frameworks, MMDetection (version 11.0) and MMDetection (version 3.2.0). The training process of the CUDA accelerated model with version 11.3 was used. The input image resolution was 640*640. The Adam method combining momentum optimization and adaptive learning rate strategy was used to optimize the model training process. The initial learning rate was 1e-3, and the image batch size passed to the network training each time was 64. The baseline model and the improved model were trained for 200 rounds according to the above parameter settings, and mosaic data enhancement was turned off in the last 10 rounds. The apple grading model was built using the scikit-learn machine learning library.

[0124] 2. Evaluation indicators of fruit stem detection model

[0125] The present invention uses common evaluation indicators of detection tasks to evaluate the performance of the fruit stem detection model, namely, recall rate R (Recall), precision rate P (Precision), mAP (meanAveragePrecision), AP (AveragePrecision), model computing power (GFLOPS) and FPS, the specific calculation formula is as follows:

[0126]

[0127] Where n refers to the number of categories. In the experiment, there is only one category, fruit stalks. mAP refers to the average precision calculated on the fruit stalk category. P refers to the proportion of all instances detected as fruit stalks that are actually fruit stalks, which measures the degree of false detection of the model. R refers to the proportion of all instances that are actually fruit stalks that are correctly detected as fruit stalks, which measures the degree of missed detection of the model. TP refers to the number of fruit stalks correctly predicted by the model, FP refers to the number of background content mistakenly detected as fruit stalks by the model, and FN refers to the actual number of fruit stalks that the model failed to detect.

[0128] 3. Experimental analysis of fruit stalk detection;

[0129] 3.1 Comparative Experiments on Different Feature Extraction Networks

[0130] To prove the effectiveness of the FasterNet-t0 feature extraction network, a comparative experiment with different feature extraction networks was set up, mainly including mainstream lightweight networks (MobileNetv3 and GhostNet), and three different sizes of FasterNet-t in the FasterNet network series. To ensure the fairness of the experimental comparison, the feature fusion network is PANet of YOLOv5s. The specific comparative experimental results of different feature extraction networks in the fruit stem detection task are shown in Table 2.

[0131] Table 2: Comparative experiments of different feature extraction networks

[0132]

[0133] According to the experimental results in Table 2, for the fruit stem detection task, the FasterNet-t0 network has the best comprehensive performance. The model is reduced by 2.8MB, GFLOPS is reduced by 4.6, and the accuracy is almost unchanged. This shows that the PConv operator reduces the redundant calculation of the model to a certain extent and reduces the complexity of the model.

[0134] Although MobileNetv3 and GhostNet networks have reduced the number of network parameters and computational complexity to a large extent, the mAP@0.5:0.95 value has been severely lost, making it difficult to meet the high-precision requirements of actual fruit stem tasks. In addition, the accuracy values ​​of FasterNet-t1 and FasterNet-t2 have improved to varying degrees, but the model computational complexity and size are relatively large. After comprehensively comparing different models, FasterNet-t0 is the best choice as a feature extraction network.

[0135] 3.2 Improved process ablation experiment

[0136] This paper sets up a series of ablation experiments to prove the improvement effect of the proposed improvement strategy on the fruit stem detection task. Based on the YOLOv5s detection network, it is specifically divided into: using FasterNet-t0 to improve the feature extraction network, proposing DBB-PANet to improve the feature fusion network, and combining the two methods for improvement. The specific ablation experimental results of different improvement strategies in the fruit stem detection task are shown in Table 3.

[0137] Table 3: Ablation experiments of the improved process

[0138]

[0139] According to the ablation experiment results in Table 3, it can be seen that after using FasterNet-t0 as the feature extraction network, the computational load GFLOPS of the model is reduced by 4.6, the model size is reduced by 2.8, and the model is lightweight, which shows that the PConv operator in FasterNet effectively reduces the frequency of memory access, and the accuracy is almost the same as the baseline model; the P value of the feature fusion network model improved by DBB-PANet is increased by 0.3%, the R value is increased by 2.3%, and the accuracy is slightly increased, indicating that DBB-PANet has an improved effect on the performance of the fruit stalk detection task, and the multi-scale structural re-parameterization strategy is used to effectively enhance the recognition ability of fruit stalks of different sizes; the two strategies are combined to improve the YOLOv5-s model. Compared with the baseline model, the P value is increased by 2.7%, the mAP@0.5 value is increased by 0.8%, and the computational load GFLOPS of the model is reduced by 1.8. DBB-PANet is used to fully integrate the feature information extracted by the FasterNet-t0 network to achieve an increase in detection accuracy while reducing the computational load of the model.

[0140] Fig.12 Comparison of different evaluation indicators between FDNet and YOLOv5-s. (a) is the bboxloss training process curve. (b) is the mAP@0.5 training process curve. (c) is the obj_loss training process curve. (d) is the PRCurves of YOLOv5-s on the validation set. (e) is the PRCurves of FDNet on the validation set.

[0141] In order to more intuitively show that the proposed FDNet is better than the baseline model YOLOv5-s, the bbox_loss, obj_loss, and mAP@0.5 of the training process are visualized and compared. Fig.12 As shown, from Fig.12 It can be seen from (a) and (c) in the figure that the two different loss curves during FDNet training are better than YOLOV5-s, indicating that the model converges better and the mAP@0.5 value is also higher than YOLOv5-s. The specific effects are as follows: Fig.12 (b) As shown. At the same time, the Precision-RecallCurve graph is used to further analyze the detection accuracy of FDNet and YOLOv5-s on the validation set. Fig.12 (d) shows the detection results of FDNet. Fig.12 (e) shows the detection results of YOLOv5-s, where the mAP@0.5 value of FDNet is 0.969, which is higher than the mAP@0.5 value of YOLOv5-s.

[0142] 3.3 Model Compression Comparison Experiment

[0143] In order to fully demonstrate the impact of pruning strategies on model performance, the present invention selects different model compression strategies to carry out sufficient comparative experiments, namely Slim, L1, and Lamp, three mainstream pruning algorithms. At the same time, the two key indicators of inference time and FPS are evaluated in graphics card devices with low resources. The specific comparative experiments of different pruning strategies are shown in Table 4.

[0144] Table 4: Model compression comparison experiment

[0145]

[0146] Note: Latency and FPS values ​​are tested under the 1080Ti graphics card environment.

[0147] According to the model compression comparison experiment in Table 4, when the compression ratio is set to 2.0, the Lamp pruning algorithm has the best comprehensive effect in the comparison of multiple evaluation indicators. Compared with the Slim and L1 pruning algorithms, the Lamp algorithm has the smallest model size after pruning, the mAP@0.5:0.95 value decreases the least, and the inference speed is the fastest on low-resource devices. Its Latency value is 0.268±0.032 and the FPS value is 363.0, which fully demonstrates the effectiveness of the Lamp pruning algorithm. For the adaptive structured pruning at the channel level, the original convolution structure of the pruned model is still retained, which is more friendly to hardware resources. In addition, in order to explore the optimal strategy for model compression, comparative tests with different compression ratios were added based on the Lamp pruning algorithm. It was found that when the compression ratio was 4.0, the compression effect of the model was the best, with an mAP@0.5 value of 96.6, a GFLOPs value of 3.4, a model size of 2.5, a Latency of 0.200±0.113, and a FPS of 499.5. In order to more intuitively visualize the effect of pruning, the changes in the number of channels in each layer of the model before and after Lamp pruning (compression ratio of 4.0) were plotted.

[0148] 3.4 Model comparison experiment

[0149] A series of comparative experiments are set up to prove the superiority of the FDNet proposed in the present invention in the fruit stem detection task, and a full comparison and verification is carried out with the cutting-edge mainstream target detection model on multi-dimensional evaluation indicators. The FDNet proposed in the present invention is a lightweight single-stage target detection algorithm. Therefore, the YOLO series of lightweight single-stage detection algorithms are mainly selected for comparison, mainly including YOLOv3, YOLOv5, YOLOv6, YOLOv8, YOLOv9, YOLOv10, and YOLOv11. In addition, in order to fully verify the performance of FDNet, the current mainstream non-YOLO series target detection algorithms are also added, mainly including FasterRCNN, GFL, RTMDet, YOLOx and TOOD. The specific comparative experimental results are shown in Table 5.

[0150] Table 5: Comparative experiments of different detection models

[0151]

[0152] According to the quantitative analysis results of different detection models in Table 5, it can be seen that the FDNet (uncompressed) fruit stem detection model proposed in the experiment has better detection performance than other detection models, and the mAP@0.5 value reaches the highest, with a specific value of 96.9%. Among the YOLO series target detection algorithms, YOLOv6s has the best detection performance, but its parameter volume and computational complexity are large, and the requirements for device resources are high. Compared with it, the FDNet algorithm has a mAP@0.5 value that is 0.09 higher, and the mAP@0.5:0.95 value is almost the same, but the GFLOPS and model size are much smaller than YOLOv6s; compared with other models with similar computational complexity, the FDNet model has higher detection accuracy. Among non-YOLO series target detection algorithms, the FDNet model is at the leading level in terms of detection accuracy, computational complexity, and model size. After the model is processed using the compression strategy, the P value of the FDNet-p model is 96.3%, the R value is 89.4%, the mAP@0.5 value is 96.6%, the mAP@0.5:0.95 value is 67.9%, the GFLOPS value is only 3.4, and the model size is only 2.5MB. Compared with the various indicators of FDNet, FDNet-p achieves lossless compression of the model, which fully demonstrates the efficiency of the Lamp pruning algorithm. Among many target detection models, the FDNet-p model has the smallest resource consumption. After using LAMP pruning, the computational complexity of the model is greatly reduced, the GFLOPS value is the smallest, only 3.4, and the accuracy of FDNet-prune remains at a high level.

[0153] In order to more intuitively show the excellent performance of the compressed FDNet model (FDNet-p), the lightweight detection models of the YOLO series (v5, v6, v8, v9, v10 and v11) are selected for comparative analysis. The specific visualization effects are as follows: Fig.13 In the fruit stem detection task, compared with the other lightweight models, the proposed FDNet-p requires the least GFLOPS, has the smallest model size, and is easy to deploy on devices with insufficient resources in the pipeline, while maintaining the highest mAP@0.5 value, meeting the accuracy requirements of the fruit stem detection task.

[0154] The detection effect diagram set in the real scene further demonstrates the strong robustness and high detection accuracy of the FDNet-p proposed in this paper. In order to ensure the fairness of the detection effect, the lightweight detection model of the YOLO series (v5, v6, v8, v9, v10 and v11) is still used for comparative analysis of the detection effect. The specific detection results are compared as follows: Fig.14 As shown. Four different scenarios were selected: 1. Multiple targets + complex background (Sample1), FDNet-p had the highest detection accuracy in three different apples, while other models had lower detection accuracy; 2. Single target + complex background (Sample2), the YOLOv9-n model mistakenly identified the background as a fruit stalk; 3. Simple background + calyx interference (Sample3), other models had different degrees of misdetection, identifying the calyx as a fruit stalk, while FDNet-p accurately identified the fruit stalk. 4. Top view (Sample4), it can be seen that FDNet-p has the best detection effect.

[0155] 4. Apple grading experiment analysis

[0156] 4.1 Evaluation indicators of apple grading model

[0157] In the apple grading model research, the performance of the apple grading model is comprehensively analyzed by setting a variety of evaluation indicators, including P (Precision), R (Recall), JaccardScore and F1Score. F1Score and JaccardScore are used to measure the overall performance and accuracy of the model in each category respectively; at the same time, in order to comprehensively analyze the performance of the model and better understand the performance of the model in multi-category problems, three different averaging methods are introduced: micro, macro and weighted. Micro averaging (micro) evaluates the global performance and reflects the proportional influence of each sample; macro averaging (macro) emphasizes the importance of each category; and weighted averaging (weighted) focuses on the problem of category imbalance; these indicators reflect the sensitivity, correctness and applicability of the model from different angles. The specific calculation formulas are shown in Table 6.

[0158] Table 6: Evaluation indicators of the classification model

[0159]

[0160] Note: C represents the number of categories, N represents the total number of samples, and ni represents the number of samples in the i-th category.

[0161] 4.2 Comparative Experiment on Apple Grading Algorithms

[0162] The grading algorithm of the present invention is based on three characteristics: color, fruit diameter and fruit shape. By setting up a series of comparative experiments of mainstream machine learning algorithms, multiple evaluation indicators are used to evaluate the performance of different algorithms in apple grading algorithms. In order to determine the best apple grading model, a total of 6 apple grading models are established, and the algorithm performance is evaluated using a 10-fold cross-validation algorithm to effectively reduce the deviation in the model evaluation process. The ensemble learning algorithm model is RF (RandomForest), BC

[0163] (BaggingClassification) and GBDT (GradientBoosting), and the non-ensemble learning algorithms are KNN (KNearestNeighbors), SVM, and MLP. The specific experimental results of different apple grading algorithms are shown in Table 7.

[0164] Table 7: Comparison of 10-fold cross validation experimental results of different apple grading algorithms

[0165]

[0166] According to the cross-validation experimental results of the grading algorithms in Table 7, it can be seen that the overall performance of the non-ensemble learning algorithm is weaker than that of the ensemble learning algorithm. The GBDT apple grading algorithm has the best comprehensive performance in various evaluation indicators among the ensemble learning algorithms. Specifically, compared with other ensemble learning algorithms, the F1Score values ​​of the three different average calculation methods of the GBDT algorithm are all optimal, which are 0.9513, 0.9476 and 0.9506 respectively. At the same time, it also achieves the highest values ​​of the three evaluation indicators of JaccardScore, P and R, reflecting the excellent classification ability of the GBDT algorithm, with strong robustness and generalization ability, and is suitable as an apple grading model.

[0167] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An innovative apple grading and evaluation method, characterized in that: The following steps are involved: S1: For the apple stalk detection task, we proposed web crawlers and manual collection methods to collect high-quality images, and built a dataset for the stalk detection task. We used the X-AnyLabeling2.3.0 annotation tool to manually annotate the stalk data and build a stalk detection dataset. S2: For fruit stem detection and fruit body auxiliary positioning, the FDNet lightweight fruit stem detection algorithm is proposed. The efficient FasterNet is combined with the SPPF module as the feature extraction network, and a multi-scale structural re-parameterized feature fusion network is adopted to enhance the model perception ability. The model compression strategy based on the LAMP score is adopted to achieve an extremely lightweight detection algorithm. S3: For apple feature extraction, the MBFilter algorithm structure is proposed to ensure the clarity of apple details. At the same time, the pre-processing algorithm is used to accurately segment the fruit body and the background. The DPC-AKNN algorithm is used for apple segmentation. AdaptiveKNN is used to adaptively calculate local density and dynamically process apple images of different densities and shapes. S4. For the apple grading model, the image processing method is used to extract the three important features of apple color ratio, fruit diameter and fruit shape, and the grade is calibrated and an apple grading dataset is constructed. At the same time, the GBDT algorithm is used to realize an efficient apple grading model.

2. The innovative apple grading and evaluation method according to claim 1, characterized in that: The step S2 specifically includes an apple grading pipeline device, a FasterNet feature extraction network and a DBB-PANet feature fusion network; the apple grading pipeline device includes a transmission control system, an image acquisition terminal, fruit stem detection, fruit body image preprocessing, multi-feature extraction, grading model evaluation and automatic sorting to achieve efficient grading of apples.

3. The innovative apple grading and evaluation method according to claim 2, characterized in that: The FasterNet feature extraction network is represented by FasterBackbone, which consists of three modules: Embedding, Merging and FasterBlock. It extracts the visual feature information of the fruit stalk at the shallow level of the network and the semantic feature information of the fruit stalk at the deep level. Among them, the Embedding and Merging modules are used to convert the dimension and size of the feature map. The FasterBlock module is constructed based on the partial convolution operator to fuse the feature information across channels.

4. The innovative apple grading and evaluation method according to claim 2, characterized in that: The DBB-PANet feature fusion network introduces the DBB module to reconstruct the C3 module and constitutes the C3-DBB feature fusion module. The C3-DBB module uses the multi-scale idea to capture rich feature information during the training stage. The specific scheme is to extract feature information using four strategies: 1x1conv-bn, 1x1conv-bn+AVG-bn, 1x1conv-bn+3x3conv-bn and 3x3conv-bn, and merge four different multi-branches into a single-branch structure.

5. The innovative apple grading and evaluation method according to claim 4, characterized in that: For the 1x1conv-bn+AVG-bn and 1x1conv-bn+3x3conv-bn strategies, the two inputs are The output formula is: O'=(Input F (1) +REP(b (1) ))e F (2) +REP(b (2) ) Where e represents the convolution operation and REP(·) represents the padding operation on the bn layer. Because F (1) Represents a 1x1 convolution, using a linear recombination strategy to merge parameters into F (2) In the convolution kernel, as shown in the formula, Where T represents the dimension conversion operation; Constructing partial bias terms As shown in the formula: Combine all bias terms, as shown in the formula: The AVG operation of branch (2) can be equivalent to a convolution with the same size and stride. The specific operation is shown in the formula: Finally, they are merged into a new convolution operation within the branch, as shown in the formula: Concat(·)=Inpute F'+REP(b').

6. The innovative apple grading and evaluation method according to claim 1, characterized in that: In step S2, channel pruning based on LAMP score is adopted to calculate the L2 norm of all weights of each input channel and sort them, establish index mapping according to the newly generated sequence, remove weights and smaller channels, so as to achieve model compression. The LAMP score is defined as follows: Where u represents the target channel, (C W (u)) 2 represents the sum of squares of target channel weights, ∑ u≤v (C W (v)) 2 Represents the sum of the squared weights of all remaining channels except the target channel that have not been clipped and whose index is greater than the target channel index.

7. The innovative apple grading and evaluation method according to claim 1, characterized in that: In step S3, the calculation formula of the MBFilter algorithm is: I med (p)=med({I(q)|q∈N(p)}) Among them I med , G med Respectively represent median filtering and bilateral filtering, f s 、f r Represents the Gaussian function of the spatial domain and the color domain, W p is the normalization factor, and N(p) represents the neighborhood of pixel p.

8. The innovative apple grading and evaluation method according to claim 1, characterized in that: In step S3, based on the segmented apple image, the RGB color space is converted into the HSV color space, and the formula for calculating the apple red coloring ratio ratio is: Among them, pixel red Indicates the number of red pixels in the apple, pixel all Represents the number of all pixels in the apple; For the apple shape index E, the specific calculation process is as follows: Where h represents the longitudinal diameter and w represents the transverse diameter.

9. The innovative apple grading and evaluation method according to claim 1, characterized in that: In step S4, the GBDT algorithm for the input apple feature data set is: D={((x c ,x d ,x s ) i ,(y g ) i )|i=1,2,L,N}, where x c represents the coloring ratio feature, x d represents the fruit diameter characteristics, x s Indicates the fruit shape characteristics, y g Indicates the grade of the apple.

Images