Small sample data-oriented fruit external quality image classification method

Abstract

The invention discloses a small sample data-oriented fruit external quality image classification method, and relates to the technical field of image classification. According to the method, a fruit external quality classification model is constructed and is mainly divided into three layers: a fruit classifier, a fruit defect recognizer and a fruit quality classifier. Firstly, the fruit classifier improves a Faster R-CNN model, a multi-scale subnet is added in a feature extraction part to adapt to fruit feature extraction of different scales, and the accuracy of fruit classification is improved; secondly, the main function of the fruit recognizer is to detect the classification layer to output the defect category of the fruit, and in order to better fit the marginalization or centralization of the feature of the defect, the pooling strategy of the CNN model is changed, the pooling mode is selected and changed in a targeted manner, and the defect classification effect is better realized; and finally, a fruit classifier is established through fruit feature dependence maximization, and accurate classification of fruits under small sample data is realized.

Description

Technical Field

[0001] This invention relates to the field of fruit quality image classification technology, and to a method for classifying fruit external quality images for small sample data. Background Technology

[0002] During the fruit production process, human and natural factors can influence the quality of the fruit, resulting in significant differences in yield. Factors such as shape, size, and color can all affect the quality of the fruit. Therefore, it is necessary to grade the fruit according to its quality before it is sold in the market to determine the selling price.

[0003] Traditional fruit quality inspection relies primarily on manual sorting, depending on subjective experience and the fruit's appearance. With technological advancements, machines are increasingly replacing manual labor in fruit sorting, significantly improving efficiency and reducing costs. Machine vision technology, for example, classifies fruit based on features such as color, shape, and texture. However, its drawback is its heavy reliance on external features. Inaccurate feature data leads to inaccurate quality assessments. For instance, Chinese patent application CN201911349365.4 discloses a method for adjusting the pose of spherical fruits, using machine vision to identify different fruit poses in real time and employing servo-controlled adjustment mechanisms. However, this method overlooks fruit diversity and fails to adequately detect the external features of different fruits, resulting in poor generalization ability of the fruit quality inspection model and inaccurate classification.

[0004] With the rise of artificial intelligence, many researchers are combining AI technology with agriculture. Because the diversity of fruits aligns with the training characteristics of neural networks relying on large datasets, many neural network models, such as AlexNet, CNN, ResNet, and YOLO, have been used in applications like fruit classification, ripeness detection, and quality classification. While these neural networks can solve the aforementioned tasks well, they don't perform well in more complex scenarios. The models mentioned above require a necessary objective condition: a sufficient number of training samples. (See existing technology, "Application of Deep Learning in Fruit Quality Detection and Grading," by Tian Youwen, 2021). Collecting massive amounts of training samples means significant labor and time costs, prompting researchers to consider how to achieve accurate detection while reducing costs. This led to the proposal of few-shot learning, a method that uses a small number of samples for classification and regression, thus addressing the cost issue. Furthermore, few-shot learning allows the model to learn independently and exhibits good generalization ability across different tasks.

[0005] To address the problems of inaccurate classification and high training costs in existing technologies, this paper proposes a fruit external quality image classification method for small sample data. Summary of the Invention

[0006] The purpose of this invention is to provide a fruit external quality image classification method for small sample data, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for classifying fruit external quality images for small sample data includes the following steps:

[0009] S10. Construct image datasets for grading various fruits;

[0010] S11. Remove non-fruit images from the Fruit-Dataset dataset and select some fruit images that are free of defects and those that are defective;

[0011] The dataset Fruit-Dataset contains 262 types of fruit, most of which was obtained by web scraping and contains some incorrect images.

[0012] S12. Preprocess the fruit image to obtain a fruit grading image;

[0013] The preprocessing includes random cropping, random rotation, color transformation, and blending; the fruit image size is uniformly scaled to 512x512 for easy input into the model.

[0014] S13. Manually label the category, defect category, and quality level of the fruit grading images;

[0015] The defect categories include: black spots, rot, rust spots, and scratches; the quality levels include: Level 1, Level 2, Level 3, and Level 4.

[0016] Furthermore, to train the model, the dataset needs to be partitioned, specifically including:

[0017] S20. The fruit grading image dataset is divided into a training set, a support set, and a test set;

[0018] The training set is divided into a base class set and a query set, which includes C categories; the support set includes K categories, where K≤C, and each category contains at least M fruit images, M≥1, where M is a natural number; the intersection of the categories in the training set and the categories in the support set is 0; the categories in the test set are a subset of the categories in the support set, and the test set is unlabeled.

[0019] Furthermore, in order to grade fruits based on their external quality, a relevant fruit external quality classification model is established. The specific process includes:

[0020] S30. Establish a classification model for the external quality of fruits;

[0021] The fruit external quality classification model includes a fruit classifier, a fruit defect identifier, and a fruit quality grader.

[0022] S31. The fruit classifier uses the Fruit Faster R-CNN model, and the specific steps include:

[0023] The Faster R-CNN model is a model for object detection, where R represents the region;

[0024] The Fruit Faster R-CNN model improves upon the ResNet50 model to classify fruits in images.

[0025] S311. Input the fruit images from the training set into the Ads-ResNet50 network to obtain the first feature map of the fruit;

[0026] In the AdS-ResNet50 network, AdS stands for Adaptive Scale, which serves as the feature extraction module for the fruit classifier. It adapts to the feature extraction of fruits of different sizes by adding a multi-scale sub-network structure. The multi-scale sub-network structure consists of a bottom-up module, a top-down module, and a horizontal module.

[0027] The bottom-up module consists of multiple downsampling layers and convolutional layers. It extracts the features of the fruit grading image and inputs them into the bottom-up module. The features of each convolutional layer are then connected to the convolutional features of each layer in the top-up module via a 1x1 convolutional kernel using residual connections. The output features of the top-up module are input into the horizontal module and, after passing through convolutional layers and connection layers, the first feature map of the fruit is obtained.

[0028] S312. The candidate region RPN network and the target region pooling ROIPooling network are used to share the first feature map of the fruit;

[0029] Wherein, the RPN network is a region selection network used to generate candidate boxes for feature regions; and ROIPooling is interest pooling used to predict feature map extraction.

[0030] S313. The RPN network distinguishes the background and target fruit of the fruit image through a softmax classifier and initially obtains the candidate sampling region location;

[0031] The candidate region locations are obtained through Bbox regression.

[0032] S314. The ROIPooling network processes the first feature map of the fruit to obtain the second feature map of the fruit;

[0033] The second feature map of the fruit is obtained by pooling and normalizing the first feature map of the fruit.

[0034] S315. Classify the target fruit according to the second feature image of the target fruit to determine the category of the target fruit;

[0035] The target fruit category integrates the second feature map information of the fruit through a fully connected layer, and is then classified by a softmax classifier.

[0036] S316. Output the category and region information of the target fruit.

[0037] Furthermore, by inputting the output of the fruit classifier into the fruit defect classification, the specific steps include:

[0038] S32. The fruit defect detector is used to detect surface defects in the target fruit;

[0039] S321. Obtain the defective area of ​​the target fruit;

[0040] S322. Segment the image of the defective region to obtain a defective image of the target fruit;

[0041] The defective image of the target fruit is segmented using a fuzzy color clustering method.

[0042] S323. Use the fruit defect identifier to classify the target fruit defect image;

[0043] The fruit defect detector uses a CNN model and employs different pooling methods for different feature pooling regions. Different pooling functions are selected based on the pooling value and the variance of the feature map. The calculation formula is as follows:

[0044]

[0045] Wherein, S is the feature value obtained after pooling the defect image of the target fruit. Minimum pooling is used to extract features from the defective image of the target fruit, where n is the pooling step size and σ is the minimum pooling step size. P σ is the value in the pooling region. FM Let t be the variance on the feature map of the defective image of the target fruit. min t is the minimum value in pooling. max The maximum value in pooling, tave The average of the maximum and minimum values ​​in pooling;

[0046] To avoid the feature pooling region of the target fruit defect image being too small and the maximum feature not being obvious, a correction factor is added. This correction factor is the ratio of the sum of the differences between the feature values ​​and the pooling region to the sum of the feature values ​​of the pooling region. The calculation formula is as follows:

[0047] μ=(t sum -t max +t min ) / t sum

[0048] S324. Output the defect category of the target fruit.

[0049] Furthermore, the output of the fruit defect identifier is used as an influencing factor for fruit quality grading, and the fruit quality grader is constructed. Specific steps include:

[0050] S33. The target fruit is graded using the fruit quality grader;

[0051] S331. Obtain the target fruit and the defect category of the target fruit;

[0052] The target fruit defect category is obtained by the fruit defect identifier detecting surface defects in the target fruit.

[0053] S332. Calculate the degree of impact of the defects of the target fruit on the surface of the target fruit, using the following formula:

[0054]

[0055] Among them, TFd a TF represents the defect area of ​​the target fruit defect image. a Let be the area of ​​the target fruit, and α be the parameter representing the influence of the defects of the target fruit on the target fruit;

[0056] The influencing parameter refers to the relationship between the size of the defects on the fruit surface and the result of fruit quality grading.

[0057] The area of ​​the target fruit is calculated based on the boundary regression box of the RPN network, using the following formula:

[0058]

[0059]

[0060] TF a =t w *t h

[0061] Among them, t x t y t w and t h These represent the x-coordinate, y-coordinate, height, and width of the predicted target fruit located at the center of the fruit image;

[0062] S333. Design the fruit quality grader and classify the fruit using the second feature map of the fruit.

[0063] The fruit quality grading system is based on maximizing fruit feature dependence features to build a classifier, and references fruit defect features to improve classification accuracy. The objective equation for fruit grading is:

[0064]

[0065] Wherein, the first term of the objective equation is the experience loss generated by the target fruit during the grading process, the second term is the regularization of the grading process by comprehensively utilizing the target fruit, and the parameters α and β represent the degree of influence of the defects of the target fruit on the appearance of the fruit and the defect type parameter of the target fruit, respectively.

[0066] The first term of the objective equation is defined as:

[0067] Ω1(f,x,y)=||P T xy|| 2

[0068] Where, f(x) = p T x is the predicted grade of the target fruit, p is the characteristic of the target fruit, and y is the grade label of the target fruit;

[0069] The second term of the objective equation is defined as:

[0070]

[0071] Where, ||f(x) i )-f(x j )|| 2 The similarity in grade distribution between the target fruit and other fruits of the same type is denoted by N, where N is the number of fruits of the same type.

[0072] The fruit quality classifier is a classifier built based on the maximum fruit features. In addition to considering the original fruit features, such as color, texture, and size, it considers not only the degree of impact of defects on the fruit surface but also the impact of defect type on fruit quality.

[0073] Furthermore, the fruit external quality classification model is trained based on the training set, with the loss function being:

[0074] L = L fc +L fd +L gc

[0075] Among them, L fc Let L be the loss function of the fruit classifier. fd Let L be the loss function of the fruit defect detector. gc Let the loss function be the fruit quality classifier.

[0076] The loss function of the fruit classifier mainly includes fruit classification loss and bounding box prediction loss, and the calculation formula is as follows:

[0077]

[0078] in, For fruit classification, the cross-entropy loss function is p. i To predict fruit categories, For real fruit categories, Let t be the loss function for predicting fruit boxes. i For the fruit prediction box location information, λ represents the actual position information of the fruit frame, and λ is the balance parameter.

[0079] The fruit defect identifier loss function uses the cross-entropy loss function to calculate the loss for the four defect categories, as shown in the formula:

[0080]

[0081] Where N is the number of fruits in the training set, c is the category of defect, and y ic For fruit defect category labels, use 0 or 1 to represent; p ic This represents the predicted probability of fruit defect categories.

[0082] The fruit quality classifier uses a 0-1 loss function, calculated as follows:

[0083]

[0084] Where f(x) is the fruit quality classification and prediction function.

[0085] S40. Using the trained fruit external quality classification model, classify the quality of the fruit images in the test set;

[0086] The test set and the support set are input into the trained fruit external quality classification model to obtain the grading results;

[0087] S50. Output the quality grading results of the fruit.

[0088] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0089] 1. A fruit classifier is established as the first layer module of the fruit appearance quality classification model. The fruit classifier uses Faster R-CNN as its base model and introduces an Ads-ResNet50 network to improve fruit classification performance, providing accurate predictions based on the characteristics of specific fruits during final quality grading. The Ads-ResNet50 network mainly adds a multi-scale sub-network structure, which can adapt to the features of fruits of different sizes during feature extraction, improving the model's fitting ability.

[0090] 2. The second layer of the fruit appearance quality classification model is the fruit defect detector. This detector's main function is to detect the type of fruit defect and its impact on the fruit. The fruit defect detector uses a fruit CNN model for defect classification. Considering the diverse morphologies of fruit surface defects in reality, traditional CNN models cannot learn the main features. Therefore, the fruit CNN model uses different pooling methods based on the defect pooling regions at different locations to improve the accuracy of defect classification, providing an important reference factor for subsequent fruit quality grading.

[0091] 3. A fruit quality classifier is used for classifying fruit quality levels. It primarily establishes a classifier function by maximizing the dependence of fruit features. This is achieved by introducing factors such as the area and type of fruit defects while classifying grades based on basic features. This fruit quality classifier not only conforms to real-world fruit grading standards but also overcomes the shortcomings of existing models that do not adequately consider all factors in fruit quality classification, thus improving the accuracy of fruit quality grading. Attached Figure Description

[0092] Figure 1 A flowchart of a fruit external quality image classification method for small sample data provided in an embodiment of the present invention;

[0093] Figure 2 This is an example of fruit grading provided in an embodiment of the present invention;

[0094] Figure 3 This is a structural diagram of the Fruit Faster R-CNN model provided in an embodiment of the present invention;

[0095] Figure 4 This is an Ads-ResNet50 network diagram provided in an embodiment of the present invention;

[0096] Figure 5 The predicted fitted fruit image provided in the embodiment of the present invention;

[0097] Figure 6This is a comparison diagram of experiments using a fruit classifier provided in an embodiment of the present invention;

[0098] Figure 7 This is a comparative experimental diagram of the fruit quality grading device provided in an embodiment of the present invention; Detailed Implementation

[0099] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0100] Please see Figures 1 to 7 A method for classifying fruit external quality images for small sample data, the technical solution is as follows: Specifically, refer to... Figure 1 As shown in the flowchart, this invention provides a method for classifying fruit external quality images for small sample data, which includes steps S10 to S50 in a specific implementation.

[0101] Specifically, S10. Obtain a dataset of multiple fruit images; S20. Construct the dataset; S30. Establish a fruit external quality classification model; S40. Use the trained fruit external quality classification model to classify the quality of the fruit images in the test set; S50. Output the fruit quality classification results.

[0102] S10. Construct image datasets for grading various fruits;

[0103] S11. Take the non-fruit images from the Fruit-Dataset dataset and select some fruit images that are free of defects and those that are defective;

[0104] S12. Preprocess the fruit image to obtain a fruit grading image;

[0105] The preprocessing includes: random cropping, random rotation, color change, and blending;

[0106] Image fusion is an algorithm used in computer vision to enhance images by combining images from different classes, thereby expanding the training dataset. Its core principle is to linearly combine two different images in a certain proportion to generate new samples, and the labels of these new samples are also obtained through linear combination.

[0107] The fruit images are uniformly scaled to 512x512 for easy input into the model.

[0108] S13. Manually label the category, defect category, and quality level of the fruit grading images;

[0109] The defect categories include: black spots, rot, rust spots, and scratches; the quality levels include: Level 1, Level 2, Level 3, and Level 4.

[0110] S20. Construct the dataset;

[0111] Furthermore, in this embodiment, to demonstrate the beneficialness of the proposed method, the fruit grading image dataset is divided into training, support, and test sets by selecting 8000 images of fruits with and without defects, in a 5:3:2 ratio. Figure 2 As shown, it includes common fruit categories such as strawberries, apples, oranges, and pears.

[0112] The training set and the support set have no overlap in the types of fruits, and the test set contains a subset of the types of fruits in the support set.

[0113] S30. Establish a classification model for the external quality of fruits;

[0114] The fruit external quality classification model includes a fruit classifier, a fruit defect identifier, and a fruit quality grader.

[0115] S31. The fruit classifier uses the Fruit Faster R-CNN model, and the specific steps include:

[0116] The fruit Faster R-CNN model improves upon the ResNet50 model to classify fruits in images.

[0117] S311. Input the fruit images from the training set into the Ads-ResNet50 network to obtain the first feature map of the fruit;

[0118] The first feature map of the fruit includes fruit color features, fruit shape features, fruit texture features, and fruit size features.

[0119] Furthermore, the Ads-ResNet50 network, used for fruit image feature extraction, comprises a bottom-up module, a top-down module, and a horizontal module. The bottom-up module is constructed using ResNet50 and includes multiple convolutional modules, each with numerous convolutional layers. When extracting fruit features upwards, the feature map dimension of each layer is reduced by half, and the output of each convolutional module is used by the top-down module. The top-down module uses the output of each convolutional module for a 2x upsampling operation. Then, the feature maps from the two paths are merged by element-wise addition. Finally, each layer uses a 3x3 convolutional filter to generate the first feature map of the fruit.

[0120] Furthermore, the first feature map of the fruit is input into the candidate region network.

[0121] S312. The candidate region RPN network and the target region pooling ROIPooling network are used to share the first feature map of the fruit;

[0122] S313. The RPN network distinguishes the background and target fruit of the fruit image through a softmax classifier and initially obtains the candidate sampling region location;

[0123] The candidate region locations are obtained through Bbox regression.

[0124] S314. The ROIPooling network processes the first feature map of the fruit to obtain the second feature map of the fruit;

[0125] The second feature map of the fruit is obtained by pooling and normalizing the first feature map of the fruit.

[0126] S315. Classify the target fruit according to the second feature image of the target fruit to determine the category of the target fruit;

[0127] The target fruit category integrates the second feature map information of the fruit through a fully connected layer, and is then classified by a softmax classifier.

[0128] S316. Output the category and region information of the target fruit.

[0129] In this embodiment, the Faster R-CNN model is used as the fruit classifier in this step. Figure 3The accuracy of classification is improved by modifying the backbone network. The main function of the backbone network, AdS-ResNet50, is to extract features from fruit images. Considering the inconsistent sizes of different fruits, even after scaling the images to a uniform size before training, it is still impossible to extract effective features of the target fruits in the images. Therefore, a multi-scale subnetwork structure is introduced into the original ResNet50 network. This can effectively capture feature information of target fruits of different sizes in a uniform-scale background. Figure 4 The AdS-ResNet50 network architecture is shown in the image. To illustrate the feasibility of the solution, [the image is shown in the image]. Figure 4 The experiment compares the Faster R-CNN model and the Fruit Faster R-CNN model on different numbers of fruit images.

[0130] Figure 5 The results demonstrate that the Fruit Faster R-CNN model exhibits superior performance across different stages. As the number of training fruits increases, the accuracy of the Fruit Faster R-CNN model also increases. Comparing it to the Faster R-CNN model, it can be seen that the Accu-Score (Acc) of the Fruit Faster R-CNN model is at least 20% higher than that of Faster R-CNN at the same stage. Furthermore, to illustrate the good generalization ability of the AdS-ResNet network in extracting features from fruits of different sizes, in... Figure 6 The image demonstrates the use of an AdS-ResNet network to extract fruit features and outputs the fit of the predicted bounding boxes. Figure 6 It can be seen that the AdS-ResNnet network has a good fruit feature extraction function. Figure 6 The fitting results for three different fruits—grapes, apricots, and apples—are presented. The fitting result for grapes, which have a smaller scale, reaches 82.3%, while the fitting result for apples, which have a larger scale, reaches 95.6%. This demonstrates that AdS-ResNet50 has excellent generalization performance.

[0131] In this embodiment, four common fruit defect categories are selected, and a fruit defect identifier is established to classify the surface defects of the fruit, providing a reference for subsequent fruit quality grading.

[0132] Furthermore, by inputting the output of the fruit classifier into the fruit defect classification, the specific steps include:

[0133] S32. The fruit defect detector is used to detect surface defects in the target fruit;

[0134] S321. Obtain the defective area of ​​the target fruit;

[0135] S322. Segment the image of the defective region to obtain a defective image of the target fruit;

[0136] The defective image of the target fruit is segmented using a fuzzy color clustering method.

[0137] The fuzzy color clustering method refers to converting the RGB color model into the HSL color model; defining fuzzy sets H, S, and L using triangular membership functions to form a fuzzy color set; and defining a fuzzy color similarity measure based on the fuzzy colors to analyze the similarity between two fuzzy colors.

[0138] S323. Use the fruit defect identifier to classify the target fruit defect image;

[0139] The fruit defect detector uses a CNN model and employs different pooling methods for different feature pooling regions. Different pooling functions are selected based on the pooling value and the variance of the feature map. The calculation formula is as follows:

[0140]

[0141] Wherein, S is the feature value obtained after pooling the defect image of the target fruit. Minimum pooling is used to extract features from the defective image of the target fruit, where n is the pooling step size and σ is the minimum pooling step size. P σ is the value in the pooling region. FM Let t be the variance on the feature map of the defective image of the target fruit. min t is the minimum value in pooling. max The maximum value in pooling, t ave The average of the maximum and minimum values ​​in pooling;

[0142] Furthermore, CNN is a widely used deep learning model in computer vision, known for its advantage in extracting local features. These advantages include: local awareness: Based on convolutional operations, CNN extracts local features from the input image through sliding convolutional kernels. This local awareness captures local patterns and structures in the image, effectively preserving spatial information. Compared to fully connected neural networks, CNN better maintains positional relevance during feature extraction, resulting in better performance for image processing tasks. Parameter sharing: CNN's convolutional layers use parameter sharing, meaning the same weight parameters are used for feature extraction across all locations in the input image. This mechanism reduces the number of parameters, effectively decreasing model complexity and computational overhead. Parameter sharing also helps extract local invariance in the image, enhancing the robustness of CNN to translation, rotation, and scale changes. Multi-level feature representation: Through the stacking of multiple convolutional and pooling layers, CNN can gradually extract more abstract and high-level feature representations. Lower convolutional layers capture low-level features such as edges and textures, while higher convolutional layers learn more abstract semantic features.

[0143] Furthermore, to avoid the feature pooling region of the target fruit defect image features being too small and the maximum value feature not being obvious, a correction factor is added. This correction factor is the ratio of the sum of the differences between the feature values ​​and the pooling region to the sum of the feature values ​​of the pooling region, and the calculation formula is as follows:

[0144] μ=(t sum -t max +t min ) / t sum

[0145] S324. Output the defect category of the target fruit.

[0146] Furthermore, to illustrate the benefits of the optimized pooling layer in the fruit CNN model, a comparison was made with the original CNN model using average pooling. Table 1 shows the classification results of fruit defects by the fruit CNN model and the original CNN model. The table demonstrates that changing the pooling method effectively adapts to the features of different defect types.

[0147] Table 1. Comparison of Acc (%) between CNN model and Fruit CNN for different defect types

[0148]

[0149] Table 1 shows experiments conducted on common fruit defect types. To illustrate that the fruit CNN model, which uses a specific pooling method based on a particular pooling region, is more suitable for classifying fruit defects than the original CNN model using average pooling, experiments were designed to compare the differences in the accuracy (Acc) evaluation index between the models. The table shows that the fruit CNN model achieves over 90% accuracy across all four defect types. Specifically, for fruits of different sizes, the accurate classification by the upper-level fruit classifier provides accurate data for defect detection, improving the precision of defect detection. Therefore, it can be concluded that targeted selection of pooling methods is beneficial for improving classification accuracy.

[0150] This step in this embodiment serves as the third layer of the fruit appearance quality grading model: the fruit quality grader. The objective function of the fruit grader is established by maximizing fruit feature dependencies. To adapt to the scenario of fruit appearance quality grading, fruit defect factors are added to the basic feature-based grading scenario. The model denotes the impact of defects on the overall fruit as α and the fruit defect type factor as β as the influence of these factors on the fruit grading results. Figure 7 The model results are presented for two defect factors, a single defect factor, and no defect factors.

[0151] exist Figure 7 The paper presents the experimental results of four models: Model 1, Model 2, Model 3, and Model 4. Model 1 uses basic fruit characteristics for fruit grading; Model 2 considers the impact of defects on the fruit surface (α) as a factor influencing fruit grading; Model 3 considers the defect type (β) as a factor; and Model 4 considers both α and β as factors. In real-world scenarios, when purchasing fruit, customers make decisions based on the defects on the fruit's surface and their size, either refusing to buy defective fruit or requesting a price reduction. Therefore, considering this factor is necessary in model design. Based on the experimental results, using a fruit grading image dataset to evaluate the accuracy (Acc) and recall (Rec) of the four models, the fourth model demonstrates superior performance, outperforming Model 1 and the model based on a single defect factor.

[0152] Furthermore, the output of the fruit defect identifier is used as an influencing factor for fruit quality grading, and the fruit quality grader is constructed. Specific steps include:

[0153] S33. The target fruit is graded using the fruit quality grader;

[0154] S331. Obtain the target fruit and the defect category of the target fruit;

[0155] The target fruit defect category is obtained by the fruit defect identifier detecting surface defects in the target fruit.

[0156] S332. Calculate the degree of impact of the defects of the target fruit on the surface of the target fruit, using the following formula:

[0157]

[0158] Among them, TFd a TF represents the defect area of ​​the target fruit defect image. a Let be the area of ​​the target fruit, and α be the parameter representing the influence of the defects of the target fruit on the target fruit;

[0159] The influencing parameter refers to the relationship between the size of the defects on the fruit surface and the result of fruit quality grading.

[0160] The area of ​​the target fruit is calculated based on the boundary regression box of the RPN network, using the following formula:

[0161]

[0162]

[0163] TF a =t w *t h

[0164] Among them, t x t y t w and t h These represent the x-coordinate, y-coordinate, height, and width of the predicted target fruit located at the center of the fruit image;

[0165] S333. Design the fruit quality grader and classify the fruit using the second feature map of the fruit.

[0166] The fruit quality grading system is based on maximizing fruit feature dependence features to build a classifier, and references fruit defect features to improve classification accuracy. The objective equation for fruit grading is:

[0167] In the fruit objective equation, the first term establishes a classification function using the basic fruit features extracted by the fruit classifier, and the calculation formula is as follows:

[0168] Ω1(f,x,y)=||P T xy|| 2

[0169] Where, f(x) = p Tx is the predicted grade of the target fruit, p is the characteristic of the target fruit, and y is the grade label of the target fruit;

[0170] The second term of the objective equation is defined as:

[0171]

[0172] Where, ||f(x) i )-f(x j )|| 2 The similarity in grade distribution between the target fruit and other fruits of the same type is denoted by N, where N is the number of fruits of the same type.

[0173] The fruit quality classifier is a classifier built based on the maximum fruit features. In addition to considering the original fruit features, such as color, texture, and size, it considers not only the degree of impact of defects on the fruit surface but also the impact of defect type on fruit quality.

[0174] Furthermore, the fruit external quality classification model is trained based on the training set, with the loss function being:

[0175] L = L fc +L fd +L gc

[0176] Among them, L fc Let L be the loss function of the fruit classifier. fd Let L be the loss function of the fruit defect detector. gc Let the loss function be the fruit quality classifier.

[0177] The loss function of the fruit classifier mainly includes fruit classification loss and bounding box prediction loss, and the calculation formula is as follows:

[0178]

[0179] in, For fruit classification, the cross-entropy loss function is p. i To predict fruit categories, For real fruit categories, Let t be the loss function for predicting fruit boxes. i For the fruit prediction box location information, λ represents the actual position information of the fruit frame, and λ is the balance parameter.

[0180] The fruit defect identifier loss function uses the cross-entropy loss function to calculate the loss for the four defect categories, as shown in the formula:

[0181]

[0182] Where N is the number of fruits in the training set, c is the category of defect, and y ic For fruit defect category labels, use 0 or 1 to represent; p ic This represents the predicted probability of fruit defect categories.

[0183] The fruit quality grader uses a 0-1 loss function, calculated as follows:

[0184]

[0185] Where f(X) is the fruit quality classification and prediction function.

[0186] S40. Using the trained fruit external quality classification model, classify the quality of the fruit images in the test set;

[0187] The test set and the support set are input into the trained fruit external quality classification model to obtain the grading results;

[0188] S50. Output the quality grading results of the fruit.

[0189] In summary, this embodiment uses the Fruit-Dataset dataset to train the model. To meet the needs of few-shot learning scenarios, the dataset is divided into training, support, and test sets. The entire fruit external quality classification model consists of three layers: a fruit classifier, a fruit defect detector, and a fruit quality grader. The fruit classifier uses the Faster R-CNN model to classify fruits in images and can distinguish between the image background and the target fruit. To adapt to the classification of fruits at different scales, an Ads-ResNet50 network is proposed. This model incorporates a multi-scale subnetwork structure, which can better capture feature information at different scales and improve classification accuracy. Secondly, the fruit defect detector is used to detect defects in the target fruit and segments the defective image using existing techniques. To improve the accuracy of fruit defect classification, a CNN model is used to better extract local features. Furthermore, to address the marginalization or centralization of defect features, the convolution strategy of the CNN model is improved, selecting appropriate convolution methods for different distributions of defect features. The effectiveness of the improvement is illustrated in Table 1 through experiments. Finally, the fruit quality grader is the last layer of the fruit external quality classification model. Based on maximizing fruit feature dependencies, a classification model is established, taking fruit defect factors as an important consideration. Quality is graded based on the defects of different fruits, enabling the model to better adapt to the detection of different fruits.

[0190] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0191] One or more embodiments of this specification are intended to cover all such alternative steel surface defect datasets falling within the broad scope of the appended claims; such datasets, modifications, and variations. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of this disclosure.

Claims

1. A method for classifying fruit external quality images for small sample data, characterized in that, include: S10. Construct image datasets for grading various fruits; S20. The fruit grading image dataset is divided into a training set, a support set, and a test set; S30. Establish a classification model for the external quality of fruits; The fruit external quality classification model includes a fruit classifier, a fruit defect identifier, and a fruit quality grader. S31. The fruit classifier uses the Fruit Faster R-CNN model, and the specific steps include: S311. Input the fruit grading images from the training set into the Ads-ResNet50 network to obtain the first feature map of the fruit; The Ads-ResNet50 network serves as the feature extraction module of the fruit classifier, adapting to feature extraction of fruits of different sizes by adding a multi-scale sub-network structure. S312. The candidate region RPN network and the target region pooling ROIPooling network are used to share the first feature map of the fruit; S313. The RPN network distinguishes the background and target fruit of the fruit grading image through a softmax classifier operation, and initially obtains the candidate sampling region location; S314. The ROIPooling network processes the first feature map of the fruit to obtain the second feature map of the fruit; S315. Classify the target fruit according to the second feature map of the fruit to determine the category of the target fruit; S316. Output the category and region information of the target fruit; S32. The fruit defect detector is used to detect surface defects in the target fruit; S33. The target fruit is graded using the fruit quality grader; S40. Using the trained fruit external quality classification model, classify the quality of the fruit images in the test set; S50. Output the fruit quality grading results.

2. The method according to claim 1, characterized in that, S10 includes: S11. Remove non-fruit images from the Fruit-Dataset dataset and select some fruit images that are free of defects and those that are defective; S12. Preprocess the fruit image to obtain a fruit grading image; S13. Manually label the category, defect category, and quality level of the fruit grading images; The defect categories include: black spots, rot, fruit rust spots, and scratches; the quality levels include: Level 1, Level 2, Level 3, and Level 4.

3. The method according to claim 1, characterized in that, S20 includes: The training set is divided into a base class set and a query set, which includes C fruit categories; the support set includes K categories, where K≤C, and each category contains at least M fruit images, M≥1, where M is a natural number; the intersection of the categories in the training set and the categories in the support set is 0; the categories in the test set are a subset of the categories in the support set.

4. The method according to claim 1, characterized in that, S311 includes: The first feature map of the fruit includes: fruit color features, fruit shape features, fruit texture features, and fruit size features.

5. The method according to claim 1, characterized in that, The Ads-ResNet50 network includes: The multi-scale subnet structure consists of bottom-up modules, top-down modules, and horizontal modules; The bottom-up module obtains fruit image features through convolutional layers and downsampling layers; Next, the fruit image features are input into the top-down module, and the output features of the bottom-up convolutional layer are passed through a 1x1 convolutional kernel and then residually connected to the convolutional layer of the top-down module to obtain a connection feature map. The connection feature map is input into the horizontal module, and after passing through the convolutional layer and the connection layer, the first feature map of the fruit is obtained.

6. The method according to claim 1, characterized in that, S32 includes: The fruit defect detector performs defect detection based on the output fruit type and region information. Specific steps include: S321. Obtain the defective area of ​​the target fruit; S322. Segment the image of the defective region to obtain a defective image of the target fruit; S323. Use the fruit defect identifier to classify the target fruit defect image; S324. Output the defect category of the target fruit.

7. The method according to claim 6, characterized in that, The fruit defect detector uses a CNN model and employs different pooling methods for different feature pooling regions. Different pooling functions are selected based on the pooling value and the variance of the feature map. The calculation method is as follows: Wherein, S is the feature value obtained after pooling the defect image of the target fruit. Minimum pooling is used to extract features from the defective image of the target fruit, where n is the pooling step size and σ is the minimum pooling step size. P σ is the value in the pooling region. FM Let t be the variance on the feature map of the defective image of the target fruit. min t is the minimum value in pooling. max The maximum value in pooling, t ave The average of the maximum and minimum values ​​in pooling, where μ is the modification factor.

8. The method according to claim 1, characterized in that, S33 includes: The fruit grade classifier combines the output of the fruit defect detector to perform quality grading, and the specific steps include: S331. Obtain the target fruit and the fruit defect category; S332. Calculate the degree of impact of the defects of the target fruit on the surface of the target fruit, using the following formula: Among them, TFd a TF represents the defect area of ​​the target fruit defect image. a Let be the area of ​​the target fruit, and α be the parameter representing the influence of the defects of the target fruit on the target fruit; The area of ​​the target fruit is calculated based on the boundary regression box of the RPN network, using the following formula: TF a =t w *t h Among them, t x t y t w and t h These represent the x-coordinate, y-coordinate, height, and width of the predicted target fruit located at the center of the fruit image; S333. Design the fruit quality grader and classify the fruit using the second feature map of the fruit. S334. Output the quality level of the fruit image.

9. The method according to claim 8, characterized in that, The fruit quality grading system is based on maximizing fruit feature dependencies to establish a classifier, with the objective equation being: Wherein, the first term of the objective equation is the experience loss generated by the target fruit during the grading process, the second term is the regularization function of the grading process using the target fruit, and the parameters α and β represent the degree of influence of the defects of the target fruit on the appearance of the fruit and the defect type parameter of the target fruit, respectively. The first term of the objective equation is defined as: Ω1(f,x,y)=||P T year|| 2 Where, f(x) = p T x is the predicted grade of the target fruit, p is the characteristic of the fruit, x is the target fruit, and y is the grade label of the target fruit; The second term of the objective equation is defined as: Where, ||f(x) i )-f(x j )|| 2 The similarity in grade distribution between the target fruit and other fruits of the same type is denoted by N, where N is the number of fruits of the same type.

10. The method according to claim 9, characterized in that, The defect type parameter: The defect type parameter, denoted as β, is obtained by extracting the characteristics of fruit defect types and normalizing them to obtain the parameter that affects fruit grading.

Images