Fruit quality comprehensive grading method and device based on machine vision and X-ray

Abstract

The present invention provides a comprehensive fruit quality grading method and device based on machine vision and X-rays. The method first collects appearance images of the fruit to be graded, calculates the characteristic values ​​of three features: surface defects, shape, size, and color, and then classifies the fruit appearance into multiple grades based on factor analysis and expert experience. The fruit appearance images are annotated and labels are generated. Secondly, an appearance grading network is constructed, and the trained appearance grading network serves as the first primary classifier. X-ray images of the fruit to be graded are then collected and annotated. Three classifiers are constructed based on artificial features and CNN features. The results of the three classifiers are fused using a decision-level fusion method to create a second primary classifier. Finally, a secondary classifier is established based on the comprehensive fruit quality grading rules to output the grading results. By combining appearance quality and internal defect information, comprehensive grading indicators are achieved, meeting the demand for high-quality fruit grading.

Description

Technical Field

[0001] The present invention relates to the technical field of comprehensive quality grading of fruit appearance and interior, in particular to a method and device for comprehensive fruit quality grading based on machine vision and X-rays. Background Art

[0002] Currently, fruit appearance grading relies primarily on manual labor, which is subject to subjectivity and inefficiency. Furthermore, manual labor is unable to detect internal defects in fruit. Nondestructive testing has been successfully applied to detect internal defects in fruits such as citrus, providing a means for intelligent fruit quality grading.

[0003] Application No. 202011437542.7 discloses a neural network-based nondestructive fruit defect detection method and fruit grading method. This method acquires pear appearance images, X-ray images, slice images, and a slice chemical detection dataset. A fruit quality grading model is built based on the neural network. Labeled pear X-ray datasets are fed into the neural network for training. The trained model is then used to test the pear dataset. Pear quality grading incorporates both appearance and internal features. However, this method only performs nondestructive detection and grading of pear defects, extracting relatively few features and failing to fully reflect the overall quality of the fruit.

[0004] Application No. 201810695675.0 discloses a fruit quality visual inspection and grading device and method. The grading device includes a conveyor belt, a visual inspection system, a fruit grabbing and placing robot, a fruit box system, a housing, and a control system. The device utilizes a double-tapered roller fruit conveyor and flipper, which captures the appearance of the fruit in all directions. The robot grasps and grades the fruit with high accuracy, making the entire device relatively stable and reliable. However, this method only focuses on external appearance and cannot identify internal defects. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the technical problem to be solved by the present invention is to propose a comprehensive fruit quality grading method and device based on machine vision and X-rays.

[0006] The technical solution adopted by the present invention to solve the technical problem is: to provide a comprehensive fruit quality grading method based on machine vision and X-rays, characterized in that the method comprises the following steps:

[0007] The first step is to obtain the appearance images of the fruits to be graded through the CCD image acquisition module, pre-process the appearance images, and form an appearance image database with a large number of appearance images; extract the defect areas of the pre-processed appearance images to obtain an appearance defect image database;

[0008] The second step is to calculate the characteristic values ​​of three features: fruit surface defects, fruit shape and color. The characteristics of fruit surface defects include: total defect area, number of defects, and ratio of total defect area to number of defects; the characteristics of fruit shape include: ellipticity, perimeter, projected area, height, width, aspect ratio, and rectangularity; the characteristics of color include: R channel mean and variance, G channel mean and variance, and ratio of R channel mean to G channel mean;

[0009] For the appearance defect images in the appearance defect image database, the eigenvalues ​​of each fruit surface defect feature are extracted; for the appearance images in the appearance image database, the eigenvalues ​​of the fruit shape and size features and the color features are extracted to obtain a eigenvalue data table of the three features of fruit surface defects, fruit shape and size, and color; the appearance images are annotated using factor analysis, including using principal component analysis to extract the principal components of each feature in the eigenvalue data table, and calculating the variance contribution rate of each principal component, and calculating the comprehensive score of each appearance image. The comprehensive score is a linear combination of each principal component and its corresponding variance contribution rate, and then the comprehensive scores of each appearance image are sorted from high to low. Based on expert experience, the appearance quality of the fruit to be graded is divided into three grades: excellent, first grade, and second grade. According to the grading results, the appearance images are annotated and labels are generated;

[0010] Step 3: Build an appearance classification network and use the trained appearance classification network as the first primary classifier.

[0011] Step 4: Acquire X-ray images of the fruit to be graded through the X-ray image acquisition module and establish an X-ray image database; preprocess the X-ray images to obtain preprocessed X-ray images; slice the fruit samples that have undergone X-ray imaging, and annotate the preprocessed X-ray images based on defect information reflected by the slices, with the annotated information indicating whether there are defects, to obtain an annotated X-ray image;

[0012] Step 5: Extract the HOG and LBP features of the preprocessed X-ray image and construct classifiers for each of these features using the SVM model. A CNN classifier is constructed based on the neural network for the preprocessed X-ray image. The classification results of the three classifiers are fused at the decision level to obtain a second primary classifier to classify the internal defects of the fruit to be graded.

[0013] Step 6: Formulate comprehensive grading rules for fruit quality and establish a secondary classifier based on the ensemble learning strategy. The outputs of the first and second primary classifiers are used as inputs to the secondary classifier, which outputs the grading results according to the comprehensive grading rules for fruit quality, thus completing the entire grading process.

[0014] The present invention also provides a comprehensive fruit quality grading device based on machine vision and X-rays, comprising a support, a transmission mechanism, a CCD image acquisition module, an X-ray image acquisition module, and a grading mechanism; characterized in that the transmission mechanism comprises a tray, a driving sprocket, a driven sprocket, a transmission motor, a transmission shaft, a No. 1 chain, a No. 2 chain, a No. 3 chain, and a No. 4 chain;

[0015] Among them, the transmission motor is installed on one side of the bracket, the output shaft of the transmission motor is connected to one end of the transmission shaft, and the other end of the transmission shaft is rotatably connected to the other side of the bracket, and four driving sprockets are installed on the transmission shaft, and the four driving sprockets are respectively engaged with chain No. 1, chain No. 2, chain No. 3 and chain No. 4; chain No. 1 and chain No. 2 are respectively installed on one side of the bracket through multiple passive sprockets, and chain No. 1 and chain No. 2 do not interfere with each other; chain No. 3 and chain No. 4 are respectively installed on the other side of the bracket through multiple passive sprockets, and chain No. 3 and chain No. 4 do not interfere with each other, and the two chains on the same side of the bracket form a staggered distance in the horizontal direction for the passage of pallets; multiple pallets are distributed on the chain at intervals, and the four end corners of each pallet are respectively connected to chain No. 1, chain No. 2, chain No. 3 and chain No. 4; under the action of the transmission motor, the four chains make synchronous circular motion on the bracket to realize the lifting and horizontal movement of the pallet;

[0016] The CCD image acquisition module includes a first CCD camera and a second CCD camera. The two CCD cameras are located on the upper part of the bracket and on both sides of the bracket. The two CCD cameras are connected to the PC end using a network cable interface.

[0017] The X-ray image acquisition module includes an X-ray machine and an imaging board; the imaging board is located on the upper part of the bracket, and both sides of the imaging board are connected to the bracket; the X-ray machine is located in the middle of the bracket, and the transmitting end of the X-ray machine is facing the imaging board. The X-ray machine and the imaging board are connected to the PC through a USB interface and a network port respectively.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. To avoid the low grading accuracy caused by manual labeling during fruit grading, the present invention selects three features: fruit surface defects, fruit shape and size, and color as grading indicators for fruit appearance quality. After calculating the characteristic values ​​under each grading indicator, the factor analysis method is used to calculate the comprehensive score of the appearance image. Then, combined with expert experience, a comprehensive evaluation of the appearance quality of the graded fruit is conducted. The appearance image is annotated based on this evaluation result, laying the foundation for ensuring grading accuracy at the model training level.

[0020] 2. Combining CNN and SVM to build an appearance grading network to grade appearance quality; CNN has strong image feature extraction capabilities, and SVM has strong generalization capabilities for classifying small sample data, improving the accuracy of appearance quality classification.

[0021] 3. Since the grayscale changes of the core, stalk, calyx and other parts in the X-ray images of fruits are similar to the grayscale changes of defects inside the fruit, which will affect the identification of internal defects, the multi-channel fusion theory is used to combine deep convolutional neural networks and traditional artificial features for classification and prediction of the presence or absence of internal defects, thereby improving the classification accuracy of internal defects.

[0022] 4. The appearance grading network is used as the first primary classifier, and the HOG feature, LBP feature and CNN feature classifier after decision-level fusion are used as the second primary classifier. A comprehensive grading rule for fruit quality is formulated. Based on this, a secondary classifier is established to combine appearance quality and internal defect information to complete the comprehensive grading of fruit quality. The grading indicators are more comprehensive and meet the grading needs of high-quality fruits.

[0023] 5. This device, combined with the above-mentioned grading method, can complete the fruit quality grading with high accuracy and is suitable for grading fruit quality in orchards, wholesale markets, large supermarkets and other places. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of the VGG-16 network structure;

[0025] Figure 2 Schematic diagram of the overall structure of the device of the present invention;

[0026] Figure 3 It is a front view of the device of the present invention;

[0027] Figure 4 It is a left side view of the device of the present invention;

[0028] Figure 5 A top view of the device of the present invention;

[0029] Figure 6 Schematic diagram of the control of the device of the present invention;

[0030] In the figure, 1. bracket; 2. Chain No. 1; 3. sprocket bracket; 4. X-ray imaging plate; 5. sprocket; 6. CCD camera No. 1; 7. Chain No. 2; 8. Conveyor shaft; 9. Grading box; 10. Pallet; 11. Chain No. 3; 12. X-ray machine; 13. Chain No. 4; 14. Sorting servo; 15. Sorting paddle; 16. Conveyor motor; 17. CCD camera No. 2; 18. Driving sprocket. DETAILED DESCRIPTION

[0031] Specific embodiments of the present invention are given below. The embodiments are only used to explain the present invention and are not used to limit the scope of protection of the present invention.

[0032] The present invention is a comprehensive fruit quality grading method based on machine vision and X-ray (hereinafter referred to as the method), which comprises the following steps:

[0033] The first step is to obtain the appearance images of the fruits to be graded through the CCD image acquisition module and preprocess the appearance images. A large number of appearance images constitute an appearance image database, and the number of appearance images in the appearance image database is greater than 1000. The defect areas of the preprocessed appearance images are extracted to obtain an appearance defect image database.

[0034] Preprocessing: Perform grayscale transformation on the appearance image, binarize the grayscale image and use the Otsu algorithm to select the threshold. Pixels with grayscale values ​​less than the threshold are set to 0, and pixels with grayscale values ​​greater than the threshold are set to 255. Therefore, the foreground value of the binarized image is 255 and the background value is 0, thus obtaining the appearance mask. Morphological processing is performed on the appearance mask, and the processed appearance mask is multiplied with the original appearance image to segment the fruit appearance, obtain the segmented image, and establish the appearance image database.

[0035] Extract defect areas: grayscale the segmented image, perform binarization on the grayscale image, use the Otsu algorithm to select a suitable threshold, set the pixels with grayscale values ​​less than the threshold to 0, and set the pixels with grayscale values ​​higher than the threshold to 255 to obtain a defect mask; perform morphological processing on the defect mask, multiply the processed defect mask with the original segmented image, extract the defect area, obtain the appearance defect image, and establish an appearance defect image database.

[0036] The second step is to calculate the feature values ​​of three features: fruit surface defects, fruit shape, size, and color. Based on expert experience, the fruit appearance is divided into three grades: excellent, first-class, and second-class. The fruit appearance images are annotated and labels are generated.

[0037] According to national standards, the appearance quality of fruit mainly involves three characteristics: fruit surface defects, fruit shape and size, and color. The characteristics selected for fruit surface defect indicators include: total defect area, number of defects, and the ratio of total defect area to number of defects.

[0038] The calculation formula for the total defect area S is as follows:

[0039]

[0040] Among them, f(x,y) is the binary pixel of the appearance defect image, m f 、n fRepresents the total number of pixels on the x-axis and y-axis of the appearance defect image. S also represents the number of all pixels in the actual defect area.

[0041] The features selected for fruit size index are: ellipticity, perimeter, projected area, height, width, aspect ratio and rectangularity;

[0042] The calculation formula for the projected area A is as follows:

[0043]

[0044] Among them, g(x,y) is the binary pixel of the appearance image, m g 、n g Indicates the total number of pixels of the appearance image on the x-axis and y-axis;

[0045] Ovality measures the complexity of the fruit's shape and is calculated as follows:

[0046]

[0047] L is the boundary perimeter of the appearance image;

[0048] The perimeter is the contour length of the appearance image, and the calculation formula is as follows:

[0049]

[0050] N x Indicates the number of contour pixels in the horizontal direction, N y Indicates the number of contour pixels in the vertical direction, N d is the number of contour pixels in non-horizontal or vertical directions;

[0051] The height H refers to the maximum value of the appearance image in the vertical direction, that is, the height of the minimum bounding rectangle of the appearance image; the width W refers to the maximum value of the appearance image in the horizontal direction, that is, the width of the minimum bounding rectangle of the appearance image; the aspect ratio b refers to the ratio of the length to the width of the bounding rectangle of the appearance image, and is calculated as follows:

[0052]

[0053] The rectangularity c reflects the degree to which the outline of the fruit fills its minimum circumscribed rectangle. The calculation formula is as follows:

[0054]

[0055] The features selected for color index are: R channel mean N R and variance S R , G channel mean N G With variance S G , the ratio of the mean value of the R channel to the mean value of the G channel NR / N G , the corresponding calculation formula is as follows:

[0056]

[0057]

[0058]

[0059]

[0060] Where N is the total number of pixels in each channel, R and G are the pixel values ​​of the pixels in the corresponding channel respectively;

[0061] For the appearance defect images in the appearance defect image database, the eigenvalues ​​of each fruit surface defect feature are extracted; for the appearance images in the appearance image database, the eigenvalues ​​of the fruit shape, size and color features are extracted, and then a eigenvalue data table of the three features of fruit surface defects, fruit shape, size and color is obtained; each data in the eigenvalue data table is standardized, and the standardized data is subjected to the KMO test and Bartlett test. When the KMO test coefficient is greater than 0.5 and the P value of the Bartlett test is less than 0.05, the prerequisites for factor analysis are met;

[0062] The principal component analysis method is used to extract the principal components of each feature in the eigenvalue data table, and the variance contribution rate of each principal component is calculated; the comprehensive score of each appearance image is calculated, and the comprehensive score is a linear combination of each principal component and its corresponding variance contribution rate. The comprehensive scores of each appearance image are then sorted from high to low, and the appearance quality of the fruits to be graded is divided into three grades: excellent, first grade, and second grade based on expert experience. The appearance images are annotated and labels are generated based on these grading results.

[0063] Step 3: Build an appearance classification network and use the trained appearance classification network as the first primary classifier.

[0064] Convolutional Neural Networks (CNNs) have excellent learning capabilities for image features. To further improve the accuracy of CNNs in grading fruit appearance quality, we combined the advantages of Support Vector Machines (SVMs) in processing small samples and generalizing well. We replaced the Softmax layer of the CNN model with an SVM model to obtain an appearance grading network, further improving grading accuracy.

[0065] The CNN model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and a Softmax layer. The input layer is used to input a data set, the convolutional layer is used to extract features, and the pooling layer is used to reduce the image dimension and expand the receptive field. The fully connected layer is used to integrate local information that distinguishes categories, and the Softmax layer is used to output the predicted probability of the image at each level. The core of the SVM model is to find the optimal hyperplane so that the distance between all sample points and the hyperplane is greater than a certain value. In this embodiment, the CNN model uses VGG16 as the backbone network, and its structure is as follows: Figure 1 As shown in FIG, the annotated appearance image is used as a training sample image to train the appearance grading network to obtain a trained appearance grading network.

[0066] Step 4: Use the X-ray image acquisition module to obtain X-ray images of the fruit to be graded and establish an X-ray image database; perform preprocessing on the X-ray images, including Gaussian filtering and adaptive histogram equalization, to obtain preprocessed X-ray images; slice the fruit samples that have undergone X-ray imaging to observe internal defects in the fruit to be graded, and annotate the preprocessed X-ray images based on the defect information reflected by the slices, with the annotation information indicating whether there are defects, to obtain an annotated X-ray image; and divide the annotated X-ray images into a training set, a validation set, and a test set.

[0067] Step 5: Extract HOG and LBP features from the preprocessed X-ray image, and use the SVM model to construct classifiers for these two artificial features. A CNN classifier is constructed based on the VGG16 network for the preprocessed X-ray image. Defects are predicted using the HOG, LBP, and CNN channels, and the classification results of the three classifiers are fused using a decision-level fusion method to establish a second primary classifier for classifying internal defects in the fruit.

[0068] The three classifiers mentioned above are trained using the training set, and the test set is input into the three trained classifiers for prediction, and the classification performance evaluation index and prediction probability P of each classifier are obtained. q , q = 1, 2, ..., Q, where Q is the number of classifiers, i.e., the number of channels; the predicted probability includes the probability of internal defects and non-defects; the classification performance evaluation indicators include accuracy, recall rate, F1 score, and precision;

[0069] According to the number of classifiers, each classification performance evaluation index and the total number of classification performance evaluation indexes K, a multi-channel evaluation matrix D is constructed. Q,K =(d q,k ) Q×K ; Calculate the weights of each classification performance evaluation index according to formula (11):

[0070]

[0071] Among them, α k represents the weight of the k-th classification performance evaluation index, represents the relative importance of the k-th classification performance evaluation index in the decision-making process, represents the standard deviation of the k-th classification performance evaluation index among all classifiers, d q,k represents the kth classification performance evaluation index in the qth classifier, that is, the classification performance evaluation index corresponding to the qth row and kth column of the multi-channel evaluation matrix; r kg The correlation coefficient matrix R = (r kg ) K×K The terms in are calculated as shown in formula (12); is the average value of the k-th classification performance evaluation index in all classifiers, and the calculation formula is as shown in formula (13);

[0072]

[0073]

[0074] Among them, d q,g represents the g-th classification performance evaluation index in the q-th classifier, g = 1,…,K;

[0075] According to formulas (14) and (15), the multi-channel evaluation matrix is ​​normalized and weighted to obtain the weight of the k-th classification performance evaluation index in each classifier;

[0076]

[0077]

[0078] Formula (16) is used to calculate the weight of each classifier when performing multi-channel fusion:

[0079]

[0080] Among them, Dis q,min and Dis q,max The calculation formula is as follows:

[0081]

[0082]

[0083] Visible Dis q,min and Dis q,max They are vectors With vector and The Euclidean distance of and are the minimum and maximum values ​​of the K classification performance evaluation indicators in each classifier;

[0084]

[0085]

[0086]

[0087] Formula (21) is used to calculate the fusion score P of the internal defects of the fruit to be graded, and the category corresponding to the maximum fusion score is taken as the internal defect classification result, where the category is whether the fruit has defects or not.

[0088] Step 6. Formulate comprehensive grading rules for fruit quality: appearance quality is divided into excellent, first-class, and second-class, and internal quality is divided into defective and non-defective. The grade with excellent appearance and no internal defects is set as high quality, the grade with first-class appearance and no internal defects is set as medium quality, and the grades with excellent appearance and internal defects, first-class appearance and internal defects, and second-class appearance are set as low quality; based on the ensemble learning strategy, the outputs of the first and second primary classifiers are used as the inputs of the secondary classifier, and the secondary classifier outputs the grading results according to the comprehensive grading rules for fruit quality, thus completing the entire grading process.

[0089] like Figures 2 to 6 As shown, the present invention also provides a fruit quality comprehensive grading device based on machine vision and X-ray (hereinafter referred to as the device), comprising a bracket 1, a transmission mechanism, a CCD image acquisition module, an X-ray image acquisition module and a grading mechanism;

[0090] The bracket 1 is constructed of aluminum profiles, and a closed housing made of a black acrylic plate (not shown in the figure) is provided on the outside of the bracket 1 to eliminate interference from the external environment on CCD imaging and ensure the uniformity of the acquisition environment of the CCD image acquisition module; an LED light strip is provided on the inside of the closed housing as a light source for the CCD image acquisition module;

[0091] The conveying mechanism includes a tray 10, a driving sprocket 18, a driven sprocket 5, a conveying motor 16, a conveying shaft 8, a No. 1 chain 2, a No. 2 chain 7, a No. 3 chain 11 and a No. 4 chain 13;

[0092] Among them, the transmission motor 16 is installed on one side of the bracket 1, and the output shaft of the transmission motor 16 is connected to one end of the transmission shaft 8 through a coupling, and the other end of the transmission shaft 8 is rotatably connected to the other side of the bracket 1. Four active sprockets 18 are fixed on the transmission shaft 8, and the four active sprockets 18 are respectively engaged with the No. 1 chain 2, the No. 2 chain 7, the No. 3 chain 11 and the No. 4 chain 13; the No. 1 chain 2 and the No. 2 chain 7 are respectively installed on one side of the bracket 1 through a plurality of passive sprockets 5, and the No. 1 chain 2 and the No. 2 chain 7 do not interfere with each other, and the passive sprocket 5 is connected to the bracket 1 through the sprocket bracket 3; the No. 3 chain 11 and the No. 4 chain 13 are respectively installed on the other side of the bracket 1 through a plurality of passive sprockets 5, and the No. 3 chain 11 and the No. 4 chain 13 do not interfere with each other. The two chains on the same side of the bracket 1 form a staggered distance in the horizontal direction for the pallet 10 to pass through; the No. 1 chain 2 and the No. 2 chain 7 are a group, close to the inner side of the bracket 1; the No. 3 chain 11 and The fourth chain 13 is a group, close to the outside of the bracket 1; a plurality of trays 10 are distributed on the chain at intervals, and the four end corners of each tray 10 are respectively fixedly connected to the No. 1 chain 2, the No. 2 chain 7, the No. 3 chain 11 and the No. 4 chain 13. The length of the tray 10 is equivalent to the offset distance, and the thickness of the No. 2 chain 7 and the No. 3 chain 11 is greater than the thickness of the No. 1 chain 2 and the No. 4 chain 13, so that the connecting parts of the tray 10 and the No. 2 chain 7 and the No. 3 chain 11 are respectively located below the No. 1 chain 2 and the No. 4 chain 13, ensuring that the four chains can realize the horizontal movement of the tray 10; the transmission motor 16 drives the active sprocket 18 to rotate, so that the four chains perform synchronous circular motion on the bracket 1. Under the circular motion of the four chains, the tray 10 can be vertically lifted and moved horizontally while maintaining a horizontal posture; the tray 10 is provided with a plurality of groups of grooves for placing fruits to be graded along the length direction, and each group includes two grooves distributed on both sides of the tray 10;

[0093] The CCD image acquisition module includes a first CCD camera 6 and a second CCD camera 17. The two CCD cameras are located on the upper part of the bracket 1 and on both sides of the bracket 1, and are used to collect the appearance images of the fruits to be graded; the two CCD cameras are respectively connected to a PC with a gigabit network port using an RJ45 network cable interface to transmit the collected appearance images to the PC;

[0094] The X-ray image acquisition module is used to collect X-ray images of fruits to be graded, and includes an X-ray machine 12 and an imaging plate 4; the imaging plate 4 is located on the upper part of the bracket 1, and both sides of the imaging plate 4 are connected to the bracket 1; the X-ray machine 12 is located in the middle of the bracket 1, and the transmitting end of the X-ray machine 12 is facing the imaging plate 4. The X-rays emitted by the X-ray machine 12 are irradiated on the imaging plate 4, and the imaging plate 4 collects X-ray images of the fruits to be graded; the X-ray machine 12 and the imaging plate 4 are respectively connected to the PC end through the USB interface and the gigabit network port, and the image transmission, X-ray triggering and other control functions are realized through the PC end. The X-ray images collected by the X-ray image acquisition module are transmitted to the PC end;

[0095] The grading mechanism includes a grading servo 14, a grading paddle 15 and a grading box 9; the grading box 9 is located at the lower part of the bracket 1, and the grading box 9 is provided with three groups of grading grids along the length direction, which are used to sort high-quality, medium-quality and low-quality fruits respectively, and each group includes two grading grids distributed on both sides of the grading box 9. The tray 10 can pass between the two grading grids and stay at the position of each group of grading grids; three groups of grading servos 14 are located above the grading box 9, and each group of grading servos 14 is located between two grading grids of the corresponding same group of grading grids. Each group includes multiple grading servos 14 with the same number of groove groups on the tray 10, and a grading paddle 15 is installed on the output shaft of each grading servo 14. Under the action of the grading servo 14, the grading paddle 15 can rotate to both sides of the grading box 9 to distribute the fruits to be graded in the corresponding positions to the corresponding grading grids, thereby realizing comprehensive grading of fruits.

[0096] The positions of each group of grading grids on the grading box 9, the positions of the CCD image acquisition module and the X-ray image acquisition module on the bracket 1 are respectively installed with through-beam photoelectric switches, which are used to detect the position of the pallet 10, so as to accurately control the start and stop of the conveying mechanism, and also to achieve accurate positioning of the pallet 10 at the image acquisition position and the grading mechanism; the through-beam photoelectric switch is connected to the PLC controller. The model of the through-beam photoelectric switch is E3F-D5N3-5L, which is NPN type and includes two parts, a transmitting end and a receiving end. There is a signal terminal at the receiving end. When there is an obstacle blocking the transmitting end and the receiving end of the through-beam switch, the receiving end sends a signal to the PLC controller, indicating that the pallet 10 has accurately reached the predetermined position.

[0097] The device uses a Siemens S7-200 smart family PLC controller, which offers strong resistance to environmental interference. It features Ethernet, RS-485, 24 digital inputs, 16 digital outputs, expansion module interfaces, and a memory card interface. The PLC connects to the PC's serial port via the RS-485 interface and controls the X-ray machine 12, the stepping servo 14, and the conveyor motor 16. The stepping servo 14 is a 42BYGH24 model with a torque of 0.13 N and a step angle of 1.8°. The stepping servo driver is a TB6600.

[0098] CCD camera 1 (6) and CCD camera 2 (17) are both MicroVision MV-HS2000GM / C models, with a pixel size of 2.4 x 2.4 μm, a C-mount lens, and a BT-11C0618MP10 lens. X-ray machine 12 is SF100BY, and imaging board 4 is Venu1717X, used to receive X-rays that penetrate the fruit and produce images. The X-ray image acquisition module is used for medical low-dose radiation imaging, and the radiation impact on the human body is negligible. Therefore, it is not shielded from radiation.

[0099] The working principle and workflow of the device are as follows:

[0100] The movement process of one of the trays 10 is taken as an example for explanation. The initial position of the tray 10 is located at the right end of the lower part of the device. The fruit to be graded is placed on the groove of the tray 10, and the device is started. The conveying motor 16 drives the active sprocket 18 to rotate, so that the four chains make a synchronous circular motion counterclockwise (from the main view direction of the device) on the bracket 1, so that the tray 10 rises from the lower part of the device, and starts to move horizontally to the left after rising to the maximum height. When the tray 10 moves to the CCD image acquisition module, the tray 10 blocks the opposing photoelectric switch, and the receiving end of the opposing photoelectric switch sends a signal to the PLC controller, the conveying motor 16 stops rotating, and the four chains stop moving around. The tray 10 stops at the CCD image acquisition module, and then the CCD image acquisition module collects the appearance image of the fruit to be graded and uploads it to the PC. After the appearance image is collected, the PLC controller sends a control signal to make the conveying motor 6 continue to rotate, the four chains continue to make a counterclockwise circular motion, and the tray 10 continues to move horizontally to the left. When it moves to the X-ray image acquisition module, the tray 10 The through-beam photoelectric switch is blocked, and the receiving end of the through-beam photoelectric switch sends a signal to the PLC controller, causing the conveying motor 16 to stop rotating, causing the four chains to stop rotating, and the tray 10 to stop at the X-ray image acquisition module. The X-ray image acquisition module collects X-ray images of the fruits to be graded and transmits them to the PC end. The PC end performs comprehensive grading on all the fruits to be graded on the tray 10 according to the above method; at the same time, after the X-ray image acquisition is completed, the conveying motor 16 continues to rotate, causing the tray 10 to vertically descend to the lower part of the device, and then the tray 10 moves horizontally to the right. When it moves to the position of the first group of grading grids in the grading box 9, the conveying motor 16 stops rotating, the tray 10 stops moving, and the PC end transmits the grading result to the PLC controller. The PLC controller controls the rotation of the grading servo 14 at the corresponding position according to the grading result, and the grading paddle 15 sorts the fruits to be graded into the corresponding grading grids. The tray 10 will stay at the position of each group of grading grids on the grading box 9 until all the fruits on the tray 10 are graded and the next cycle can be entered.

[0101] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A comprehensive fruit quality grading method based on machine vision and X-ray, characterized in that: The method comprises the following steps: The first step is to obtain the appearance images of the fruits to be graded through the CCD image acquisition module, pre-process the appearance images, and form an appearance image database with a large number of appearance images; extract the defect areas of the pre-processed appearance images to obtain an appearance defect image database; The second step is to calculate the characteristic values ​​of three features: fruit surface defects, fruit shape and color. The characteristics of fruit surface defects include: total defect area, number of defects, and ratio of total defect area to number of defects; the characteristics of fruit shape include: ellipticity, perimeter, projected area, height, width, aspect ratio, and rectangularity; the characteristics of color include: R channel mean and variance, G channel mean and variance, and ratio of R channel mean to G channel mean; For the appearance defect images in the appearance defect image database, the eigenvalues ​​of each fruit surface defect feature are extracted; for the appearance images in the appearance image database, the eigenvalues ​​of the fruit shape and size features and the color features are extracted to obtain a eigenvalue data table of the three features of fruit surface defects, fruit shape and size, and color; the appearance images are annotated using factor analysis, including using principal component analysis to extract the principal components of each feature in the eigenvalue data table, and calculating the variance contribution rate of each principal component, and calculating the comprehensive score of each appearance image. The comprehensive score is a linear combination of each principal component and its corresponding variance contribution rate, and then the comprehensive scores of each appearance image are sorted from high to low. Based on expert experience, the appearance quality of the fruit to be graded is divided into three grades: excellent, first grade, and second grade. According to the grading results, the appearance images are annotated and labels are generated; Step 3: Build an appearance classification network and use the trained appearance classification network as the first primary classifier. Step 4: Acquire X-ray images of the fruit to be graded through the X-ray image acquisition module and establish an X-ray image database; preprocess the X-ray images to obtain preprocessed X-ray images; slice the fruit samples that have undergone X-ray imaging, and annotate the preprocessed X-ray images based on defect information reflected by the slices, with the annotated information indicating whether there are defects, to obtain an annotated X-ray image; Step 5: Extract the HOG and LBP features of the preprocessed X-ray image and construct classifiers for each of these features using the SVM model. A CNN classifier is constructed based on the neural network for the preprocessed X-ray image. The classification results of the three classifiers are fused at the decision level to obtain a second primary classifier to classify the internal defects of the fruit to be graded. Step 6: Formulate comprehensive grading rules for fruit quality. Based on the ensemble learning strategy, the outputs of the first and second primary classifiers are used as inputs to the secondary classifier. The secondary classifier outputs the grading results according to the comprehensive grading rules for fruit quality, thus completing the entire grading process.

2. The fruit quality comprehensive grading method based on machine vision and X-ray according to claim 1, wherein The fifth step includes: training the three classifiers mentioned above, using the trained three classifiers to make predictions, extracting the classification performance evaluation index of each classifier and calculating the prediction probability of each classifier, the prediction probability including the probability of internal defects and the probability of internal non-defects; constructing a multi-channel evaluation matrix based on the number of classifiers, each classification performance evaluation index and the total number of classification performance evaluation indexes; calculating the weight of each classification performance evaluation index in each classifier and the weight of each classifier when performing multi-channel fusion; Formula (21) is used to calculate the fusion score P of the internal defects of the fruit to be graded, and the category corresponding to the maximum fusion score is taken as the final classification result; Among them, P q represents the predicted probability of the qth classifier, q=1,2,…,Q, Q is the number of classifiers, ω q Represents the weight of the qth classifier when performing multi-channel fusion.

3. The fruit quality comprehensive grading method based on machine vision and X-ray according to claim 2, is characterized in that, The classification performance evaluation indicators include accuracy, recall, F1 score and precision. The weights of each classification performance evaluation indicator are calculated according to formula (11); Among them, α k represents the weight of the kth classification performance evaluation index, K represents the total number of classification performance evaluation indicators, represents the relative importance of the k-th classification performance evaluation index in the decision-making process, Represents the k-th classification performance evaluation index d q,k The standard deviation among all classifiers, d q,k represents the k-th classification performance evaluation index in the q-th classifier; r kg The correlation coefficient matrix R = (r kg ) K×K The terms in are calculated as shown in formula (12); is the average value of the k-th classification performance evaluation index in all classifiers, and the calculation formula is as shown in formula (13); Among them, d q,g represents the g-th classification performance evaluation index in the q-th classifier, g = 1,…,K; According to formulas (14) and (15), the multi-channel evaluation matrix is ​​normalized and weighted to obtain the weight of the k-th classification performance evaluation index in each classifier; Formula (16) is used to calculate the weight of each classifier when performing multi-channel fusion: Among them, Dis q,min and Dis q,max The calculation formula is as follows: Dis q,min and Dis q,max They are vectors With vector and The Euclidean distance of and are the minimum and maximum values ​​of the K classification performance evaluation indicators in each classifier.

4. The fruit quality comprehensive grading method based on machine vision and X-ray according to claim 1, is characterized in that, The appearance classification network uses VGG16 as the backbone network, and replaces the Softmax layer of the VGG16 network with the SVM model.

5. The fruit quality comprehensive grading method based on machine vision and X-ray according to claim 1 is characterized in that, The comprehensive grading rules for fruit quality are as follows: appearance quality is divided into superior, first and second grades, and internal quality is divided into defective and no defect. The grade with superior appearance and no internal defect is defined as high quality, the grade with first grade appearance and no internal defect is defined as medium quality, and the grades with superior appearance and internal defect, first grade appearance and internal defect, and second grade appearance are defined as low quality.

6. A fruit quality comprehensive grading device based on machine vision and X-rays using the method of claim 1, comprising a support, a transmission mechanism, a CCD image acquisition module, an X-ray image acquisition module, and a grading mechanism; characterized in that: The transmission mechanism includes a tray, a driving sprocket, a driven sprocket, a transmission motor, a transmission shaft, a No. 1 chain, a No. 2 chain, a No. 3 chain and a No. 4 chain; Among them, the transmission motor is installed on one side of the bracket, the output shaft of the transmission motor is connected to one end of the transmission shaft, and the other end of the transmission shaft is rotatably connected to the other side of the bracket, and four driving sprockets are installed on the transmission shaft, and the four driving sprockets are respectively engaged with chain No. 1, chain No. 2, chain No. 3 and chain No. 4; chain No. 1 and chain No. 2 are respectively installed on one side of the bracket through multiple passive sprockets, and chain No. 1 and chain No. 2 do not interfere with each other; chain No. 3 and chain No. 4 are respectively installed on the other side of the bracket through multiple passive sprockets, and chain No. 3 and chain No. 4 do not interfere with each other, and the two chains on the same side of the bracket form a staggered distance in the horizontal direction for the passage of pallets; multiple pallets are distributed on the chain at intervals, and the four end corners of each pallet are respectively connected to chain No. 1, chain No. 2, chain No. 3 and chain No. 4; under the action of the transmission motor, the four chains make synchronous circular motion on the bracket to realize the lifting and horizontal movement of the pallet; The CCD image acquisition module includes a first CCD camera and a second CCD camera. The two CCD cameras are located on the upper part of the bracket and on both sides of the bracket. The two CCD cameras are connected to the PC end using a network cable interface. The X-ray image acquisition module includes an X-ray machine and an imaging board; the imaging board is located on the upper part of the bracket, and both sides of the imaging board are connected to the bracket; the X-ray machine is located in the middle of the bracket, and the transmitting end of the X-ray machine is facing the imaging board. The X-ray machine and the imaging board are connected to the PC through a USB interface and an Internet port respectively; The grading mechanism includes a grading steering gear, a grading paddle, and a grading box; the grading box is located at the lower part of the bracket, and the grading box is provided with multiple groups of grading grids along the length direction, each group including two grading grids distributed on both sides of the grading box, and the tray can pass through the middle of the two grading grids and stay at the position of each group of grading grids; multiple groups of grading steering gears are located above the grading box, and each group of grading steering gears is located between two grading grids of the corresponding same group of grading grids, and each group includes multiple grading steering gears with the same number of groove groups on the tray, and a grading paddle is installed on the output shaft of each grading steering gear, and the grading paddles are used to sort the fruits to be graded at the corresponding position into the corresponding grading grid; The position of each group of grading grids on the grading box, the position of the bracket for mounting the CCD image acquisition module, and the position of the X-ray image acquisition module are respectively installed with a beam type photoelectric switch.

7. The fruit quality comprehensive grading device based on machine vision and X-ray according to claim 6 is characterized in that: The tray is provided with a plurality of groups of grooves for placing fruits to be graded along the length direction, and each group includes two grooves distributed on both sides of the tray.

8. The fruit quality comprehensive grading device based on machine vision and X-ray according to claim 6 or 7, characterized in that: The bracket is constructed of aluminum profiles, a closed shell made of black plate is provided on the outside of the bracket, and an LED light strip is provided on the inside of the closed shell as the light source of the CCD image acquisition module.

Images