Fruit grading method and system based on FPGA

Abstract

The invention provides a fruit grading method and system based on an FPGA (Field Programmable Gate Array), relates to the technical field of image recognition, and realizes automatic grading and classification of apples by using the FPGA as a hardware platform and combining computer vision and machine learning algorithms. By utilizing the programmability and the parallel computing capability of the FPGA, the system can quickly process a large amount of image data, and the real-time performance and the efficiency of the grading system are improved. Meanwhile, computer vision, an image processing technology and a machine learning technology are adopted, and apple features are extracted and analyzed, so that the accuracy and the stability of the grading system are improved. In addition, the grading system has good expandability, can be designed in a customized mode according to requirements, and is suitable for fruit grading systems of different scales and requirements.

Description

Technical Field

[0001] The invention relates to the technical field of image recognition, in particular to a fruit grading method and system based on FPGA. Background Art

[0002] Traditional fruit grading usually relies on manual work, which is time-consuming and easily affected by subjective factors, resulting in inconsistency and inaccuracy in grading results. With the development of computer vision and machine learning technology, automated fruit grading systems have gradually attracted attention.

[0003] At present, the realization of the automatic fruit grading system is mainly based on computer vision technology, image processing technology and machine learning algorithms. Among them, computer vision technology can extract various characteristics of fruits, such as color, shape, size and texture, as the basis for classification and grading. Machine learning algorithms can realize automatic classification and grading of fruit characteristics through training and learning of large amounts of data.

[0004] However, the existing automated grading systems still have some limitations in terms of real-time performance, accuracy, scalability, and mobility. Real-time performance refers to the system's ability to process and analyze large amounts of image data in a short period of time to adapt to high-speed assembly line production environments. Accuracy refers to the system's ability to accurately identify and classify fruits to ensure product quality and safety. Scalability refers to the system's ability to be customized according to demand to adapt to fruit grading systems of different scales and requirements. Mobility refers to the ability to move equipment in special production environments to meet different production needs. Summary of the invention

[0005] In view of the deficiencies in the prior art, the object of the present invention is to provide a fruit grading method based on FPGA, comprising:

[0006] Step 1: Collect images of the fruit to be evaluated;

[0007] Step 2: Recognize the image of the fruit to be evaluated based on FPGA and determine the first grade of the fruit to be evaluated;

[0008] Step 2.1: preprocessing the image of the fruit to be evaluated;

[0009] Step 2.2: The pre-processed image is processed by a pre-trained fruit grade assessment model to obtain a first grade of the fruit to be assessed, wherein the first grade represents the grade of the contour of the fruit to be assessed, and the fruit grade assessment model is obtained by model training based on training samples, wherein the training samples include input samples and output samples, wherein the input samples are fruit images, and the output samples are contour grades of the fruit images;

[0010] Step 3: According to the RGB of each pixel in the image of the fruit to be evaluated, the hue value, lightness value and saturation value of each pixel are calculated to determine the target color of the fruit to be evaluated; according to the threshold interval of the hue value, the threshold interval of the lightness value and the threshold interval of the saturation value preset for the target color, and the hue value, lightness value and saturation value of each pixel in the image of the fruit to be evaluated, it is determined whether each pixel in the image of the fruit to be evaluated belongs to the target color; according to the ratio of the number of pixels belonging to the target color in the fruit to be evaluated to the number of all pixels in the image of the fruit to be evaluated, the second grade of the fruit to be evaluated is determined;

[0011] Step 4: In a preset grading table, according to the first grade and the second grade, determine the target grade of the fruit to be assessed;

[0012] Step 5: Output the image of the fruit to be assessed, the image processed by the fruit grade assessment model, and the target grade of the fruit to be assessed;

[0013] The calculation contents in step 2 to step 4 are all calculated in FPGA by means of parallel operation and pipeline operation.

[0014] Optionally, the fruit grade assessment model includes a convolution layer, an activation layer, a pooling layer and a fully connected layer, the convolution layer includes C1, C2, C3, and C4, the C1 includes 30 convolution kernels, the number of channels of C1 is 30, the C2 includes 30 convolution kernels, the number of channels of C2 is 30, the C3 includes 16 convolution kernels, the number of channels of C3 is 16, the C4 includes 16 convolution kernels, the number of channels of C4 is 16, the function in the activation layer is a LeakyRelu function, the pooling layer includes S1 and S2, the fully connected layer includes a first fully connected layer, a second fully connected layer and a third fully connected layer, the first fully connected layer includes 120 neurons, the second fully connected layer includes 84 neurons, and the third fully connected layer includes 4 neurons;

[0015] Based on this, step 2.2 specifically includes:

[0016] The preprocessed image is input into C1, activation layer, C2, activation layer, S1, C3, activation layer, C4, activation layer, S2, the first fully connected layer, the second fully connected layer and the third fully connected layer in sequence to obtain the first grade of the fruit to be evaluated.

[0017] Optionally, step 2.1 specifically includes:

[0018] Step 2.1.1: Calculate the grayscale value of the pixel points in the image of the fruit to be evaluated to obtain a grayscale image;

[0019] Step 2.1.2: The grayscale image is processed by a median filtering noise reduction algorithm to obtain a first image. Specifically, a 3*3 sliding window is defined, and the sliding window is used to slide in the grayscale image. For the grayscale values ​​of the pixels in 3 rows and 3 columns in the sliding window, the maximum grayscale value of each row, the middle grayscale value of each row, and the minimum grayscale value of each row are determined, and the minimum value of the three maximum values, the middle value of the three middle values, and the maximum value of the three minimum values ​​are obtained, and then the minimum value of the three maximum values, the middle value of the three middle values, and the minimum value of the three minimum values ​​are obtained. The intermediate value between the maximum and minimum values ​​is taken as the first intermediate value, and it is judged whether the pixels in the 2nd row and 2nd column of the sliding window are in the contour position of the fruit to be evaluated; if the pixels in the 2nd row and 2nd column of the sliding window are in the contour position of the fruit to be evaluated, the grayscale values ​​of the pixels in the 2nd row and 2nd column of the sliding window are kept unchanged; if the pixels in the 2nd row and 2nd column of the sliding window are not in the contour position of the fruit to be evaluated, the grayscale values ​​of the pixels in the 2nd row and 2nd column of the sliding window are replaced with the first intermediate value, and the sliding window is moved so that each pixel in the grayscaled image is subjected to the above operation, and after all pixels are subjected to the above operation, a first image is obtained;

[0020] Step 2.1.3: The first image is processed by an improved Sobel sharpening algorithm to obtain a second image. Specifically, the grayscale value of the first image in the 3*3 sliding window is calculated according to 8 preset groups of different weight parameters to obtain a second intermediate value, each group of weight parameters is a 3*3 matrix, and the grayscale values ​​of pixels in 2 rows and 2 columns in the sliding window are replaced by the second intermediate value. The sliding window is moved so that each pixel in the first image is subjected to the above operation. After all pixels are subjected to the above operation, the second image is obtained;

[0021] Step 2.1.4: Binarize the second image to obtain a preprocessed image.

[0022] Optionally, step 2.1.1 specifically includes:

[0023] For each pixel in the image of the fruit to be evaluated, the grayscale value is calculated using the following formula:

[0024] Gray(i,j)=0.299*R(i,j)+0.587*G(i,j)+0.114*B(i,j)

[0025] Among them, (i, j) is the coordinate of the pixel point in the image of the fruit to be evaluated, R(i, j) is the red component of the pixel point in the image of the fruit to be evaluated, G(i, j) is the green component of the pixel point in the image of the fruit to be evaluated, B(i, j) is the blue component of the pixel point in the image of the fruit to be evaluated, and Gray(i, j) is the grayscale value of the pixel point in the image of the fruit to be evaluated.

[0026] Optionally, in step 3, according to the RGB of each pixel in the image of the fruit to be evaluated, the hue value, lightness value and saturation value of each pixel are calculated to determine the target color of the fruit to be evaluated, which specifically includes:

[0027] According to the RGB of each pixel, the maximum value and the minimum value of the R value, G value, and B value are obtained. According to the maximum value and the minimum value, the hue value, lightness value, and saturation value of each pixel are calculated, which is specifically achieved through the following formula:

[0028]

[0029] V=max

[0030] Among them, H is the hue value of the pixel, S is the saturation value of the pixel, V is the lightness value of the pixel, G is the G value, B is the B value, R is the R value, min is the minimum value among the R value, G value, and B value, and max is the maximum value among the R value, G value, and B value.

[0031] Optionally, the fruit grade assessment model is deployed on FPGA hardware, and the weight parameter is converted from floating point data to fixed point data through the following formula, which is specifically implemented through the following formula:

[0032] Fixed_data=round(float_data*2 s ),s∈Z

[0033] Among them, Fixed_data is the fixed-point data of the weight parameter, round(x) is rounding, s is the scaling factor, Z represents an integer, and float_data is the floating-point data of the weight parameter.

[0034] An FPGA-based fruit grading system, used to implement the FPGA-based fruit grading method according to claim 1, comprising an image acquisition sensor, an FPGA and a display;

[0035] The image acquisition sensor is used to acquire images of the fruit to be evaluated, and transmit the acquired images of the fruit to be evaluated to the FPGA;

[0036] The FPGA is used to receive an image of the fruit to be assessed, implement the contents of step 2 to step 4 in claim 1, and transmit the image processed by the fruit grade assessment model and the target grade of the fruit to be assessed to the display;

[0037] The display is used to output and display the image of the fruit to be assessed, the image processed by the fruit grade assessment model, and the target grade of the fruit to be assessed.

[0038] Optionally, when the display outputs the image of the fruit to be evaluated, the image processed by the fruit grade evaluation model and the target grade of the fruit to be evaluated, the row and column area ranges of the image of the fruit to be evaluated on the screen of the display are set to [100,323], indicating from the 100th row to the 323rd row, and from the 100th column to the 323rd column; the row range of the image processed by the fruit grade evaluation model is [0,223], indicating from the 0th row to the 223rd row; the column range is [400,623], indicating from the 400th column to the 623rd column; the row range of the target grade of the fruit to be evaluated is [0,99], indicating from the 0th row to the 99th row; the column range is [100,199], indicating from the 100th column to the 199th column.

[0039] The beneficial effects of adopting the above technical solution are:

[0040] The present invention uses FPGA as a hardware platform and combines computer vision and machine learning algorithms to achieve automatic grading and classification of apples. Utilizing the programmability and parallel computing capabilities of FPGA, the system can quickly process a large amount of image data and improve the real-time performance and efficiency of the grading system. At the same time, computer vision, image processing technology and machine learning technology are used to extract and analyze apple features to improve the accuracy and stability of the grading system. In addition, the grading system of the present invention has good scalability and can be customized according to needs to adapt to fruit grading systems of different scales and requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of a fruit grading method based on FPGA in an embodiment of the present invention;

[0042] Figure 2 8 sets of weight parameter templates in the embodiment of the present invention, wherein Figure a is a template of a weight parameter of 0°, Figure b is a template of a weight parameter of 45°, Figure c is a template of a weight parameter of 90°, Figure d is a template of a weight parameter of 135°, Figure e is a template of a weight parameter of 180°, Figure f is a template of a weight parameter of 225°, Figure g is a template of a weight parameter of 270°, and Figure h is a template of a weight parameter of 315°;

[0043] Figure 3 This is a schematic diagram of the structure of a fruit grade assessment model in an embodiment of the present invention;

[0044] Figure 4 Schematic diagram of the structure of a fruit grading system based on FPGA in an embodiment of the present invention;

[0045] Figure 5Schematic diagram of the functional structure of a fruit grading system based on FPGA in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0047] In view of the problems existing in the prior art, the present invention provides a fruit grading method and system based on FPGA, wherein a fruit grading method based on FPGA is mainly the software design part, and a fruit grading system based on FPGA is mainly the design and implementation part of the software on the hardware. The present invention accurately processes and analyzes fruit images, and identifies the grade of apples on the assembly line according to the grade classification standard. The technical solution of the present invention is: the hardware platform FPGA interacts with the image acquisition device to capture the moving apples on the assembly line, and the collected images are transmitted to the hardware platform for recognition and processing: first, the image data stream is divided into two paths after the quality is improved by image processing technology, one path is to complete the grading of color standards by image processing technology, and the other path is to complete the grading of fruit shape and defect standards by deep learning algorithm. Finally, the judgment results of the two paths of data are summarized to obtain the final grading result, and the graded display is completed in a specific area on the display device. Combined with Figure 1 , which may specifically include the following steps:

[0048] Step 1: Collect images of the fruit to be evaluated;

[0049] Step 2: Recognize the image of the fruit to be evaluated based on FPGA and determine the first grade of the fruit to be evaluated;

[0050] Step 2.1: preprocessing the image of the fruit to be evaluated;

[0051] Step 2.1.1: Calculate the grayscale value of the pixel in the image of the fruit to be evaluated to obtain a grayscale image.

[0052] Specifically, the present invention aims at the defect that FPGA cannot perform decimal operations and in accordance with the principle of avoiding waste of resources, and adopts the method of shifting the binary weight coefficient left by 8 bits (equivalent to multiplying by 256) to convert the coefficient into an integer for operation. According to the empirical value of the sensitivity of the human eye to the R, G, and B components, the obtained weight parameters are 0.299, 0.587, and 0.114, respectively. Based on this, for each pixel in the image of the fruit to be evaluated, the gray value is calculated by the following formula:

[0053] Gray(i,j)=0.299*R(i,j)+0.587*G(i,j)+0.114*B(i,j)

[0054] Among them, (i, j) is the coordinate of the pixel point in the image of the fruit to be evaluated, R(i, j) is the red component of the pixel point in the image of the fruit to be evaluated, G(i, j) is the green component of the pixel point in the image of the fruit to be evaluated, B(i, j) is the blue component of the pixel point in the image of the fruit to be evaluated, and Gray(i, j) is the grayscale value of the pixel point in the image of the fruit to be evaluated.

[0055] Among them, when calculating in FPGA, Gray is grayscale data, R is the red component; G is the green component; B is the blue component.

[0056] In order to avoid timing problems caused by too much logic, the present invention uses the FPGA pipeline design method to sacrifice a certain area in exchange for faster processing speed. Specifically, the grayscale process adopts a three-stage pipeline design: the first stage is responsible for multiplication, the second stage performs addition, and the third stage performs shift restoration. Since the entire grayscale process consumes three clock cycles, the flag representing the valid data needs to be delayed by three cycles to correspond to the data.

[0057] Step 2.1.2: The grayscale image is processed by a median filtering noise reduction algorithm to obtain a first image. Specifically, a 3*3 sliding window is defined, and the sliding window is used to slide in the grayscale image. For the grayscale values ​​of the pixels in 3 rows and 3 columns in the sliding window, the maximum grayscale value of each row, the middle grayscale value of each row, and the minimum grayscale value of each row are determined, and the minimum value of the three maximum values, the middle value of the three middle values, and the maximum value of the three minimum values ​​are obtained, and then the minimum value of the three maximum values, the middle value of the three middle values, and the minimum value of the three minimum values ​​are obtained. The intermediate value between the maximum and minimum values ​​is taken as the first intermediate value, and it is judged whether the pixels in the 2nd row and 2nd column of the sliding window are in the contour position of the fruit to be evaluated; if the pixels in the 2nd row and 2nd column of the sliding window are in the contour position of the fruit to be evaluated, the grayscale values ​​of the pixels in the 2nd row and 2nd column of the sliding window are kept unchanged; if the pixels in the 2nd row and 2nd column of the sliding window are not in the contour position of the fruit to be evaluated, the grayscale values ​​of the pixels in the 2nd row and 2nd column of the sliding window are replaced with the first intermediate value, and the sliding window is moved so that each pixel in the grayscaled image is subjected to the above operation, and after all pixels are subjected to the above operation, a first image is obtained;

[0058] The definition of a 3*3 sliding window is as follows:

[0059] Generating a 3×3 window requires caching three rows of pixels. During the design process, a first-in-first-out FIFO memory is used to cache two rows of pixels, and the row caches are connected in a daisy-chain form, that is, the output of the current row cache is connected to the input of the next row cache, and the third row of pixels appears as a real-time input.

[0060] It should be noted that, for the pixels at the boundary of the grayscale image, for example, the pixels in the first row, the first column, the last row and the last column of the grayscale image, when performing the above operations, since the above pixels cannot be located at the second row and second column of the sliding window, the pixels on the boundary are copied to the outside of the image, and the rows and columns within the boundary are mirrored outside the image, and the pixels in the first row are copied to the 0th row, so that the pixels in the first row can be located at the second row and second column of the sliding window.

[0061] Step 2.1.3: The first image is processed by an improved Sobel sharpening algorithm to obtain a second image. Since the energy of the image is mainly concentrated in the low-frequency area, while the noise and edge information are more present in the high-frequency area. The high-frequency components are enhanced by performing differential operations on the image to make the image clearer. The Sobel sharpening algorithm is an image processing technology based on the Sobel operator, which is used to enhance the edges and details of the image. The idea of ​​the algorithm is to first average the image, then perform differential operations, and finally calculate the gradient. Through the averaging operation, the low-frequency area of ​​the image can be filtered and the noise in the high-frequency area can be effectively suppressed. The Sobel operator is relatively less sensitive to noise. The calculation principle of the Sobel sharpening algorithm.

[0062] For the implementation of the Sobel algorithm, the present invention also uses a FIFO data buffer to implement sliding window processing. The 3x3 window only needs two FIFO buffers to cache the data of the previous row and the previous row, and uses the currently transmitted data as the third row, thereby completing the arrangement of three rows of data.

[0063] Specifically, for the grayscale value of the first image in the 3*3 sliding window, calculation is performed according to 8 preset groups of different weight parameters to obtain a second intermediate value, each group of weight parameters is a 3*3 matrix, and the grayscale values ​​of the pixels in 2 rows and 2 columns in the sliding window are replaced by the second intermediate value. The sliding window is moved so that each pixel in the first image is subjected to the above operation. After all pixels are subjected to the above operation, the second image is obtained;

[0064] The present invention uses 8-direction convolution factors for convolution, that is, 8 groups of different weight parameters, to make the edge detection effect of the image more refined and the characteristics of the image more complete. On the basis of the original direction, 45°, 135°, 180°, 225°, 270° and 315° are added, and finally the templates of eight directions are obtained as the weight parameters. The 8 groups of different weight parameters are referenced Figure 2The present invention utilizes the parallel computing characteristics of FPGA to simultaneously complete the gradient calculation in 8 directions, and the time required is the same as that of the original Sobel algorithm, thereby meeting the requirements of the real-time system while optimizing the algorithm.

[0065] The data processed by the image denoising algorithm is first stored in two FIFO memories for buffering. A 3x3 window is constructed by delaying row by row, so that each pixel can form a 3x3 window with its 8 adjacent pixels. Unlike the median filter, the sharpening filter contains weight parameters, which are operated on the corresponding 9 pixels in the window. These pixels are represented as matrix_p00~matrix_p22 in the figure. Due to the characteristics of sequential logic, the result of each calculation will be output in the next cycle. Finally, this paper adopts the stepwise approximation method, that is, by continuously comparing data, the square root operation of the square of the calculation results in the horizontal and vertical directions G_square_data is performed to obtain the final result G_data.

[0066] Step 2.1.4: Binarize the second image to obtain a preprocessed image.

[0067] Step 2.2: The pre-processed image is processed by a pre-trained fruit grading model to obtain a first grade of the fruit to be assessed, wherein the first grade represents the grade of the contour of the fruit to be assessed, for example,

[0068] The fruit grade assessment model is obtained by model training based on training samples, wherein the training samples include input samples and output samples, the input samples are fruit images, and the output samples are contour grades of the fruit images;

[0069] Among them, the contour levels in the output samples are manually labeled, specifically using the following standard annotations:

[0070] Superior quality, the fruit shape has the characteristics of this product and there are no defects in the fruit shape.

[0071] First-class product: slight defects in fruit shape are allowed, but the fruit shape must be flawless.

[0072] For second-class products, defects in fruit shape are allowed, but the basic characteristics of the product are still maintained. Deformed fruits are not allowed, and the area of ​​damage from thorns (broken skin scratches), hail, insect damage, brown rust, etc. must be less than 10%.

[0073] Substandard products have deformed fruit shapes, and the area of ​​damage caused by thorns (broken skin and scratches), hail, insect damage, brown rust, etc. is more than 10%.

[0074] Among them, with respect to the outline of the fruit, superior quality, first quality, second quality and substandard quality correspond to the first grade.

[0075] Since the recognition accuracy and structural stability of the Lenet-5 network have been widely verified, and the application scenario of the network is similar to that of the present invention, the fruit grade assessment model in the present invention is improved based on the Lenet-5 network, and specifically includes the following parts:

[0076] (1) Convolutional layer: The convolutional layer in the original network structure is divided into two types, one is 6 5x5 convolution kernels, and the other is 16 5x5 convolution kernels. For the first form of convolutional layer, first of all, in terms of the number of channels, considering that the features of apple images are more complex than handwritten numbers, in order to extract features to a greater extent, the number of channels is expanded to 30. Secondly, in terms of the size of the convolution kernel, from the perspective of hardware platform implementation, the 5x5 convolution kernel is replaced with two 3x3 convolution kernels. Under the premise of not changing the receptive field of the convolution kernel, the calculation scale is reduced and the feature extraction capability is improved. In the case of 224x224 input of the present invention, the amount of multiplication and addition calculations can be reduced by 1.3 times.

[0077] (2) Activation function: In order to avoid the gradient vanishing problem, nonlinear relationships are introduced to process complex data. The present invention uses the LeakyRelu function to replace the Sigmoid function in the original Lenet-5 network. The range of input value mapping is expanded from [0, 1] to [-∞, +∞]. The most important thing is that when the network is deployed on the hardware platform, the LeakyRelu function will enhance the execution effect of signals greater than a certain value.

[0078] (3) Pooling layer and fully connected layer: The present invention changes the number of neurons in the output layer of the fully connected layer based on the original model. The final network structure includes four convolutional layers, of which the C1 and C2 convolutional layers include 30 convolutional kernels respectively, and the C3 and C4 convolutional layers include 16 convolutional kernels. In addition, the model also includes two maximum pooling layers S1 and S2, which are located after the activation layers of the C2 and C4 convolutional layers respectively. Finally, three fully connected layers are used for classification, which contain 120, 84 and 4 neurons respectively.

[0079] Therefore, the improved model, that is, the fruit grade assessment model includes a convolutional layer, an activation layer, a pooling layer and a fully connected layer. The convolutional layer includes C1, C2, C3, and C4. The C1 contains 30 convolution kernels, and the number of channels of C1 is 30. The C2 contains 30 convolution kernels, and the number of channels of C2 is 30. The C3 contains 16 convolution kernels, and the number of channels of C3 is 16. The C4 contains 16 convolution kernels, and the number of channels of C4 is 16. The function in the activation layer is the LeakyRelu function. The pooling layer includes S1 and S2. The fully connected layer includes a first fully connected layer, a second fully connected layer and a third fully connected layer. The first fully connected layer includes 120 neurons, the second fully connected layer includes 84 neurons, and the third fully connected layer includes 4 neurons.

[0080] It should be noted that the training samples of the fruit grading model in this section are for Fuji apples, and the parameters of each layer of the above-mentioned fruit grading model are also for Fuji apples. Therefore, the above-mentioned parameters have a high accuracy rate for Fuji apple grading. For the grading of other fruits, it may be necessary to adjust the training samples and train again.

[0081] Based on this, the fruit grade assessment model was tested, specifically:

[0082] The host computer experimental environment of the present invention is carried out in Matlab, including the construction, training and testing of the neural network. In addition, since the final network model needs to be deployed on the FPGA platform, it is necessary to store the parameters on the hardware platform in advance. After many experiments, a set of hyperparameters with relatively good grading effect performance was finally determined. The determined training parameters include the total number of training rounds (Epoch) of 50; the maximum number of iterations (Max_iteration) of 6000; the number of samples per training (Batch_size) of 64; the learning rate (Learning_rate) of 0.0001; and the optimizer (Optimizer) of Sgdm. Matlab is needed to convert the trained network parameters according to the requirements of the hardware platform. The network model before and after the improvement is trained and verified using a homemade apple data set, and the grading effect of the improved network model is evaluated by comparing the accuracy and loss performance.

[0083] The improved network model improves the classification accuracy of apple images by 3%, and the validation set loss and training loss of the improved network have better fitting. In order to further evaluate the performance of the network model, the present invention uses a confusion matrix for detailed analysis.

[0084] The horizontal axis represents the predicted classification of apples (Predict class), and the vertical axis represents the actual classification of apples (Actualclass). The test results are as follows: among the 25 samples, the predicted results of premium apples are consistent with the actual results in 22, with an accuracy of 88%; the predicted results of first class apples are consistent with 21, with an accuracy of 84%; the predicted results of second class apples are consistent with 23, with an accuracy of 92%; the predicted results of substandard apples are consistent with 22, with an accuracy of 88%. The average accuracy reaches 88%, which is basically consistent with the above network verification results.

[0085] Based on this, combined Figure 3 , step 2.2 specifically includes:

[0086] The preprocessed image is input into C1, activation layer, C2, activation layer, S1, C3, activation layer, C4, activation layer, S2, the first fully connected layer, the second fully connected layer and the third fully connected layer in sequence to obtain the first grade of the fruit to be evaluated.

[0087] Specifically, the preprocessed image is input into C1, and a first sliding window is constructed according to the size of the convolution kernel of C1. Based on the first sliding window, a convolution operation is performed on the preprocessed image in each channel of C1 according to the weight of the channel, and then all images output by C1 are processed by the activation layer LeakyRelu function, and then the image processed by the activation layer function is input into C2. According to the size of the convolution kernel of C2, a second sliding window is constructed, and based on the second sliding window, a convolution operation is performed on the image processed by the activation layer function in each channel of C2 according to the weight of the channel, and then all images output by C2 are processed by the activation layer LeakyRelu function, and then the image processed by the activation layer function is input into S1, and according to the 2*2 sliding window in S1, slide in each image processed by the activation layer function, slide in each image processed by the activation layer function through the 2*2 sliding window, and select the maximum value in the 2*2 sliding window, and then all images output by S1 are input into C3, and a third sliding window is constructed according to the size of the convolution kernel of C3, and based on the third sliding window, a third sliding window is constructed. A sliding window is constructed, and at the same time, in each channel of C3, a convolution operation is performed on all images output by S1 according to the weight of the channel, and then all images output by C3 are processed by the activation layer LeakyRelu function, and then the images processed by the activation layer function are input into C4, and according to the size of the convolution kernel of C4, a fourth sliding window is constructed, and based on the fourth sliding window, at the same time, in each channel of the convolution layer C4, a convolution operation is performed on all images processed by the activation layer function according to the weight of the channel, and then all images output by C4 are processed by the activation layer LeakyRelu. Function is processed, and then the image processed by the activation layer function is input into S2. According to the 2*2 sliding window in S2, the 2*2 sliding window is slid in each image processed by the activation layer function, and the maximum value in the 2*2 sliding window is selected, and then all the images output by S2 are passed through the 120 neurons in the first fully connected layer to obtain 120 data, and the 120 data are input into to obtain 84 data, and the 84 data are input into the 4 neurons in the third fully connected layer to obtain 4 data, thus obtaining the first grade of the fruit to be evaluated.

[0088] When implementing convolutional neural networks on a software platform, floating-point operations are mainly used. In order to ensure the accuracy and stability of network training, 32-bit floating-point precision or 64-bit floating-point precision is usually used to train the weight parameters in the network. However, high-precision floating-point representation means that memory consumption is large. At the same time, as the depth of the network increases, the number of weight parameters grows geometrically, which greatly slows down the speed of model training. When deploying the network on an FPGA hardware platform, using fixed-point integer data has a higher cost-effectiveness. Based on this, the fruit grading model in the present invention is deployed on FPGA hardware, and fixed-point quantization operations are performed on weight parameters. Specifically, the following formula is used to convert relevant parameters such as weights into fixed-point numbers and store them in the FPGA platform.

[0089] Fixed_data=round(float_data*2 s ),s∈Z

[0090] Among them, Fixed_data is the fixed-point data of the weight parameter, round(x) is rounding, s is the scaling factor, Z represents an integer, and float_data is the floating-point data of the weight parameter.

[0091] Since the present invention is implemented on FPGA, in the software part, the image processing program and the image recognition program are both written into the FPGA using the hardware description language VHDLVerilog, wherein the image processing program is used to preprocess the image data captured by the image acquisition device and complete the grading according to the color standard. The image recognition program is used to analyze the preprocessed image and extract the target fruit shape and defect features. Therefore, when programming the fruit grade assessment model of the present invention, it is necessary to further design the convolution module corresponding to the convolution layer, the pooling module corresponding to the pooling layer, and the fully connected layer module corresponding to the fully connected layer, specifically:

[0092] (1) The convolution module design and implementation are as follows:

[0093] The traditional software method of implementing convolution requires multiple layers of loops, which involves a large number of parallel operations. Therefore, the present invention takes full advantage of the hardware parallelism based on the computational characteristics of the convolution layer and divides the convolution module into two parts: a cache module and a convolution calculation module.

[0094] The cache module consists of two parts: data cache Data_ram and weight cache Param_ram. It is mainly implemented using on-chip storage resources. In addition to the memory itself, the module also needs to design corresponding read and write control programs. After a frame of image data is cached, the completion flag signal is set to a high level and passed to the convolution calculation module. The convolution calculation module sends a parameter fetch command to Param_ram and a data read command to Data_ram at the same time. Since the weight parameters read in sequence are much smaller than the image data, the 3x3 window only requires 3 cycles. Before the image data stabilizes, the weight parameters need to be maintained until the image data and the weights are completely corresponding to each other before the convolution calculation is performed. Repeat the above steps until all channel calculations are completed.

[0095] The convolution calculation module is the key to the hardware implementation of the entire convolution layer. First, according to the size of the convolution kernel, the calculation module reads data commands from Data_ram, including read address data_rd_addr and read data row_cnt. Data_ram inputs image data and builds a window template through appropriate timing control. To obtain the first three rows of data, it is necessary to set a delay of 3 cycles to ensure that the first three elements of a row of data are calculated in the same cycle. Such a design can effectively support the construction of window templates and parallel processing of data. Secondly, the convolution calculation module reads the weight parameters from Param_rom through the parameter read address param_rd_addr and the read parameter command conv_cnt.

[0096] (2) The design and implementation of the pooling module are as follows:

[0097] The present invention uses 2×2 maximum pooling to maximize the pixel features by selecting the most characteristic pixels for output. The implementation of the pooling module is carried out in three steps. First, the data in the same industry is compared. The input data processed by the previous layer is counted in rows and columns. Considering that this article uses the maximum pooling of the window, for a single window, when the first row count is an odd number and the column count is an even number, it proves that the data entering the module is at least two, which meets the pooling requirements. Secondly, the input original data is delayed by one cycle to ensure that the first data of the first row corresponds to the second data. Next, a FIFO is called as a data cache, and the maximum value of the first data and the second data is selected as the input fifo_wr_data of the FIFO. The FIFO caches the maximum values ​​of all pairwise data separated by 2 in a row.

[0098] (3) The design and implementation of the fully connected module are as follows:

[0099] The fully connected layer is located at the end of the network structure. It is responsible for converting the original neurons arranged into a two-dimensional matrix into a one-dimensional matrix. Due to the large number of weights, in order to save hardware resources, this paper places the fully connected layer in the PS (Processing System) part, and is designed and deployed by ARM. All the parameters involved in the fully connected layer are stored in the DDR3 memory. During the calculation process, the weight parameters of the specified address are read and multiplied and added with the data of the corresponding address in the DDR3 transmitted by the PL (Programmable Logic) part, thereby completing the calculation process of the fully connected layer.

[0100] Step 3: According to the RGB of each pixel in the image of the fruit to be evaluated, the hue value, lightness value and saturation value of each pixel are calculated to determine the target color of the fruit to be evaluated; according to the threshold interval of the hue value, the threshold interval of the lightness value and the threshold interval of the saturation value preset for the target color, and the hue value, lightness value and saturation value of each pixel in the image of the fruit to be evaluated, it is determined whether each pixel in the image of the fruit to be evaluated belongs to the target color; according to the ratio of the number of pixels belonging to the target color in the fruit to be evaluated to the number of all pixels in the image of the fruit to be evaluated, the second grade of the fruit to be evaluated is determined;

[0101] It should be noted that the target color is determined based on the fruit to be evaluated. For example, if the fruit to be evaluated is Fuji apple, the target color is red. If the fruit to be evaluated is green mango, the target color is green. The present invention is for identifying Fuji apples, so the target color is red. In the following specific implementation description, the target color is red.

[0102] Among them, the RGB color space accurately controls the mixing of the three primary colors of red, green and blue by adjusting the intensity of each channel to obtain the desired color. In contrast, the HSV color space is more user-friendly and can better reflect human subjective feelings about color. This color space is mainly composed of three dimensions: hue (H), saturation (S), and lightness (V). The desired color can be more easily selected and controlled by adjusting these three dimensions. This paper determines the coloring degree of red by the hue component. Since the red range in the hue is 0 to 10 and 156 to 180, the extraction of color features is to count the proportion of pixels within this range and classify them according to the classification standard. Therefore, in the classification of color conditions in the present invention, HSV is more suitable.

[0103] Based on this, in step 3, the hue value, lightness value and saturation value of each pixel are calculated according to the RGB of each pixel in the image of the fruit to be evaluated, and the target color of the fruit to be evaluated is determined, which specifically includes:

[0104] According to the RGB of each pixel, the maximum value and the minimum value of the R value, G value, and B value are obtained. According to the maximum value and the minimum value, the hue value, lightness value, and saturation value of each pixel are calculated, which is specifically achieved through the following formula:

[0105]

[0106] V=max

[0107] Among them, H is the hue value of the pixel, S is the saturation value of the pixel, V is the lightness value of the pixel, G is the G value, B is the B value, R is the R value, min is the minimum value among the R value, G value, and B value, and max is the maximum value among the R value, G value, and B value.

[0108] When implemented in FPGA, in the implementation process of the color space conversion algorithm, it is preferred to obtain the maximum value (Max) and minimum value (Min) of the RGB three-channel components in the first cycle, and first complete the calculation of brightness and saturation, so as to leave enough time for the solution process of hue (H). In order to improve efficiency, we choose to use combinational logic to complete the difference operation of the maximum and minimum values ​​in the first cycle. From the above formula, it can be seen that the difference operation between the RGB three-channel components in the final formula needs to be multiplied by 60, so in the first cycle, the extracted RGB three-channel components are multiplied by 60 respectively to keep the result consistent with the final multiplication by 60. Since the size relationship of the RGB three-channel components determines different calculation methods, in the second cycle, it is necessary to first determine the size relationship of the three-channel components and determine the value of the maximum channel component to select the corresponding difference operation formula and the final calculation formula. Finally, all calculations are completed in the third cycle. In this way, the system can implement the color space conversion algorithm using only three clock cycles, which greatly improves the execution speed of the algorithm. After the conversion is completed, the pixels are counted according to the range of red pixels.

[0109] After conversion, the standard obtained according to the color distribution is divided into superior products, first-class products, second-class products and substandard products. The area occupied by red or red stripes is more than 88% as superior products; the area occupied by red or red stripes is between 78% and 88% as first-class products; the area occupied by red or red stripes is between 53% and 78% as second-class products; and the area occupied by less than 53% is substandard products. Among them, for the color of Fuji apples, superior products, first-class products, second-class products and substandard products correspond to the second grade.

[0110] Step 4: In a preset grading table, according to the first grade and the second grade, determine the target grade of the fruit to be assessed;

[0111] In specific implementation, the present invention can set target levels for different first and second levels. For example, if the first level is 1 and the second level is 1, the target level is 1; if the first level is 1 and the second level is 3, the target level is 2. Based on this list, the rows represent the 4 levels of the first level, the columns represent the 4 levels of the second level, and the corresponding target levels, thereby obtaining a rating table. Based on this, after obtaining the first and second levels, the present invention can search in the rating table to determine the target level of the fruit to be evaluated.

[0112] Step 5: Output the image of the fruit to be assessed, the image processed by the fruit grading model, and the target grade of the fruit to be assessed.

[0113] The calculation contents in step 2 to step 4 are all calculated in FPGA by means of parallel operation and pipeline operation.

[0114] The present invention also provides a fruit grading system based on FPGA, which is used to implement a fruit grading method based on FPGA. Figure 4 , including image acquisition sensor, FPGA and display;

[0115] The image acquisition sensor is used to acquire images of the fruit to be evaluated, and transmit the acquired images of the fruit to be evaluated to the FPGA;

[0116] The image data acquisition work in the present invention is completed by hardware driver and software driver together.

[0117] (1) The hardware driver uses a low-power CMOS image sensor from OmniVision and an adapter board. The CMOS image sensor has 5 million pixels. In the specific implementation, the CMOS image sensor is an OV5640 camera and the adapter board is a PMOD-5M acquisition component. The present invention collects apple images with a resolution of 640x480, and the maximum frame rate that the sensor can achieve is 90fps.

[0118] (2) The implementation of software driver is divided into three steps:

[0119] Step 1: The present invention writes a clock power-on initialization program to provide clock and reset signals for image acquisition, specifically:

[0120] In the initialization module, Zedboard controls the power-down enable (PWDN), reset (RESETB) and system input clock (XVCLK). After the chip pin voltage (DOAVDD) and analog voltage (AVDD) are pulled high for at least 6ms, PWDN is pulled low and maintained. After PWDN is pulled low for at least 1ms, the low-level reset (RESETB) is pulled high, and RESETB is ensured to be pulled low for at least 2ms before AVDD is pulled low.

[0121] Step 2: Implement data transmission within the data collection work by writing the I2C transmission protocol. The complete I2C protocol consists of two parts, specifically:

[0122] (1) The I2C data write sequence process begins with the host sending a start signal (START), followed by sending the slave device address to select the slave device with which to communicate. If the slave device successfully receives the command and is ready to communicate, it will send an acknowledgement bit (ACK), otherwise it will not respond. After receiving the acknowledgement bit, the host sends data to the slave device. The master device sends data to the data line SDA in sequence when the clock line SCL is at a low level. Finally, after the slave device completes reception, it sends an acknowledgement bit (ACK), indicating that it is ready to receive the next byte of data. After writing all the data, the host sends a stop condition to indicate the end of a data write.

[0123] (2) The entire process of I2C read data timing can be divided into two parts. In the first part, the host needs to send a write data instruction before sending a read data instruction. After the slave responds, the host sends a start signal again. Only then does the data reading process really begin. Similar to writing data, the host sends the slave device address and read flag command. After the slave responds, the slave device sends data to the SDA line in sequence. After the host reads the data, the host does not respond, which means that the data receiving process is completed.

[0124] Step 3: Write the image data acquisition module, and sample the image sensor according to the relevant instructions of the register, and read the acquired image data row by row and column by column. The output image data exists in the form of 16-bit data of RGB565. However, only 8 bits of data can be obtained each time, so it is necessary to splice two consecutive bytes of data to obtain complete 16-bit data. The first byte can be used as the high 8-bit data and the second byte as the low 8-bit data. The complete 16-bit RGB565 data can be obtained by combining the two bytes.

[0125] The FPGA is used to receive an image of the fruit to be assessed, implement the contents of step 2 to step 4 in claim 1, and transmit the image processed by the fruit grade assessment model and the target grade of the fruit to be assessed to the display;

[0126] Among them, the size of the FPGA platform in this embodiment is a flat board of 6.3 inches × 5.3 inches;

[0127] Combined with Figure 4 ,FPGA ​​includes an image driving module, an image processing module, a convolutional neural network grading module, a system main control module and an image display driving module;

[0128] The system main control module is the core control part of the entire system. It is mainly used to drive the hardware components to realize the corresponding functions through the processor system on the FPGA platform. The image driver module, image processing module, convolutional neural network grading module and image display driver module are all written in the PL part. The whole system is put into operation by writing some instructions in the ARM of the PS part. In the specific implementation, the ARM model is Cortex-A9.

[0129] The image driving module is used to send instructions to the image acquisition sensor, driving the image acquisition sensor to transmit the collected fruit image to be evaluated; specifically, the image acquisition sensor transmits the fruit image to be evaluated to the image driving module, and the image driving module transmits the fruit image to be evaluated to the image processing module;

[0130] The image processing module is used to pre-process the image of the fruit to be assessed and determine the second grade of the fruit to be assessed;

[0131] The convolutional neural network grading module is used to process the fruit grading model trained in advance to obtain the first grade part of the fruit to be assessed;

[0132] The image display driver module is used to implement the configuration of some displays, extracting the processed data from the DDR storage controller and transmitting it to the display through the protocol;

[0133] The display is used to output and display the image of the fruit to be assessed, the image processed by the fruit grade assessment model and the target grade of the fruit to be assessed. The display adopts an LCD liquid crystal display with a resolution of 1440×900.

[0134] When outputting the image of the fruit to be assessed, the image processed by the fruit grade assessment model and the target grade of the fruit to be assessed on the display, the present invention includes the following designs based on the actual use of the display:

[0135] (1) Display path implementation

[0136] Further, the following steps are included:

[0137] Further, the step (11) is specifically as follows:

[0138] VGA monitors display images in a progressive scanning mode, scanning the pixels that make up the image from top to bottom and from left to right on the display screen under the horizontal field synchronization signal. A complete line scan cycle includes the horizontal synchronization pulse signal (HSync), the back porch (BackPorch), the left border (left Border), the valid image (ValidVideo), the right border (RightBorder), and the front porch (Front Porch). Under the timing standard of 640x480@60Hz, a line scan cycle includes 800 pixels, of which the synchronization signal occupies 96 pixels, the back porch occupies 40 pixels, the left border occupies 8 pixels, the valid image occupies 640 pixels, the right border occupies 8 pixels, and the front porch occupies 8 pixels. The parameters and processes of the field scan cycle and the line scan cycle are different. The field scan cycle is defined as the refresh time of a complete image, including the pixel scanning of multiple lines, as well as the time of the upper and lower borders, the front porch and the back porch. During the field scanning cycle, the field synchronization signal occupies 2 pixels, the back edge occupies 25 pixels, the upper and lower borders each occupy 8 pixels, the effective image occupies 480 pixels, and the front shoulder occupies 2 pixels.

[0139] (12) Display driver design

[0140] Further, the step (12) is specifically as follows:

[0141] In the system design, the display driver module not only needs to display the collected image data in real time, but also needs to design the corresponding screen display framework to display the apple detection image, the corresponding level of apple image and the specific grade in real time. Considering the size of apples, the system designs a 224x224 area size to display the detection image area and the image of the same grade. Specifically, the row and column area ranges of the image of the fruit to be evaluated are set to [100,323], which means from the 100th row to the 323rd row, and from the 100th column to the 323rd column. The row range of the image processed by the fruit grade evaluation model is [0,223], which means from the 0th row to the 223rd row; the column range is [400,623], which means from the 400th column to the 623rd column. The original size of the image of the target grade of the fruit to be evaluated is 100x100, so when displayed on the screen, the area of ​​the same size is selected for display. The specific row range is [0,99], which means from row 0 to row 99; the column range is [100,199], which means from column 100 to column 199. Since the target grade image of the fruit to be assessed and the image processed by the fruit grade assessment model are not generated in real time, it is necessary to use 4 memories with a bit width of 1 and a bit depth of 10000 to store digital images 1 to 4 in advance, and use 4 memories with a bit width of 1 and a bit depth of 50176 to store images of the same grade. The display driver module can be called according to the required grading level.

[0142] (2) Overall system construction

[0143] Based on this, a program can be programmed according to the design of the display part to achieve display on the display. Specifically, the image of the fruit to be evaluated, the image processed by the fruit grade evaluation model and the target grade of the fruit to be evaluated can be output and displayed according to the range displayed on the display in (12).

[0144] Among them, the present invention provides a fruit grading system based on FPGA, combined with Figure 5 From the perspective of functional implementation, it can be divided into image acquisition module, image processing module, image grading module and image display module. The specific implementation contents are as follows:

[0145] Image acquisition module: The system uses image sensors to collect images of Fuji series apples of different grades and angles in real time to obtain images of the fruits to be evaluated, thereby providing high-quality input data.

[0146] Image processing module: used to realize the preprocessing of the fruit images to be evaluated in the previous article, and to determine the second level of the fruit images to be evaluated; this module uses image processing algorithms to eliminate noise interference and enhance feature information to improve image quality, highlight image features, and at the same time achieve accurate grading of color standards.

[0147] Image grading module: It is used to implement the processing of the pre-trained fruit grading model in the previous article to obtain the first grade of the fruit to be assessed. This module uses a grading algorithm that combines image processing and deep learning to perform feature extraction and image grading to achieve accurate and efficient processing of the automatic grading task of apples.

[0148] Image display module: It is used to realize the output display of the image of the fruit to be evaluated, the image processed by the fruit grading model and the target grade of the fruit to be evaluated. This module can display the grading results to the user in real time through the LCD. The interface is required to be intuitive and clear and able to effectively display the grading information.

[0149] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A fruit grading method based on FPGA, characterized in that: include: Step 1: Collect images of the fruit to be evaluated; Step 2: Recognize the image of the fruit to be evaluated based on FPGA and determine the first grade of the fruit to be evaluated; Step 2.1: preprocessing the image of the fruit to be evaluated; Step 2.2: The pre-processed image is processed by a pre-trained fruit grade assessment model to obtain a first grade of the fruit to be assessed, wherein the first grade represents the grade of the contour of the fruit to be assessed, and the fruit grade assessment model is obtained by model training based on training samples, wherein the training samples include input samples and output samples, wherein the input samples are fruit images, and the output samples are contour grades of the fruit images; Step 3: According to the RGB of each pixel in the image of the fruit to be evaluated, the hue value, lightness value and saturation value of each pixel are calculated to determine the target color of the fruit to be evaluated; according to the threshold interval of the hue value, the threshold interval of the lightness value and the threshold interval of the saturation value preset for the target color, and the hue value, lightness value and saturation value of each pixel in the image of the fruit to be evaluated, it is determined whether each pixel in the image of the fruit to be evaluated belongs to the target color; according to the ratio of the number of pixels belonging to the target color in the fruit to be evaluated to the number of all pixels in the image of the fruit to be evaluated, the second grade of the fruit to be evaluated is determined; Step 4: In a preset grading table, according to the first grade and the second grade, determine the target grade of the fruit to be assessed; Step 5: Output the image of the fruit to be assessed, the image processed by the fruit grade assessment model, and the target grade of the fruit to be assessed; The calculation contents in step 2 to step 4 are all calculated in FPGA by means of parallel operation and pipeline operation.

2. The fruit grading method based on FPGA according to claim 1, characterized in that: The fruit grade assessment model comprises a convolution layer, an activation layer, a pooling layer and a fully connected layer. The convolution layer comprises C1, C2, C3 and C4. The C1 comprises 30 convolution kernels, and the number of channels of C1 is 30. The C2 comprises 30 convolution kernels, and the number of channels of C2 is 30. The C3 comprises 16 convolution kernels, and the number of channels of C3 is 16. The C4 comprises 16 convolution kernels, and the number of channels of C4 is 16. The function in the activation layer is a LeakyRelu function. The pooling layer comprises S1 and S2. The fully connected layer comprises a first fully connected layer, a second fully connected layer and a third fully connected layer. The first fully connected layer comprises 120 neurons, the second fully connected layer comprises 84 neurons, and the third fully connected layer comprises 4 neurons. Based on this, step 2.2 specifically includes: The preprocessed image is input into C1, activation layer, C2, activation layer, S1, C3, activation layer, C4, activation layer, S2, the first fully connected layer, the second fully connected layer and the third fully connected layer in sequence to obtain the first grade of the fruit to be evaluated.

3. The fruit grading method based on FPGA according to claim 1, characterized in that: Step 2.1 specifically includes: Step 2.1.1: Calculate the grayscale value of the pixel points in the image of the fruit to be evaluated to obtain a grayscale image; Step 2.1.2: The grayscale image is processed by a median filtering noise reduction algorithm to obtain a first image. Specifically, a 3*3 sliding window is defined, and the sliding window is used to slide in the grayscale image. For the grayscale values ​​of the pixels in 3 rows and 3 columns in the sliding window, the maximum grayscale value of each row, the middle grayscale value of each row, and the minimum grayscale value of each row are determined, and the minimum value of the three maximum values, the middle value of the three middle values, and the maximum value of the three minimum values ​​are obtained, and then the minimum value of the three maximum values, the middle value of the three middle values, and the minimum value of the three minimum values ​​are obtained. The intermediate value between the maximum and minimum values ​​is taken as the first intermediate value, and it is judged whether the pixels in the 2nd row and 2nd column of the sliding window are in the contour position of the fruit to be evaluated; if the pixels in the 2nd row and 2nd column of the sliding window are in the contour position of the fruit to be evaluated, the grayscale values ​​of the pixels in the 2nd row and 2nd column of the sliding window are kept unchanged; if the pixels in the 2nd row and 2nd column of the sliding window are not in the contour position of the fruit to be evaluated, the grayscale values ​​of the pixels in the 2nd row and 2nd column of the sliding window are replaced with the first intermediate value, and the sliding window is moved so that each pixel in the grayscaled image is subjected to the above operation, and after all pixels are subjected to the above operation, a first image is obtained; Step 2.1.3: The first image is processed by an improved Sobel sharpening algorithm to obtain a second image. Specifically, the grayscale value of the first image in the 3*3 sliding window is calculated according to 8 preset groups of different weight parameters to obtain a second intermediate value, each group of weight parameters is a 3*3 matrix, and the grayscale values ​​of pixels in 2 rows and 2 columns in the sliding window are replaced by the second intermediate value. The sliding window is moved so that each pixel in the first image is subjected to the above operation. After all pixels are subjected to the above operation, the second image is obtained; Step 2.1.4: Binarize the second image to obtain a preprocessed image.

4. The fruit grading method based on FPGA according to claim 2, characterized in that: Step 2.1.1 specifically includes: For each pixel in the image of the fruit to be evaluated, the grayscale value is calculated using the following formula: Gray(i,j)=0.299*R(i,j)+0.587*G(i,j)+0.114*B(i,j) Among them, (i, j) is the coordinate of the pixel point in the image of the fruit to be evaluated, R(i, j) is the red component of the pixel point in the image of the fruit to be evaluated, G(i, j) is the green component of the pixel point in the image of the fruit to be evaluated, B(i, j) is the blue component of the pixel point in the image of the fruit to be evaluated, and Gray(i, j) is the grayscale value of the pixel point in the image of the fruit to be evaluated.

5. The fruit grading method based on FPGA according to claim 1, characterized in that: In step 3, the hue value, lightness value and saturation value of each pixel are calculated according to the RGB of each pixel in the image of the fruit to be evaluated, and the target color of the fruit to be evaluated is determined, which specifically includes: According to the RGB of each pixel, the maximum value and the minimum value of the R value, G value, and B value are obtained. According to the maximum value and the minimum value, the hue value, lightness value, and saturation value of each pixel are calculated, which is specifically achieved through the following formula: V=max Among them, H is the hue value of the pixel, S is the saturation value of the pixel, V is the lightness value of the pixel, G is the G value, B is the B value, R is the R value, min is the minimum value among the R value, G value, and B value, and max is the maximum value among the R value, G value, and B value.

6. The fruit grading method based on FPGA according to claim 1, characterized in that: The fruit grade assessment model is deployed on FPGA hardware, and the weight parameters are converted from floating-point data to fixed-point data through the following formula, which is specifically implemented by the following formula: Fixed_data=round(float_data*2 s ),s∈Z Among them, Fixed_data is the fixed-point data of the weight parameter, round(x) is rounding, s is the scaling factor, Z represents an integer, and float_data is the floating-point data of the weight parameter.

7. A fruit grading system based on FPGA, used to implement the fruit grading method based on FPGA according to claim 1, characterized in that: Includes image acquisition sensor, FPGA and display; The image acquisition sensor is used to acquire images of the fruit to be evaluated, and transmit the acquired images of the fruit to be evaluated to the FPGA; The FPGA is used to receive an image of the fruit to be assessed, implement the contents of step 2 to step 4 in claim 1, and transmit the image processed by the fruit grade assessment model and the target grade of the fruit to be assessed to the display; The display is used to output and display the image of the fruit to be assessed, the image processed by the fruit grade assessment model, and the target grade of the fruit to be assessed.

8. The FPGA-based fruit grading system according to claim 7, characterized in that: When the display outputs the image of the fruit to be evaluated, the image processed by the fruit grade evaluation model and the target grade of the fruit to be evaluated, the row and column area ranges of the image of the fruit to be evaluated on the screen of the display are set to [100,323], indicating from the 100th row to the 323rd row, and from the 100th column to the 323rd column, the row range of the image processed by the fruit grade evaluation model is [0,223], indicating from the 0th row to the 223rd row; the column range is [400,623], indicating from the 400th column to the 623rd column, the row range of the target grade of the fruit to be evaluated is [0,99], indicating from the 0th row to the 99th row; the column range is [100,199], indicating from the 100th column to the 199th column.

Images