Watermelon quality grading method based on multi-source information fusion

Abstract

The invention discloses a watermelon quality grading method based on multi-source information fusion, and belongs to the technical field of agricultural information.The method comprises the steps that firstly, a data set is obtained, and the data set comprises images of watermelons, sound and vibration signals during knocking, skin temperature data, weight data, varieties, sweetness, moisture content, hardness and quality scores; performing data preprocessing on the data set; constructing a watermelon variety identification model, a quality index prediction model and a comprehensive evaluation model, and training the three models by using the preprocessed data set; and obtaining a trained watermelon variety identification model, a quality index prediction model and a comprehensive evaluation model, and performing watermelon quality grading by using the obtained watermelon image, the sound and vibration signals during knocking, the skin temperature data and the weight data.

Description

Technical Field

[0001] The invention belongs to the field of agricultural information technology, and specifically relates to a watermelon quality grading method based on multi-source information fusion. Background Art

[0002] At present, there are many varieties of watermelons on the market with uneven quality and different economic values. Pre-sale grading of watermelon quality can enhance commodity competitiveness and promote consumer purchases of watermelons. However, how to accurately identify watermelon varieties and evaluate their quality grades has become an urgent problem to be solved.

[0003] The existing technology relies on manual experience and simple measuring tools to grade and classify during the picking and selling process, which has the problems of low recognition accuracy and low efficiency. Intelligent and mechanized grading and classification are still in the research stage. Therefore, there is an urgent need for a system that can automatically and accurately identify watermelon varieties and evaluate quality grades to improve the efficiency and accuracy of market supervision and consumer selection. Summary of the invention

[0004] In view of the above problems existing in the prior art, the present invention proposes a watermelon quality grading method based on multi-source information fusion, which has a reasonable design, solves the shortcomings of the prior art and has good effects.

[0005] A watermelon quality grading method based on multi-source information fusion comprises the following steps:

[0006] S1. Obtain a data set, including watermelon images, sound and vibration signals when tapping the watermelon, watermelon skin temperature data, weight data, variety, sweetness, moisture content, hardness, and quality score;

[0007] S2, preprocess the data set in S1;

[0008] S3, constructing a watermelon variety recognition model, a quality index prediction model and a comprehensive evaluation model, and using the preprocessed data set to train the three models respectively;

[0009] The input of the watermelon variety recognition model is the watermelon image, and the output is the watermelon variety; the input of the quality index prediction model is the watermelon image, the sound and vibration signal when the watermelon is knocked, the watermelon skin temperature data, and the weight data, and the output is the sweetness, moisture content and hardness; the input of the comprehensive evaluation model is the watermelon variety recognition result and the quality index prediction result, and the output is the quality score of the watermelon;

[0010] S4. Obtain a trained watermelon variety recognition model, a quality index prediction model and a comprehensive evaluation model, and use the obtained watermelon images, the sound and vibration signals when knocked, the skin temperature data and the weight data to grade the quality of the watermelons.

[0011] Furthermore, in S1, a camera, a dual-channel microphone, a dual-channel accelerometer, an infrared temperature sensor, a weighing sensor, a portable sugar meter, and a digital fruit hardness meter are used to collect data of different varieties of watermelons;

[0012] The camera is used to capture three images of the same watermelon at different positions;

[0013] The dual-channel microphone is used to collect dual-channel sound signals emitted when the watermelon is tapped;

[0014] The dual-channel accelerometer is used to collect dual-channel vibration signals when the watermelon is hit;

[0015] The infrared temperature sensor is used to measure the watermelon skin temperature data;

[0016] The weighing sensor is used to measure the weight data of the watermelon;

[0017] The portable sugar meter is used to measure the sugar content of watermelon pulp, and the measurement result is recorded as the sweetness data of the watermelon;

[0018] The digital fruit hardness tester is used to measure the hardness of watermelon pulp, and the measurement result is recorded as the hardness data of the watermelon;

[0019] The moisture content of watermelon was measured by drying method, which includes cutting the watermelon, weighing it, drying it to constant weight, weighing it again, and calculating the moisture content;

[0020] The watermelon quality is graded based on the variety, sweetness, moisture content and hardness of the watermelon and the set watermelon grading standards.

[0021] Furthermore, in S1, when using a dual-channel microphone to collect sound signals, two microphones are set, microphone 1 is placed near the middle between the top and the bottom of the watermelon, and microphone 2 is placed on the opposite side of microphone 1, and finally two independent time series sound signals are obtained;

[0022] When using a dual-channel accelerometer to collect vibration signals, two accelerometers are set, and accelerometer 1 and accelerometer 2 are placed on the top and bottom of the watermelon respectively, and finally two independent time series vibration signals are obtained;

[0023] When using the infrared temperature sensor to collect temperature data, the watermelon skin temperature data at four symmetrical points centered on the cross-section circle of the watermelon is measured, and finally four temperature values ​​are obtained;

[0024] When using a weighing sensor to collect weight data, the weight of the watermelon is measured three times in succession, and finally three weight values ​​are obtained.

[0025] Further, in said S1, in the acquired data set, the varieties of watermelon include Jingxin watermelon, seedless watermelon, black beauty, flower skin watermelon, yellow flesh watermelon, mini watermelon, winter melon watermelon, small jade watermelon, early spring red jade, black skin seedless watermelon, royal Malan watermelon, cucumber watermelon, snow top watermelon, wedding candy watermelon and Charleston gray;

[0026] The sweetness of watermelon ranges from 5 to 15° Brix, the moisture content is 85% to 95%, and the hardness is between 1 and 5N;

[0027] Watermelons are rated on a scale of 1-10 for quality.

[0028] Furthermore, the S2 specifically includes: resizing the three images taken of each watermelon to 224*224*3;

[0029] The time domain, frequency domain and time-frequency feature extraction are performed on the sound signals of the two channels respectively, and the first three important features of the sound signals of the two channels are selected using F value and principal component analysis respectively;

[0030] The vibration signals of the two channels are extracted in time domain, frequency domain and time-frequency features respectively, and the first three important features of the vibration signals of the two channels are selected using F value and principal component analysis respectively;

[0031] Among them, the time domain features include mean, variance, peak, root mean square value, skewness, and kurtosis; the frequency domain features include spectrum center, spectrum bandwidth, spectrum peak, spectrum roll-off point, and frequency smoothness; the time-frequency features include the time-frequency features obtained by short-time Fourier transform, Mel frequency cepstrum coefficients, and wavelet transform features; therefore, a total of 14 features are extracted from the signal of each channel;

[0032] The 4 temperature values ​​and 3 weight values ​​of each watermelon are averaged to obtain the temperature average value and the weight average value.

[0033] Furthermore, in S3, a watermelon variety recognition model is constructed based on a time-distributed convolutional neural network TimeDistributed CNN, including an input layer, a TimeDistributed layer, a convolutional block unit, a feature fusion layer and an output layer;

[0034] The input layer receives three watermelon images at different positions, each image size is (224, 224, 3), forming an RGB image with an input tensor size of (3, 224, 224, 3); the convolution operation is applied to each image using the TimeDistributed layer;

[0035] The convolution block unit is used to extract the features of each image, which includes four convolution blocks, and convolution block 1 includes 64 convolution layers with a convolution kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0036] Convolutional block 2 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0037] Convolutional block 3 includes 256 convolutional layers with a kernel size of 3*3, ReLU activation function, batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0038] Convolutional block 4 includes 512 convolutional layers with a kernel size of 3*3, ReLU activation function, batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0039] The feature fusion layer includes a Flatten layer, a fully connected layer 1, and a fully connected layer 2;

[0040] The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit, and further process them through the fully connected layer;

[0041] The fully connected layer 1 includes 512 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5;

[0042] The fully connected layer 2 includes 256 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5;

[0043] The output layer uses the Softmax activation function to output the probabilities corresponding to the 15 watermelon varieties, and the one with the highest probability is the final watermelon variety.

[0044] Further, in S3, the quality index prediction model includes an input layer, an image processing branch layer, a sound and vibration feature processing branch layer, a temperature and weight feature processing branch layer, a feature fusion layer and an output layer; the sound and vibration feature processing branch layer includes a sound feature processing branch and a vibration feature processing branch layer;

[0045] The input layer receives three watermelon images at different positions, three features corresponding to the sound signals of the two channels, three features corresponding to the vibration signals of the two channels, the average value of the temperature and the average value of the weight;

[0046] The three watermelon images in different positions are formed into an RGB image with a tensor size of (3, 224, 224, 3), which is input into the image processing branch layer. The six features corresponding to the sound signals of the two channels are input into the sound feature processing branch layer. The six features corresponding to the vibration signals of the two channels are input into the vibration feature processing branch layer. The average temperature and the average weight are input into the temperature and weight feature processing branch layer. The outputs of the image processing branch layer, the sound and vibration feature processing branch layer, and the temperature and weight feature processing branch layer are input into the feature fusion layer.

[0047] Furthermore, in S3, the image processing branch layer includes a TimeDistributed layer, five convolution blocks, and a feature fusion layer, and the feature fusion layer includes a Flatten layer, a fully connected layer 3, and a fully connected layer 4; the convolution operation is applied to each image using the TimeDistributed layer;

[0048] Convolutional block 5 includes 64 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0049] Convolutional block 6 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0050] Convolutional block 7 includes 256 convolutional layers with a kernel size of 3x3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer of size 2*2;

[0051] Convolutional block 8 includes 512 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0052] Convolutional block 9 includes a convolutional layer with 1024 convolution kernels of size 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer of size 2*2;

[0053] The Flatten layer is used to flatten and fuse the features of the three images output by the convolution block unit;

[0054] The fully connected layer 3 includes 512 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0055] The fully connected layer 4 includes 256 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0056] The sound and vibration feature processing branch layer includes a sound feature branch layer and a vibration feature branch layer, the sound feature branch layer includes a fully connected layer 5 and a fully connected layer 6, and the vibration feature branch layer includes a fully connected layer 7 and a fully connected layer 8;

[0057] The fully connected layer 5 and the fully connected layer 7 each include 128 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.3;

[0058] The fully connected layer 6 and the fully connected layer 8 each include 64 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.3;

[0059] The temperature and weight feature processing branch layer includes a fully connected layer 9, and the fully connected layer 9 includes 64 neurons and a ReLU activation function;

[0060] The feature fusion layer includes a fully connected layer 10, a fully connected layer 11 and a fully connected layer 12, wherein the fully connected layer 10 includes 512 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0061] The fully connected layer 11 includes 256 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0062] The fully connected layer 12 includes 128 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5;

[0063] The output layer includes 3 neurons and a linear activation function, which is used to output the sweetness, moisture content and hardness of the watermelon.

[0064] Further, in S3, the comprehensive evaluation model includes an input layer, a multi-layer fully connected layer and an output layer;

[0065] The input layer receives the watermelon variety recognition result and the quality index prediction result; the variety recognition result is an array consisting of 15 numbers, each number represents a watermelon variety, and the "one-hot encoding" method is used to ensure that only one number in the array is 1 at a time, and the rest are 0; the quality index prediction result is an array consisting of 3 data, which respectively represent the sweetness, moisture content and hardness of the watermelon.

[0066] The multi-layer fully connected layer includes a fully connected layer 13, a fully connected layer 14, a residual connected layer 1, a fully connected layer 15, a fully connected layer 16, a residual connected layer 2 and a fully connected layer 17;

[0067] The fully connected layer 13 includes 512 neurons, a ReLU activation function, and a Dropout layer;

[0068] The fully connected layer 14 includes 256 neurons, a ReLU activation function, and a Dropout layer;

[0069] The residual connection layer 1 includes 256 neurons and a ReLU activation function;

[0070] The fully connected layer 15 includes 128 neurons, a ReLU activation function, and a Dropout layer;

[0071] The fully connected layer 16 includes 64 neurons, a ReLU activation function, and a Dropout layer;

[0072] The residual connection layer 2 includes 64 neurons and a ReLU activation function;

[0073] The fully connected layer 17 includes 32 neurons, a ReLU activation function, and a Dropout layer;

[0074] The output layer includes 1 neuron and a linear activation function, and outputs the quality score of the watermelon.

[0075] Beneficial technical effects brought by the present invention:

[0076] 1. Improve grading accuracy: The present invention comprehensively analyzes the variety and quality of watermelons by integrating multi-source information, such as image information, sugar content data, and hardness information. Compared with the traditional manual experience grading method, the present invention greatly improves the grading accuracy and reduces the impact of human errors.

[0077] 2. Improve grading efficiency: The present invention adopts an intelligent algorithm to make the watermelon grading process more automated and efficient, significantly shortening the grading time and adapting to large-scale agricultural picking and sales needs.

[0078] 3. Quality visualization: Through multi-source information fusion technology, the present invention can provide intuitive visualization results of watermelon quality, help consumers better understand the quality information of watermelon, and make the consumption process more transparent.

[0079] 4. Assist in market supervision: The systematic and standardized grading method of the present invention provides reliable technical support for market supervision, helps to establish a unified watermelon quality grading standard, and promotes the standardized development of the market.

[0080] 5. Increase economic benefits: Through accurate quality grading, the present invention improves the market competitiveness of high-quality watermelons, promotes the increase in sales volume, and thus increases the economic benefits of producers and sellers. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A flowchart of obtaining a data set and preprocessing in the present invention;

[0082] Figure 2 A flow chart for watermelon variety identification in the present invention;

[0083] Figure 3 It is a flow chart of watermelon quality index prediction in the present invention;

[0084] Figure 4 A flow chart of the comprehensive evaluation of watermelon in the present invention; DETAILED DESCRIPTION

[0085] The specific implementation of the present invention is further described below in conjunction with specific embodiments:

[0086] A watermelon quality grading method based on multi-source information fusion comprises the following steps:

[0087] S1. Obtain a data set, including watermelon images, sound and vibration signals when tapping the watermelon, watermelon skin temperature data, weight data, variety, sweetness, moisture content, hardness, and quality score;

[0088] S2, preprocess the data set in S1;

[0089] S3, constructing a watermelon variety recognition model, a quality index prediction model and a comprehensive evaluation model, and using the preprocessed data set to train the three models respectively;

[0090] The input of the watermelon variety recognition model is the watermelon image, and the output is the variety; the input of the quality index prediction model is the watermelon image, the sound and vibration signal when the watermelon is knocked, the watermelon skin temperature data, and the weight data, and the output is the sweetness, moisture content and hardness; the input of the comprehensive evaluation model is the watermelon variety recognition result and the quality index prediction result, and the output is the quality score of the watermelon;

[0091] S4. Obtain a trained watermelon variety recognition model, a quality index prediction model and a comprehensive evaluation model, and use the obtained watermelon images, the sound and vibration signals when knocked, the skin temperature data and the weight data to grade the quality of the watermelons.

[0092] Specifically, in S1, Figure 1As shown, a camera, a dual-channel microphone, a dual-channel accelerometer, an infrared temperature sensor, a weighing sensor, a portable sugar meter, and a digital fruit hardness meter are used to collect data of different varieties of watermelons; the varieties of watermelons include Jingxin watermelon, seedless watermelon, black beauty, flower skin watermelon, yellow flesh watermelon, mini watermelon, winter melon watermelon, Xiaoyu watermelon, early spring red jade, black skin seedless watermelon, royal Malan watermelon, cucumber watermelon, snow top watermelon, wedding candy watermelon and Charleston gray;

[0093] The camera is used to take three images of the same watermelon at different positions. The camera can be mounted on a fixed bracket, or a non-fixed or handheld camera can be used to take images of the appearance of the watermelon.

[0094] The dual-channel microphone is used to collect dual-channel sound signals emitted when the watermelon is hit; when using the dual-channel microphone to collect sound signals, two microphones are set, microphone 1 is placed on one side of the watermelon, close to the middle position, and microphone 2 is placed on the opposite side of microphone 1, close to the middle position, to obtain sound signals of different paths, to help more comprehensively analyze the internal structure of the watermelon, and finally obtain two independent time series sound signals;

[0095] The dual-channel accelerometer is used to collect dual-channel vibration signals when the watermelon is hit. When using the dual-channel accelerometer to collect vibration signals, two accelerometers are set, and accelerometer 1 and accelerometer 2 are placed on the top and bottom of the watermelon respectively to capture vibration signals in different directions, which helps to more comprehensively analyze the hardness and internal porosity of the watermelon, and finally obtain two independent time series vibration signals.

[0096] The infrared temperature sensor is used to measure the temperature data of the watermelon skin. When the infrared temperature sensor is used to collect temperature data, the watermelon skin temperature data at four symmetrical points centered on the cross-section circle of the watermelon are measured, and finally four temperature values ​​are obtained.

[0097] The weighing sensor is used to measure the weight data of the watermelon; when the weighing sensor is used to collect weight data, the weight of the watermelon is measured three times in succession, and finally three weight values ​​are obtained.

[0098] The sugar content of the watermelon pulp is measured using a portable saccharimeter, and the measurement results are recorded as the sweetness data of the watermelon; the sweetness range of the watermelon is 5 to 15° Brix;

[0099] The moisture content of watermelon was measured by drying method, which includes cutting the watermelon, weighing it, drying it to constant weight, weighing it again, and calculating the moisture content; the moisture content is 85% to 95%;

[0100] The hardness of the watermelon pulp was measured using a digital fruit hardness tester, and the measurement results were recorded as the hardness data of the watermelon; the hardness ranged from 1 to 5N;

[0101] The above sweetness, moisture and hardness data are obtained by combining literature data and data of different watermelon varieties purchased in the actual market;

[0102] The watermelon quality is graded based on its variety, sweetness, moisture content and hardness according to the set watermelon grading standards, with the quality score ranging from 1 to 10.

[0103] In this embodiment, the watermelon quality score adopts a scoring system of 1-10 points, and the specific grading standards are shown in Table 1:

[0104] Table 1 Watermelon quality scoring standard

[0105]

[0106]

[0107] The grading criteria can be adjusted according to the needs of growers or sales staff and the comprehensive evaluation model can be retrained to improve the adaptability and scalability of the method.

[0108] Specifically, Figure 1 As shown, S2 specifically includes: resizing the three images of each watermelon to 224*224*3; extracting the time domain, frequency domain and time-frequency features of the sound signals of the two channels respectively, and selecting the first three important features of the sound signals of the two channels using the F value and principal component analysis method respectively;

[0109] The vibration signals of the two channels are extracted in time domain, frequency domain and time-frequency features respectively, and the first three important features of the vibration signals of the two channels are selected using F value and principal component analysis respectively;

[0110] Among them, the time domain features include mean, variance, peak value, root mean square value (RMS), skewness, and kurtosis; the frequency domain features include spectral center, spectral bandwidth, spectral peak, spectral roll-off, and spectral flatness; the time-frequency features include the time-frequency features obtained by short-time Fourier transform (STFT), Mel frequency cepstral coefficients (MFCC), and wavelet transform features; therefore, a total of 14 features are extracted from the signal of each channel;

[0111] Among them, the F value (ANOVA F-value) is used to evaluate the significance of each feature, and then the top three features with the highest F value are selected. This method does not depend on a specific model, and only needs to calculate the correlation between each feature and the target variable. Its calculation steps are: 1) Calculate the intra-group variance and inter-group variance: For each feature X, calculate the intra-group variance and inter-group variance under different categories y. Let k be the number of categories, n i is the number of samples in the i-th class, n is the total number of samples, is the mean of the i-th class, is the total mean. 2) Calculate the within-group variance, Calculate the between-group variance, 4) Calculate the F value, 5) Feature selection: Calculate and compare the F value of each feature, and then select the top three features with the highest F value as the most important features.

[0112] Principal component analysis (PCA) is used for feature selection. The steps are: 1) Standardize the data so that each feature has zero mean and unit variance. 2) Calculate the covariance matrix C of the standardized data. 3) Perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalue λ i and the eigenvector V i . 4) Sort by eigenvalue size and select the eigenvectors corresponding to the first three largest eigenvalues.

[0113] The 4 temperature values ​​and 3 weight values ​​of each watermelon are averaged to obtain the temperature average value and the weight average value.

[0114] Specifically, in S3, a watermelon variety recognition model is constructed based on a time-distributed convolutional neural network (TimeDistributed CNN), which includes an input layer, a TimeDistributed layer, a convolutional block unit, a feature fusion layer, and an output layer;

[0115] The input layer receives three watermelon images at different positions, each of which is of size (224, 224, 3), forming an RGB image with an input tensor size of (3, 224, 224, 3); the TimeDistributed layer is used to apply subsequent convolution operations to each image.

[0116] The convolution block unit is used to extract the features of each image. It includes four convolution blocks. Convolution block 1 includes 64 convolution layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2.

[0117] Convolutional block 2 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0118] Convolutional block 3 includes 256 convolutional layers with a kernel size of 3*3, ReLU activation function, batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0119] Convolutional block 4 includes 512 convolutional layers with a kernel size of 3*3, ReLU activation function, batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0120] The feature fusion layer is used to flatten the features of each image and further process them through the fully connected layer, which includes the Flatten layer, the fully connected layer 1, and the fully connected layer 2;

[0121] The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit;

[0122] The fully connected layer 1 includes 512 neurons, ReLU activation function, Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0123] The fully connected layer 2 includes 256 neurons, ReLU activation function, Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0124] The output layer is the classification layer, which uses the Softmax activation function to output the probabilities corresponding to the 15 watermelon varieties. The one with the highest probability is the final watermelon variety, such as Figure 2 shown.

[0125] The function of the TimeDistributed layer is to apply the same operation on multiple input data. In the watermelon variety recognition model, the TimeDistributed layer applies the convolution operation to each image, so that each image undergoes the same convolution processing, and then fuses these features for classification. This can effectively process multiple input data of the same type. Compared with traditional CNN, TimeDistributed CNN is more suitable for processing multiple inputs, because traditional CNN can only process a single image.

[0126] A quality index prediction model is constructed based on a multimodal neural network, including an input layer, an image processing branch layer, a sound and vibration feature processing branch layer, a temperature and weight feature processing branch layer, a feature fusion layer and an output layer; the sound and vibration feature processing branch layer includes a sound feature processing branch and a vibration feature processing branch layer;

[0127] The input layer receives three watermelon images in different positions, three features corresponding to the sound signals of the two channels, three features corresponding to the vibration signals of the two channels, the average temperature and the average weight;

[0128] Three watermelon images in different positions (i.e., RGB images of size (3, 224, 224, 3)) are input into the image processing branch layer, the six features corresponding to the sound signals of the two channels are input into the sound feature processing branch layer, the six features corresponding to the vibration signals of the two channels are input into the vibration feature processing branch layer, the average temperature and the average weight are input into the temperature and weight feature processing branch layer, and the outputs of the image processing branch layer, the sound and vibration feature processing branch layer, and the temperature and weight feature processing branch layer are jointly input into the feature fusion layer.

[0129] The image processing branch layer includes a TimeDistributed layer (using the TimeDistributed layer to apply subsequent convolution operations to each image), five convolution blocks, a Flatten layer, a fully connected layer 3, and a fully connected layer 4;

[0130] Convolutional block 5 includes 64 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0131] Convolutional block 6 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0132] Convolutional block 7 includes 256 convolutional layers with a kernel size of 3x3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer of size 2*2;

[0133] Convolutional block 8 includes 512 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2;

[0134] Convolutional block 9 includes a convolutional layer with 1024 convolution kernels of size 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer of size 2*2;

[0135] The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit;

[0136] The fully connected layer 3 includes 512 neurons, ReLU activation function, and Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0137] The fully connected layer 4 includes 256 neurons, ReLU activation function, Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0138] The sound and vibration feature processing branch layer includes a sound feature branch layer and a vibration feature branch layer, the sound feature branch layer includes a fully connected layer 5 and a fully connected layer 6, and the vibration feature branch layer includes a fully connected layer 7 and a fully connected layer 8;

[0139] Both the fully connected layer 5 and the fully connected layer 7 include 128 neurons, ReLU activation function, and Dropout layer, and the dropout rate of the Dropout layer is 0.3;

[0140] The fully connected layer 6 and the fully connected layer 8 both include 64 neurons, ReLU activation function, and Dropout layer, and the dropout rate of the Dropout layer is 0.3;

[0141] The temperature and weight feature processing branch layer includes a fully connected layer 9, which includes 64 neurons and a ReLU activation function;

[0142] The feature fusion layer includes fully connected layer 10, fully connected layer 11 and fully connected layer 12. The fully connected layer 10 includes 512 neurons, ReLU activation function, and Dropout layer. The dropout rate of the Dropout layer is 0.5.

[0143] The fully connected layer 11 includes 256 neurons, ReLU activation function, and Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0144] The fully connected layer 12 includes 128 neurons, ReLU activation function, Dropout layer, and the dropout rate of the Dropout layer is 0.5;

[0145] The output layer includes 3 neurons and a linear activation function, which is used to output the sweetness, moisture content, and hardness of the watermelon. Figure 3 shown.

[0146] The comprehensive evaluation model includes an input layer, multiple fully connected layers, and an output layer;

[0147] The input layer receives the watermelon variety recognition results (one-hot encoding) and quality index prediction results (sweetness, moisture content, hardness);

[0148] The variety recognition result is an array of 15 numbers, each number represents a watermelon variety. The "one-hot encoding" method is used to ensure that only one number in the array is 1 at a time, and the rest are 0, which indicates which category the watermelon belongs to. The quality index prediction result is an array of 3 data, which respectively represent the sweetness, moisture content and hardness of the watermelon. The multi-layer fully connected layer includes fully connected layer 13, fully connected layer 14, residual connection layer 1, fully connected layer 15, fully connected layer 16, residual connection layer 2 and fully connected layer 17;

[0149] The fully connected layer 13 includes 512 neurons, ReLU activation function, and Dropout layer;

[0150] The fully connected layer 14 includes 256 neurons, ReLU activation function, and Dropout layer;

[0151] The residual connection layer 1 includes 256 neurons and ReLU activation function;

[0152] The fully connected layer 15 includes 128 neurons, ReLU activation function, and Dropout layer;

[0153] The fully connected layer 16 includes 64 neurons, ReLU activation function, and Dropout layer;

[0154] The residual connection layer 2 includes 64 neurons and ReLU activation function;

[0155] The fully connected layer 17 includes 32 neurons, ReLU activation function, and Dropout layer;

[0156] The output layer includes 1 neuron and a linear activation function, which outputs the quality score of the watermelon, such as Figure 4 shown.

[0157] The present invention comprehensively analyzes the variety and quality of watermelons by fusing multi-source information, such as image information, sugar content data, and hardness information. Compared with the traditional manual experience grading method, the present invention greatly improves the grading accuracy and reduces the impact of human errors.

[0158] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A watermelon quality grading method based on multi-source information fusion, It is characterized in that The following steps are involved: S1. Obtain a data set, including watermelon images, sound and vibration signals when tapping the watermelon, watermelon skin temperature data, weight data, variety, sweetness, moisture content, hardness, and quality score; S2, preprocess the data set in S1; S3, construct watermelon variety recognition model, quality index prediction model and comprehensive evaluation model, and use the preprocessed data set to train the three models respectively; The input of the watermelon variety recognition model is the watermelon image, and the output is the watermelon variety; the input of the quality index prediction model is the watermelon image, the sound and vibration signal when the watermelon is knocked, the watermelon skin temperature data, and the weight data, and the output is the sweetness, moisture content and hardness; the input of the comprehensive evaluation model is the watermelon variety recognition result and the quality index prediction result, and the output is the quality score of the watermelon; S4. Obtain a trained watermelon variety recognition model, a quality index prediction model and a comprehensive evaluation model, and use the obtained watermelon images, the sound and vibration signals when knocked, the skin temperature data and the weight data to grade the quality of the watermelons.

2. A watermelon quality grading method based on multi-source information fusion according to claim 1, It is characterized in that In S1, a camera, a dual-channel microphone, a dual-channel accelerometer, an infrared temperature sensor, a weighing sensor, a portable sugar meter, and a digital fruit hardness meter are used to collect data of different varieties of watermelons; The camera is used to capture three images of the same watermelon at different positions; The dual-channel microphone is used to collect dual-channel sound signals emitted when the watermelon is tapped; The dual-channel accelerometer is used to collect dual-channel vibration signals when the watermelon is hit; The infrared temperature sensor is used to measure the watermelon skin temperature data; The weighing sensor is used to measure the weight data of the watermelon; The portable sugar meter is used to measure the sugar content of watermelon pulp, and the measurement result is recorded as the sweetness data of the watermelon; The digital fruit hardness tester is used to measure the hardness of watermelon pulp, and the measurement result is recorded as the hardness data of the watermelon; The moisture content of watermelon was measured by drying method, which includes cutting the watermelon, weighing it, drying it to constant weight, and weighing it again to calculate the moisture content; The watermelon quality is graded based on the variety, sweetness, moisture content and hardness of the watermelon and the set watermelon grading standards.

3. A watermelon quality grading method based on multi-source information fusion according to claim 2, It is characterized in that In S1, when using a dual-channel microphone to collect sound signals, two microphones are set, microphone 1 is placed near the middle between the top and bottom of the watermelon, and microphone 2 is placed on the opposite side of microphone 1, and finally two independent time series sound signals are obtained; When using a dual-channel accelerometer to collect vibration signals, two accelerometers are set, and accelerometer 1 and accelerometer 2 are placed on the top and bottom of the watermelon respectively, and finally two independent time series vibration signals are obtained; When using the infrared temperature sensor to collect temperature data, the watermelon skin temperature data at four symmetrical points centered on the cross-section circle of the watermelon is measured, and finally four temperature values ​​are obtained; When using a weighing sensor to collect weight data, the weight of the watermelon is measured three times in succession, and finally three weight values ​​are obtained.

4. A watermelon quality grading method based on multi-source information fusion according to claim 3, It is characterized in that In said S1, in the acquired data set, the varieties of watermelon include Jingxin watermelon, seedless watermelon, black beauty, flower skin watermelon, yellow flesh watermelon, mini watermelon, winter melon watermelon, small jade watermelon, early spring red jade, black skin seedless watermelon, royal Malan watermelon, cucumber watermelon, snow top watermelon, wedding candy watermelon and Charleston gray; The sweetness of watermelon ranges from 5 to 15° Brix, the moisture content is 85% to 95%, and the hardness is between 1 and 5N; Watermelons are rated on a scale of 1-10 for quality.

5. A watermelon quality grading method based on multi-source information fusion according to claim 4, It is characterized in that The S2 specifically includes: resizing the three images taken of each watermelon to 224*224*3; The time domain, frequency domain and time-frequency feature extraction are performed on the sound signals of the two channels respectively, and the first three important features of the sound signals of the two channels are selected using F value and principal component analysis respectively; The vibration signals of the two channels are extracted in time domain, frequency domain and time-frequency features respectively, and the first three important features of the vibration signals of the two channels are selected using F value and principal component analysis respectively; Among them, time domain features include mean, variance, peak, root mean square value, skewness, and kurtosis; frequency domain features include spectrum center, spectrum bandwidth, spectrum peak, spectrum roll-off point, and frequency smoothness; time-frequency features include time-frequency features obtained by short-time Fourier transform, Mel frequency cepstrum coefficients, and wavelet transform features; Therefore, a total of 14 features are extracted from the signal of each channel; The 4 temperature values ​​and 3 weight values ​​of each watermelon are averaged to obtain the temperature average value and the weight average value.

6. A watermelon quality grading method based on multi-source information fusion according to claim 5, It is characterized in that In S3, a watermelon variety recognition model is constructed based on a time-distributed convolutional neural network TimeDistributed CNN, including an input layer, a TimeDistributed layer, a convolutional block unit, a feature fusion layer and an output layer; The input layer receives three watermelon images at different positions, each image size is (224, 224, 3), forming an RGB image with an input tensor size of (3, 224, 224, 3); the convolution operation is applied to each image using the TimeDistributed layer; The convolution block unit is used to extract the features of each image, which includes four convolution blocks, and convolution block 1 includes 64 convolution layers with a convolution kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2; Convolutional block 2 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2; Convolutional block 3 includes 256 convolutional layers with a kernel size of 3*3, ReLU activation function, batch normalization layer, and a maximum pooling layer with a size of 2*2; Convolutional block 4 includes 512 convolutional layers with a kernel size of 3*3, ReLU activation function, batch normalization layer, and a maximum pooling layer with a size of 2*2; The feature fusion layer includes a Flatten layer, a fully connected layer 1, and a fully connected layer 2; The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit, and further process them through the fully connected layer; The fully connected layer 1 includes 512 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5; The fully connected layer 2 includes 256 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5; The output layer uses the Softmax activation function to output the probabilities corresponding to the 15 watermelon varieties, and the one with the highest probability is the final watermelon variety.

7. A watermelon quality grading method based on multi-source information fusion according to claim 6, It is characterized in that In S3, the quality index prediction model includes an input layer, an image processing branch layer, a sound and vibration feature processing branch layer, a temperature and weight feature processing branch layer, a feature fusion layer and an output layer; the sound and vibration feature processing branch layer includes a sound feature processing branch and a vibration feature processing branch layer; The input layer receives three watermelon images at different positions, three features corresponding to the sound signals of the two channels, three features corresponding to the vibration signals of the two channels, the average value of the temperature and the average value of the weight; The three watermelon images in different positions are formed into an RGB image with a tensor size of (3, 224, 224, 3), which is input into the image processing branch layer. The six features corresponding to the sound signals of the two channels are input into the sound feature processing branch layer. The six features corresponding to the vibration signals of the two channels are input into the vibration feature processing branch layer. The average temperature and the average weight are input into the temperature and weight feature processing branch layer. The outputs of the image processing branch layer, the sound and vibration feature processing branch layer, and the temperature and weight feature processing branch layer are input into the feature fusion layer.

8. A watermelon quality grading method based on multi-source information fusion according to claim 7, It is characterized in that In S3, the image processing branch layer includes a TimeDistributed layer, five convolution blocks, and a feature fusion layer, and the feature fusion layer includes a Flatten layer, a fully connected layer 3, and a fully connected layer 4; the convolution operation is applied to each image using the TimeDistributed layer; Convolutional block 5 includes 64 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2; Convolutional block 6 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2; Convolutional block 7 includes 256 convolutional layers with a kernel size of 3x3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer of size 2*2; Convolutional block 8 includes 512 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer with a size of 2*2; Convolutional block 9 includes a convolutional layer with 1024 convolution kernels of size 3*3, a ReLU activation function, a batch normalization layer, and a maximum pooling layer of size 2*2; The Flatten layer is used to flatten and fuse the features of the three images output by the convolution block unit; The fully connected layer 3 includes 512 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.5; The fully connected layer 4 includes 256 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.5; The sound and vibration feature processing branch layer includes a sound feature branch layer and a vibration feature branch layer, the sound feature branch layer includes a fully connected layer 5 and a fully connected layer 6, and the vibration feature branch layer includes a fully connected layer 7 and a fully connected layer 8; The fully connected layer 5 and the fully connected layer 7 each include 128 neurons, a ReLU activation function, and a Dropout layer. The dropout rate of the Dropout layer is 0.3; The fully connected layer 6 and the fully connected layer 8 each include 64 neurons, a ReLU activation function, and a Dropout layer. The dropout rate of the Dropout layer is 0.3; The temperature and weight feature processing branch layer includes a fully connected layer 9, and the fully connected layer 9 includes 64 neurons and a ReLU activation function; The feature fusion layer includes a fully connected layer 10, a fully connected layer 11 and a fully connected layer 12, wherein the fully connected layer 10 includes 512 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.5; The fully connected layer 11 includes 256 neurons, a ReLU activation function, and a Dropout layer, and the dropout rate of the Dropout layer is 0.5; The fully connected layer 12 includes 128 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5; The output layer includes 3 neurons and a linear activation function, which is used to output the sweetness, moisture content and hardness of the watermelon.

9. A watermelon quality grading method based on multi-source information fusion according to claim 8, It is characterized in that In S3, the comprehensive evaluation model includes an input layer, a multi-layer fully connected layer and an output layer; The input layer receives the watermelon variety recognition result and the quality index prediction result; the variety recognition result is an array composed of 15 numbers, each number represents a watermelon variety, and the "one-hot encoding" method is used to ensure that only one number in the array is 1 at a time, and the rest are 0; the quality index prediction result is an array composed of 3 data, which respectively represent the sweetness, moisture content and hardness of the watermelon; The multi-layer fully connected layer includes a fully connected layer 13, a fully connected layer 14, a residual connected layer 1, a fully connected layer 15, a fully connected layer 16, a residual connected layer 2 and a fully connected layer 17; The fully connected layer 13 includes 512 neurons, a ReLU activation function, and a Dropout layer; The fully connected layer 14 includes 256 neurons, a ReLU activation function, and a Dropout layer; The residual connection layer 1 includes 256 neurons and a ReLU activation function; The fully connected layer 15 includes 128 neurons, a ReLU activation function, and a Dropout layer; The fully connected layer 16 includes 64 neurons, a ReLU activation function, and a Dropout layer; The residual connection layer 2 includes 64 neurons and a ReLU activation function; The fully connected layer 17 includes 32 neurons, a ReLU activation function, and a Dropout layer; The output layer includes 1 neuron and a linear activation function, and outputs the variety score of the watermelon.

Images