A watermelon quality grading method based on multi-source information fusion

Abstract

The application discloses a watermelon quality grading method based on multi-source information fusion, and belongs to the field of agricultural information technology. The method firstly acquires a data set, including the image of the watermelon, sound and vibration signals when knocking, skin temperature data, weight data, variety, sweetness, moisture content, hardness and quality score; the data set is subjected to data preprocessing; a watermelon variety identification model, a quality index prediction model and a comprehensive evaluation model are constructed, and the preprocessed data set is used to train the three models respectively; the trained watermelon variety identification model, quality index prediction model and comprehensive evaluation model are obtained, and the obtained watermelon image, sound and vibration signals when knocking, skin temperature data and weight data are used to grade the quality of the watermelon.

Description

Technical Field

[0001] This invention belongs to the field of agricultural information technology, specifically a watermelon quality grading method based on multi-source information fusion. Background Technology

[0002] Currently, there are many varieties of watermelons on the market, with varying quality and economic value. Pre-sale grading of watermelon quality can enhance product competitiveness and promote consumer purchases. Therefore, accurately identifying watermelon varieties and assessing their quality grades has become an urgent problem to be solved.

[0003] Current grading and classification technologies rely on manual experience and simple measuring tools during harvesting and sales, resulting in low accuracy and efficiency. Intelligent and mechanized grading and classification are still in the research stage. Therefore, there is an urgent need for a system that can automatically and accurately identify watermelon varieties and assess their quality grades to improve the efficiency and accuracy of market supervision and consumer selection. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention proposes a watermelon quality grading method based on multi-source information fusion. This method is rationally designed, overcomes the shortcomings of existing technologies, and achieves good results.

[0005] A watermelon quality grading method based on multi-source information fusion includes the following steps:

[0006] S1. Obtain the dataset, including watermelon images, sound and vibration signals when the watermelon is tapped, watermelon skin temperature data, weight data, variety, sweetness, moisture content, firmness and quality score.

[0007] S2. Perform data preprocessing on the dataset in S1;

[0008] S3. Construct a watermelon variety identification model, a quality index prediction model, and a comprehensive evaluation model, and train the three models using the preprocessed dataset;

[0009] The watermelon variety identification model takes a watermelon image as input and outputs the watermelon variety; the quality index prediction model takes a watermelon image, the sound and vibration signals when the watermelon is tapped, the watermelon skin temperature data, and weight data as input and outputs sweetness, moisture content, and firmness; the comprehensive evaluation model takes the watermelon variety identification result and the quality index prediction result as input and outputs the watermelon quality score.

[0010] S4. Obtain the trained watermelon variety identification model, quality index prediction model, and comprehensive evaluation model. Use the obtained watermelon images, sound and vibration signals when tapped, skin temperature data, and weight data to grade the quality of the watermelon.

[0011] Furthermore, in S1, a camera, a dual-channel microphone, a dual-channel accelerometer, an infrared temperature sensor, a weighing sensor, a portable saccharimeter, and a digital fruit firmness meter are used to collect data on different varieties of watermelons.

[0012] The camera is used to capture three images of the same watermelon from different locations;

[0013] The dual-channel microphone is used to collect the dual-channel sound signals emitted when the watermelon is struck.

[0014] The dual-channel accelerometer is used to collect dual-channel vibration signals when a watermelon is struck.

[0015] The infrared temperature sensor is used to measure the temperature data of the watermelon rind.

[0016] The weighing sensor is used to measure the weight of the watermelon;

[0017] The portable saccharimeter is used to measure the sugar content of watermelon flesh and record the measurement results as watermelon sweetness data;

[0018] The digital display fruit hardness tester is used to measure the hardness of watermelon flesh and record the measurement results as watermelon hardness data.

[0019] The moisture content of watermelons was measured using the drying method, which involved cutting and weighing the watermelons, drying them to a constant weight, weighing them again, and calculating the moisture content.

[0020] Watermelon quality is graded based on variety, sweetness, moisture content, and firmness, according to established watermelon grading standards.

[0021] Furthermore, in S1, when using a dual-channel microphone to collect sound signals, two microphones are set up. Microphone 1 is placed near the middle between the top and bottom of the watermelon, and microphone 2 is placed on the opposite side of microphone 1, thus obtaining two independent time-series sound signals.

[0022] When collecting vibration signals using a dual-channel accelerometer, two accelerometers are set up, with accelerometer 1 and accelerometer 2 placed at the top and bottom of the watermelon, respectively, to obtain two independent time-series vibration signals.

[0023] When collecting temperature data using an infrared temperature sensor, the watermelon skin temperature is measured at four symmetrical points centered on the cross-section center of the watermelon, and four temperature values ​​are obtained in the end.

[0024] When collecting weight data using a weighing sensor, the weight of the watermelon was measured three times consecutively, and three weight values ​​were obtained in the end.

[0025] Furthermore, in S1, the watermelon varieties acquired in the data set include Jingxin watermelon, seedless watermelon, Black Beauty, striped watermelon, yellow-fleshed watermelon, mini watermelon, winter melon watermelon, Xiaoyu watermelon, early spring red jade, black-skinned seedless watermelon, Royal Malan watermelon, small cucumber watermelon, snow-top watermelon, Xitang watermelon, and Charleston Grey.

[0026] Watermelons have a sweetness range of 5 to 15°Brix, a water content of 85% to 95%, and a hardness of 1 to 5N.

[0027] Watermelons are rated on a scale of 1-10.

[0028] Furthermore, S2 specifically involves: resizing the three images taken of each watermelon to 224*224*3;

[0029] Time-domain, frequency-domain, and time-frequency features were extracted from the audio signals of the two channels, and the F-value and principal component analysis were used to select the top three important features of the audio signals of the two channels, respectively.

[0030] The vibration signals of the two channels were subjected to time domain, frequency domain, and time-frequency feature extraction, respectively. The F-value and principal component analysis were used to select the top three important features of the vibration signals of the two channels.

[0031] Among them, the time-domain features include mean, variance, peak value, root mean square value, skewness, and kurtosis; the frequency-domain features include spectral center, spectral bandwidth, spectral peak value, spectral roll-off point, and frequency smoothness; the time-frequency features include time-frequency features obtained by short-time Fourier transform, Mel frequency cepstral coefficients, and wavelet transform features; therefore, a total of 14 features are extracted from the signal of each channel.

[0032] The average values ​​of the four temperature values ​​and three weight values ​​for each watermelon are calculated to obtain the average temperature value and the average weight value.

[0033] Furthermore, in S3, a watermelon variety recognition model is constructed based on a time-distributed convolutional neural network (TimeDistributed CNN), including an input layer, a TimeDistributed layer, convolutional block units, a feature fusion layer, and an output layer.

[0034] The input layer receives three watermelon images from different locations, each image being (224,224,3) in size, forming an RGB image with an input tensor size of (3,224,224,3); the TimeDistributed layer is used to apply convolution operations to each image;

[0035] The convolutional block unit is used to extract features from each image. It includes four convolutional blocks. Convolutional block 1 includes 64 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a size of 2*2.

[0036] Convolutional block 2 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0037] Convolutional block 3 includes 256 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0038] Convolutional block 4 includes 512 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0039] The feature fusion layer includes a Flatten layer, a fully connected layer 1, and a fully connected layer 2;

[0040] The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit, and then further process them through a fully connected layer.

[0041] The fully connected layer 1 includes 512 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5.

[0042] The fully connected layer 2 includes 256 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5.

[0043] The output layer uses the Softmax activation function to output the probabilities corresponding to the 15 watermelon varieties, and the watermelon variety with the highest probability is the final watermelon variety.

[0044] Furthermore, in S3, the quality index prediction model includes an input layer, an image processing branch layer, a sound and vibration feature processing branch layer, a temperature and weight feature processing branch layer, a feature fusion layer, and an output layer; the sound and vibration feature processing branch layer includes a sound feature processing branch and a vibration feature processing branch layer.

[0045] The input layer receives three watermelon images from different locations, three features corresponding to the sound signals from the two channels, three features corresponding to the vibration signals from the two channels, the average temperature, and the average weight.

[0046] Three watermelon images from different locations are converted into an RGB image with a tensor size of (3,224,224,3) and input into the image processing branch layer. The six features corresponding to the sound signals from the two channels are input into the sound feature processing branch layer, the six features corresponding to the vibration signals from the two channels are input into the vibration feature processing branch layer, and the average temperature and average weight are input into the temperature and weight feature processing branch layer. The outputs of the image processing branch layer, the sound and vibration feature processing branch layer, and the temperature and weight feature processing branch layer are all input into the feature fusion layer.

[0047] Furthermore, in S3, the image processing branch layer includes a TimeDistributed layer, five convolutional blocks, and a feature fusion layer. The feature fusion layer includes a Flatten layer, a fully connected layer 3, and a fully connected layer 4. The TimeDistributed layer is used to apply the convolution operation to each image.

[0048] Convolutional block 5 includes 64 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0049] Convolutional block 6 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0050] Convolutional block 7 includes 256 convolutional layers with a kernel size of 3x3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a size of 2*2;

[0051] Convolutional block 8 includes 512 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0052] Convolutional block 9 includes 1024 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0053] The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit;

[0054] The fully connected layer 3 includes 512 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0055] The fully connected layer 4 includes 256 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0056] The sound and vibration feature processing branch layer includes a sound feature branch layer and a vibration feature branch layer. The sound feature branch layer includes a fully connected layer 5 and a fully connected layer 6. The vibration feature branch layer includes a fully connected layer 7 and a fully connected layer 8.

[0057] Both fully connected layer 5 and fully connected layer 7 include 128 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.3.

[0058] Both fully connected layer 6 and fully connected layer 8 include 64 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.3.

[0059] The temperature and weight feature processing branch layer includes a fully connected layer 9, which includes 64 neurons and a ReLU activation function;

[0060] The feature fusion layer includes a fully connected layer 10, a fully connected layer 11, and a fully connected layer 12. The fully connected layer 10 includes 512 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0061] The fully connected layer 11 includes 256 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0062] The fully connected layer 12 includes 128 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5.

[0063] The output layer consists of three neurons and a linear activation function, used to output the sweetness, moisture content, and firmness of the watermelon.

[0064] Furthermore, in S3, the comprehensive evaluation model includes an input layer, multiple fully connected layers, and an output layer;

[0065] The input layer receives watermelon variety identification results and quality index prediction results. The variety identification result is an array of 15 numbers, each number representing a watermelon variety. The "one-hot encoding" method is used to ensure that only one number in the array is 1 at any given time, and the rest are 0. The quality index prediction result is an array of 3 data, representing the watermelon's sweetness, moisture content, and firmness.

[0066] The multilayer fully connected layer includes fully connected layer 13, fully connected layer 14, residual connected layer 1, fully connected layer 15, fully connected layer 16, residual connected layer 2 and fully connected layer 17;

[0067] The fully connected layer 13 includes 512 neurons, a ReLU activation function, and a Dropout layer;

[0068] The fully connected layer 14 includes 256 neurons, a ReLU activation function, and a Dropout layer;

[0069] The residual connection layer 1 includes 256 neurons and a ReLU activation function;

[0070] The fully connected layer 15 includes 128 neurons, a ReLU activation function, and a Dropout layer;

[0071] The fully connected layer 16 includes 64 neurons, a ReLU activation function, and a Dropout layer;

[0072] The residual connection layer 2 includes 64 neurons and a ReLU activation function;

[0073] The fully connected layer 17 includes 32 neurons, a ReLU activation function, and a Dropout layer;

[0074] The output layer consists of one neuron and a linear activation function, which outputs a quality score for the watermelon.

[0075] The beneficial technical effects of this invention are as follows:

[0076] 1. Improved Grading Accuracy: This invention integrates multi-source information, such as image data, sugar content data, and firmness information, to comprehensively analyze the variety and quality of watermelons. Compared to traditional manual experience-based grading methods, this invention significantly improves grading accuracy and reduces the impact of human error.

[0077] 2. Improve grading efficiency: This invention uses intelligent algorithms to make the watermelon grading process more automated and efficient, significantly shortening the grading time and adapting to the needs of large-scale agricultural harvesting and sales.

[0078] 3. Quality Visualization: Through multi-source information fusion technology, this invention can provide intuitive visualization results of watermelon quality, helping consumers better understand watermelon quality information and making the consumption process more transparent.

[0079] 4. Supporting Market Supervision: The systematic and standardized grading method of this invention provides reliable technical support for market supervision, helps to establish a unified watermelon quality grading standard, and promotes the standardized development of the market.

[0080] 5. Increased economic benefits: Through accurate quality grading, this invention improves the market competitiveness of high-quality watermelons, promotes increased sales, and thus increases the economic benefits for producers and sellers. Attached Figure Description

[0081] Figure 1 This is a flowchart of the data acquisition and preprocessing process in this invention;

[0082] Figure 2 This is a flowchart of watermelon variety identification in this invention;

[0083] Figure 3 This is a flowchart illustrating the prediction of watermelon quality indicators in this invention.

[0084] Figure 4 This is a flowchart of the comprehensive evaluation of watermelons in this invention; Detailed Implementation

[0085] The specific embodiments of the present invention will be further described below with reference to specific examples:

[0086] A watermelon quality grading method based on multi-source information fusion includes the following steps:

[0087] S1. Obtain the dataset, including watermelon images, sound and vibration signals when the watermelon is tapped, watermelon skin temperature data, weight data, variety, sweetness, moisture content, firmness and quality score.

[0088] S2. Perform data preprocessing on the dataset in S1;

[0089] S3. Construct a watermelon variety identification model, a quality index prediction model, and a comprehensive evaluation model, and train the three models using the preprocessed dataset;

[0090] The watermelon variety identification model takes a watermelon image as input and outputs the variety; the quality index prediction model takes a watermelon image, the sound and vibration signals when the watermelon is tapped, the watermelon skin temperature data, and weight data as input and outputs sweetness, moisture content, and firmness; the comprehensive evaluation model takes the watermelon variety identification result and the quality index prediction result as input and outputs the watermelon quality score.

[0091] S4. Obtain the trained watermelon variety identification model, quality index prediction model, and comprehensive evaluation model. Use the obtained watermelon images, sound and vibration signals when tapped, skin temperature data, and weight data to grade the quality of the watermelon.

[0092] Specifically, in S1, such as Figure 1As shown, data on different varieties of watermelons were collected using a camera, dual-channel microphone, dual-channel accelerometer, infrared temperature sensor, weighing sensor, portable saccharimeter, and digital fruit firmness meter. The watermelon varieties included Jingxin watermelon, seedless watermelon, Black Beauty, striped watermelon, yellow-fleshed watermelon, mini watermelon, winter melon watermelon, Xiaoyu watermelon, early spring red jade, black-skinned seedless watermelon, Royal Malan watermelon, small cucumber watermelon, snow-top watermelon, Xitang watermelon, and Charleston Grey.

[0093] The camera is used to take three images of the same watermelon from different locations. The camera can be mounted on a fixed bracket, or a non-fixed or handheld camera can be used to take images of the watermelon's appearance.

[0094] A dual-channel microphone is used to collect the dual-channel sound signals emitted when a watermelon is tapped. When collecting sound signals using a dual-channel microphone, two microphones are set up. Microphone 1 is placed on one side of the watermelon, close to the middle, and microphone 2 is placed on the opposite side of microphone 1, close to the middle, to obtain sound signals from different paths. This helps to analyze the internal structure of the watermelon more comprehensively and ultimately obtain two independent time-series sound signals.

[0095] A dual-channel accelerometer is used to collect dual-channel vibration signals when a watermelon is struck. When collecting vibration signals using a dual-channel accelerometer, two accelerometers are set up, with accelerometer 1 and accelerometer 2 placed at the top and bottom of the watermelon, respectively, to capture vibration signals in different directions. This helps to more comprehensively analyze the watermelon's hardness and internal porosity, and ultimately obtain two independent time-series vibration signals.

[0096] Infrared temperature sensors are used to measure the temperature of watermelon rind. When collecting temperature data using infrared temperature sensors, the watermelon rind temperature is measured at four symmetrical points centered on the cross-section center, and four temperature values ​​are obtained in the end.

[0097] The weighing sensor is used to measure the weight of the watermelon. When collecting weight data using the weighing sensor, the weight of the watermelon is measured three times consecutively to obtain three weight values.

[0098] The sugar content of watermelon flesh was measured using a portable refractometer, and the results were recorded as watermelon sweetness data; the sweetness range of watermelon was 5 to 15°Brix.

[0099] The moisture content of watermelons was measured using a drying method. Specifically, the watermelons were cut, weighed, dried to a constant weight, and weighed again to calculate the moisture content. The moisture content ranged from 85% to 95%.

[0100] The hardness of watermelon flesh was measured using a digital fruit hardness tester, and the measurement results were recorded as watermelon hardness data; the hardness ranged from 1 to 5 N.

[0101] The sweetness, moisture, and firmness data mentioned above were obtained by combining literature data and data from different watermelon varieties purchased from the actual market.

[0102] Watermelons are graded based on their variety, sweetness, moisture content, and firmness, according to established watermelon grading standards, with a quality score ranging from 1 to 10.

[0103] In this embodiment, the watermelon quality is scored using a 1-10 point system, and the specific grading criteria are shown in Table 1:

[0104] Table 1 Watermelon Quality Scoring Criteria

[0105]

[0106]

[0107] The grading standards can be adjusted according to the needs of growers or sales personnel, and then the comprehensive evaluation model can be retrained to improve the adaptability and scalability of the method.

[0108] Specifically, such as Figure 1 As shown, S2 specifically involves: resizing the three images of each watermelon to 224*224*3; extracting time-domain, frequency-domain, and time-frequency features from the audio signals of the two channels respectively; and selecting the first three important features of the audio signals of the two channels using F-value and principal component analysis respectively.

[0109] The vibration signals of the two channels were subjected to time domain, frequency domain, and time-frequency feature extraction, respectively. The F-value and principal component analysis were used to select the top three important features of the vibration signals of the two channels.

[0110] The time-domain features include mean, variance, peak value, root mean square (RMS), skewness, and kurtosis; the frequency-domain features include spectral centroid, spectral bandwidth, spectral peak, spectral roll-off, and spectral flatness; the time-frequency features include time-frequency features obtained through short-time Fourier transform (STFT), Mel-frequency cepstral coefficients (MFCC), and wavelet transform features. Therefore, a total of 14 features are extracted for each channel's signal.

[0111] The method uses the ANOVA F-value to evaluate the significance of each feature, and then selects the three features with the highest F-values. This method is independent of the specific model; it only needs to calculate the correlation between each feature and the target variable. The calculation steps are: 1) Calculate the within-group variance and between-group variance: For each feature X, calculate the within-group variance and between-group variance for its different classes y. Let k be the number of classes, and n... i Let n be the number of samples in the i-th class, and n be the total number of samples. Let be the mean of the i-th class. 1) Calculate the overall mean. 2) Calculate the within-group variance. Calculate the between-group variance. 4) Calculate the F value. 5) Feature selection: Calculate and compare the F-values ​​of each feature, and then select the three features with the highest F-values ​​as the most important features.

[0112] Feature selection using Principal Component Analysis (PCA) involves the following steps: 1) Standardize the data so that each feature has zero mean and unit variance. 2) Calculate the covariance matrix C of the standardized data. 3) Perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalues ​​λ. i and eigenvector V i 4) Sort the eigenvalues ​​by size and select the eigenvectors corresponding to the three largest eigenvalues.

[0113] The average values ​​of the four temperature values ​​and three weight values ​​for each watermelon are calculated to obtain the average temperature value and the average weight value.

[0114] Specifically, in S3, a watermelon variety recognition model is constructed based on a time-distributed convolutional neural network (TimeDistributed CNN), including an input layer, a TimeDistributed layer, convolutional block units, a feature fusion layer, and an output layer;

[0115] The input layer receives three watermelon images from different locations, each with a size of (224,224,3), forming an RGB image with an input tensor size of (3,224,224,3). The TimeDistributed layer is used to apply subsequent convolution operations to each image.

[0116] The convolutional block unit is used to extract features from each image. It includes four convolutional blocks. Convolutional block 1 includes 64 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a size of 2*2.

[0117] Convolutional block 2 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0118] Convolutional block 3 includes 256 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0119] Convolutional block 4 includes 512 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0120] The feature fusion layer is used to flatten the features of each image and further process them through fully connected layers, specifically including the Flatten layer, fully connected layer 1, and fully connected layer 2.

[0121] The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit;

[0122] Fully connected layer 1 consists of 512 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.5.

[0123] Fully connected layer 2 consists of 256 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0124] The output layer is a classification layer, using the Softmax activation function to output the probabilities of 15 watermelon varieties. The variety with the highest probability is the final watermelon variety. Figure 2 As shown.

[0125] The role of the TimeDistributed layer is to apply the same operation to multiple input data. In the watermelon variety recognition model, the TimeDistributed layer applies convolution operations to each image, ensuring that each image undergoes the same convolutional processing, and then these features are fused for classification. This effectively handles multiple input data of the same type. Compared to traditional CNNs, TimeDistributed CNNs are more suitable for handling multiple input scenarios because traditional CNNs can only process single images.

[0126] A quality indicator prediction model is constructed based on a multimodal neural network, including an input layer, an image processing branch layer, a sound and vibration feature processing branch layer, a temperature and weight feature processing branch layer, a feature fusion layer, and an output layer; the sound and vibration feature processing branch layer includes a sound feature processing branch and a vibration feature processing branch layer;

[0127] The input layer receives three watermelon images from different locations, three features corresponding to the sound signals from the two channels, three features corresponding to the vibration signals from the two channels, the average temperature, and the average weight.

[0128] Three watermelon images from different locations (i.e., RGB images of size (3,224,224,3)) are input to the image processing branch layer. The six features corresponding to the sound signals from the two channels are input to the sound feature processing branch layer. The six features corresponding to the vibration signals from the two channels are input to the vibration feature processing branch layer. The average temperature and average weight are input to the temperature and weight feature processing branch layer. The outputs of the image processing branch layer, the sound and vibration feature processing branch layer, and the temperature and weight feature processing branch layer are input together to the feature fusion layer.

[0129] The image processing branch layer includes a TimeDistributed layer (which applies subsequent convolutional operations to each image), five convolutional blocks, a Flatten layer, a fully connected layer 3, and a fully connected layer 4.

[0130] Convolutional block 5 includes 64 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0131] Convolutional block 6 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0132] Convolutional block 7 includes 256 convolutional layers with a kernel size of 3x3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a size of 2*2;

[0133] Convolutional block 8 includes 512 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0134] Convolutional block 9 includes 1024 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2;

[0135] The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit;

[0136] The fully connected layer 3 consists of 512 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0137] The fully connected layer 4 consists of 256 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0138] The sound and vibration feature processing branch layer includes a sound feature branch layer and a vibration feature branch layer. The sound feature branch layer includes fully connected layer 5 and fully connected layer 6, and the vibration feature branch layer includes fully connected layer 7 and fully connected layer 8.

[0139] Both fully connected layer 5 and fully connected layer 7 contain 128 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.3.

[0140] Both fully connected layer 6 and fully connected layer 8 contain 64 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.3.

[0141] The temperature and weight feature processing branch layer includes a fully connected layer 9, which contains 64 neurons and a ReLU activation function;

[0142] The feature fusion layer includes fully connected layer 10, fully connected layer 11 and fully connected layer 12. Fully connected layer 10 includes 512 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0143] The fully connected layer 11 includes 256 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0144] The fully connected layer 12 consists of 128 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.5.

[0145] The output layer consists of three neurons and a linear activation function, used to output the sweetness, water content, and firmness of the watermelon, such as... Figure 3 As shown.

[0146] The comprehensive evaluation model includes an input layer, multiple fully connected layers, and an output layer;

[0147] The input layer receives watermelon variety identification results (one-hot encoding) and quality index prediction results (sweetness, moisture content, firmness);

[0148] The variety identification result is an array of 15 numbers, each representing a watermelon variety. A one-hot encoding method is used to ensure that only one number in the array is 1 at any given time, with the rest being 0, indicating the watermelon's category. The quality index prediction result is an array of three data points, representing the watermelon's sweetness, moisture content, and firmness. The multilayer fully connected layers include fully connected layer 13, fully connected layer 14, residual connected layer 1, fully connected layer 15, fully connected layer 16, residual connected layer 2, and fully connected layer 17.

[0149] Fully connected layer 13 includes 512 neurons, a ReLU activation function, and a Dropout layer;

[0150] Fully connected layer 14 includes 256 neurons, a ReLU activation function, and a Dropout layer;

[0151] Residual connection layer 1 consists of 256 neurons and a ReLU activation function;

[0152] Fully connected layer 15 includes 128 neurons, a ReLU activation function, and a Dropout layer;

[0153] The fully connected layer 16 includes 64 neurons, a ReLU activation function, and a Dropout layer;

[0154] Residual connection layer 2 consists of 64 neurons and a ReLU activation function;

[0155] Fully connected layer 17 includes 32 neurons, a ReLU activation function, and a Dropout layer;

[0156] The output layer consists of one neuron and a linear activation function, outputting a quality score for the watermelon, such as... Figure 4 As shown.

[0157] This invention integrates multi-source information, such as image data, sugar content data, and firmness information, to comprehensively analyze the variety and quality of watermelons. Compared to traditional manual experience-based grading methods, this invention significantly improves grading accuracy and reduces the impact of human error.

[0158] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A watermelon quality grading method based on multi-source information fusion, characterized in that, Includes the following steps: S1. Obtain the dataset, including watermelon images, sound and vibration signals when the watermelon is tapped, watermelon skin temperature data, weight data, variety, sweetness, moisture content, firmness and quality score. S2. Perform data preprocessing on the dataset in S1; S3. Construct a watermelon variety identification model, a quality index prediction model, and a comprehensive evaluation model, and train the three models using the preprocessed dataset; The watermelon variety identification model takes a watermelon image as input and outputs the watermelon variety; the quality index prediction model takes a watermelon image, the sound and vibration signals when the watermelon is tapped, the watermelon skin temperature data, and weight data as input and outputs sweetness, moisture content, and firmness; the comprehensive evaluation model takes the watermelon variety identification result and the quality index prediction result as input and outputs the watermelon quality score. The quality index prediction model includes an input layer, an image processing branch layer, a sound and vibration feature processing branch layer, a temperature and weight feature processing branch layer, a feature fusion layer, and an output layer; the sound and vibration feature processing branch layer includes a sound feature processing branch and a vibration feature processing branch layer. S4. Obtain the trained watermelon variety identification model, quality index prediction model, and comprehensive evaluation model. Use the obtained watermelon images, sound and vibration signals when tapped, skin temperature data, and weight data to grade the quality of the watermelon.

2. The watermelon quality grading method based on multi-source information fusion according to claim 1, characterized in that, In S1, a camera, a dual-channel microphone, a dual-channel accelerometer, an infrared temperature sensor, a weighing sensor, a portable saccharimeter, and a digital fruit firmness meter are used to collect data on different varieties of watermelons. The camera is used to capture three images of the same watermelon from different locations; The dual-channel microphone is used to collect the dual-channel sound signals emitted when the watermelon is struck. The dual-channel accelerometer is used to collect dual-channel vibration signals when a watermelon is struck. The infrared temperature sensor is used to measure the temperature data of the watermelon rind. The weighing sensor is used to measure the weight of the watermelon; The portable saccharimeter is used to measure the sugar content of watermelon flesh and record the measurement results as watermelon sweetness data; The digital display fruit hardness tester is used to measure the hardness of watermelon flesh and record the measurement results as watermelon hardness data. The moisture content of watermelons was measured using the drying method, which involved cutting and weighing the watermelons, drying them to a constant weight, weighing them again, and calculating the moisture content. Watermelon quality is graded based on variety, sweetness, moisture content, and firmness, according to established watermelon grading standards.

3. The watermelon quality grading method based on multi-source information fusion according to claim 2, characterized in that, In S1, when using a dual-channel microphone to collect sound signals, two microphones are set up. Microphone 1 is placed near the middle between the top and bottom of the watermelon, and microphone 2 is placed on the opposite side of microphone 1, so that two independent time-series sound signals are finally obtained. When collecting vibration signals using a dual-channel accelerometer, two accelerometers are set up, with accelerometer 1 and accelerometer 2 placed at the top and bottom of the watermelon, respectively, to obtain two independent time-series vibration signals. When collecting temperature data using an infrared temperature sensor, the watermelon skin temperature is measured at four symmetrical points centered on the cross-section center of the watermelon, and four temperature values ​​are obtained in the end. When collecting weight data using a weighing sensor, the weight of the watermelon was measured three times consecutively, and three weight values ​​were obtained in the end.

4. The watermelon quality grading method based on multi-source information fusion according to claim 3, characterized in that, In S1, the data set obtained includes watermelon varieties such as Jingxin watermelon, seedless watermelon, Black Beauty, striped watermelon, yellow-fleshed watermelon, mini watermelon, winter melon watermelon, Xiaoyu watermelon, early spring red jade, black-skinned seedless watermelon, Royal Malan watermelon, small cucumber watermelon, snow-top watermelon, Xitang watermelon, and Charleston Grey. Watermelons have a sweetness range of 5 to 15°Brix, a water content of 85% to 95%, and a hardness of 1 to 5 N. Watermelons are rated on a scale of 1-10.

5. The watermelon quality grading method based on multi-source information fusion according to claim 4, characterized in that, Specifically, S2 involves resizing the three images taken of each watermelon to 224*224*3. Time-domain, frequency-domain, and time-frequency features were extracted from the audio signals of the two channels, and the F-value and principal component analysis were used to select the top three important features of the audio signals of the two channels, respectively. The vibration signals of the two channels were subjected to time domain, frequency domain, and time-frequency feature extraction, respectively. The F-value and principal component analysis were used to select the top three important features of the vibration signals of the two channels. Among them, time-domain features include mean, variance, peak value, root mean square value, skewness, and kurtosis; frequency-domain features include spectral center, spectral bandwidth, spectral peak value, spectral roll-off point, and frequency smoothness; time-frequency features include time-frequency features obtained through short-time Fourier transform, Mel frequency cepstral coefficients, and wavelet transform features. Therefore, a total of 14 features are extracted from the signal of each channel; The average values ​​of the four temperature values ​​and three weight values ​​for each watermelon are calculated to obtain the average temperature value and the average weight value.

6. The watermelon quality grading method based on multi-source information fusion according to claim 5, characterized in that, In S3, a watermelon variety identification model is constructed based on a time-distributed convolutional neural network (TimeDistributed CNN), including an input layer, a TimeDistributed layer, convolutional block units, a feature fusion layer, and an output layer. The input layer receives three watermelon images from different locations, each image being (224, 224, 3) in size, forming an RGB image with an input tensor size of (3, 224, 224, 3); the TimeDistributed layer is used to apply convolution operations to each image; The convolutional block unit is used to extract features from each image. It includes four convolutional blocks. Convolutional block 1 includes 64 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a size of 2*2. Convolutional block 2 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2; Convolutional block 3 includes 256 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2; Convolutional block 4 includes 512 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2; The feature fusion layer includes a Flatten layer, a fully connected layer 1, and a fully connected layer 2; The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit, and then further process them through a fully connected layer. The fully connected layer 1 includes 512 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.

5. The fully connected layer 2 includes 256 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.

5. The output layer uses the Softmax activation function to output the probabilities corresponding to the 15 watermelon varieties, and the watermelon variety with the highest probability is the final watermelon variety.

7. A watermelon quality grading method based on multi-source information fusion according to claim 6, characterized in that, In S3, the input layer receives three watermelon images from different locations, three features corresponding to the sound signals from the two channels, three features corresponding to the vibration signals from the two channels, the average temperature, and the average weight. Three watermelon images from different locations are converted into an RGB image with a tensor size of (3, 224, 224, 3) and input into the image processing branch layer. The six features corresponding to the sound signals from the two channels are input into the sound feature processing branch layer, the six features corresponding to the vibration signals from the two channels are input into the vibration feature processing branch layer, and the average temperature and average weight are input into the temperature and weight feature processing branch layer. The outputs of the image processing branch layer, the sound and vibration feature processing branch layer, and the temperature and weight feature processing branch layer are all input into the feature fusion layer.

8. A watermelon quality grading method based on multi-source information fusion according to claim 7, characterized in that, In S3, the image processing branch layer includes a TimeDistributed layer, five convolutional blocks, and a feature fusion layer. The feature fusion layer includes a Flatten layer, a fully connected layer 3, and a fully connected layer 4. The TimeDistributed layer is used to apply the convolution operation to each image. Convolutional block 5 includes 64 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2; Convolutional block 6 includes 128 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2; Convolutional block 7 includes 256 convolutional layers with a kernel size of 3x3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a size of 2*2; Convolutional block 8 includes 512 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2; Convolutional block 9 includes 1024 convolutional layers with a kernel size of 3*3, a ReLU activation function, a batch normalization layer, and a max pooling layer with a kernel size of 2*2; The Flatten layer is used to flatten and fuse the features of the three images output by the convolutional block unit; The fully connected layer 3 includes 512 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.

5. The fully connected layer 4 includes 256 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.

5. The sound and vibration feature processing branch layer includes a sound feature branch layer and a vibration feature branch layer. The sound feature branch layer includes a fully connected layer 5 and a fully connected layer 6. The vibration feature branch layer includes a fully connected layer 7 and a fully connected layer 8. Both fully connected layer 5 and fully connected layer 7 include 128 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.

3. Both fully connected layer 6 and fully connected layer 8 include 64 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.

3. The temperature and weight feature processing branch layer includes a fully connected layer 9, which includes 64 neurons and a ReLU activation function; The feature fusion layer includes a fully connected layer 10, a fully connected layer 11, and a fully connected layer 12. The fully connected layer 10 includes 512 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.

5. The fully connected layer 11 includes 256 neurons, a ReLU activation function, and a Dropout layer with a dropout rate of 0.

5. The fully connected layer 12 includes 128 neurons, a ReLU activation function, a Dropout layer, and a dropout rate of 0.

5. The output layer consists of three neurons and a linear activation function, used to output the sweetness, moisture content, and firmness of the watermelon.

9. A watermelon quality grading method based on multi-source information fusion according to claim 8, characterized in that, In S3, the comprehensive evaluation model includes an input layer, multiple fully connected layers, and an output layer; The input layer receives watermelon variety identification results and quality index prediction results. The variety identification result is an array of 15 numbers, each number representing a watermelon variety. The "one-hot encoding" method is used to ensure that only one number in the array is 1 at any given time, and the rest are 0. The quality index prediction result is an array of 3 data, representing the watermelon's sweetness, moisture content, and firmness. The multilayer fully connected layer includes fully connected layer 13, fully connected layer 14, residual connected layer 1, fully connected layer 15, fully connected layer 16, residual connected layer 2 and fully connected layer 17; The fully connected layer 13 includes 512 neurons, a ReLU activation function, and a Dropout layer; The fully connected layer 14 includes 256 neurons, a ReLU activation function, and a Dropout layer; The residual connection layer 1 includes 256 neurons and a ReLU activation function; The fully connected layer 15 includes 128 neurons, a ReLU activation function, and a Dropout layer; The fully connected layer 16 includes 64 neurons, a ReLU activation function, and a Dropout layer; The residual connection layer 2 includes 64 neurons and a ReLU activation function; The fully connected layer 17 includes 32 neurons, a ReLU activation function, and a Dropout layer; The output layer consists of one neuron and a linear activation function, which outputs a watermelon variety score.

Images