Fresh grape harvesting opportunity prediction system and method based on deep learning model

Abstract

The invention provides a deep learning model-based table grape harvesting opportunity prediction system and method, the system comprises a hyperspectral image shooting device, a data acquisition device, a controller and a data processor, the hyperspectral image shooting device is used for acquiring hyperspectral image information of table grapes in different mature periods; the data acquisition device is used for acquiring physical and chemical index data of the maturity of the table grapes in different maturity periods; the controller is used for controlling the hyperspectral image shooting device and storing data acquired in the execution process of the hyperspectral image shooting device and the data acquisition device in a memory of the controller; and the data processor is used for processing the data in the memory. According to the method, the accuracy of maturity index regression prediction is improved, and the harvesting opportunity can be determined more accurately and reasonably.

Description

Technical Field

[0001] The present invention relates to the field of agricultural information technology, and in particular to a prediction system and method for harvesting time of table grapes based on a deep learning model. Background Art

[0002] Fresh-table grapes are rich in nutrition, have a good taste, are widely consumed, and have high economic benefits. They have become a representative fruit industry for the development of characteristic forestry and fruit industries in many regions across the country. The post-harvest quality of fresh-table grapes is highly correlated with their maturity at the time of harvest. Therefore, accurately testing the maturity of grapes and determining the appropriate time to harvest is of great significance for ensuring the quality and flavor of grapes after harvest and reducing post-harvest waste.

[0003] In the prior art, the following four methods are usually used to detect the maturity of grapes: 1. Manual discrimination method, professional inspectors use sensory evaluation to judge the maturity of fresh grapes. 2. Chemical experiment detection method, in the laboratory, chemical instruments and reagents are used to measure the soluble solids, pH, hardness and other related physical and chemical data of the grape samples to determine the maturity of the grapes. 3. Computer vision technology, using image processing technology to predict the maturity of grapes through grape images. 4. Spectral technology, first use a spectral instrument to obtain the spectral data of grapes, and then predict and distinguish the maturity of grapes through modeling of spectral data and maturity characterization indicators.

[0004] The above methods have the following shortcomings: Method 1 relies on the experience of the evaluators, is highly subjective, and has large errors; Method 2 has complex sample processing, cumbersome detection process, poor real-time performance, and is a destructive test, which does not meet the needs of actual production; Method 3 relies on the external color, shape and other characteristics of grape berries, which is greatly affected by the individual development differences of grapes and has low detection accuracy; Method 4, after constructing the fitting model of spectral information and maturity index, can predict and distinguish the maturity by detecting spectral information in the future, which has the characteristics of fast, non-destructive and high accuracy. Although the comprehensive effect of spectral technology is better, the training of the existing fitting model is not sufficient, which seriously affects the prediction effect. Therefore, it is very necessary to design a prediction system and method for the harvest time of fresh grapes based on deep learning model. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide a system and method for predicting the timing of harvesting table grapes based on a deep learning model.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The present invention provides a prediction system for the harvesting time of table grapes based on a deep learning model, comprising: a hyperspectral image shooting device, a data acquisition device, a controller and a data processor, wherein the hyperspectral image shooting device is used to obtain hyperspectral image information of table grapes at different maturity stages; the data acquisition device is used to obtain physical and chemical index data of maturity of table grapes at different maturity stages; the controller is used to control the hyperspectral image shooting device and store data acquired during the execution of the hyperspectral image shooting device and the data acquisition device in a memory of the controller; and the data processor is used to process the data in the memory.

[0008] Preferably, the hyperspectral image capturing device includes a grape position adjusting device and an adjustable capturing device;

[0009] The grape position adjustment device comprises a conveying device, a carrying plate, a limiting device and a rotating sphere. A plurality of the carrying plates are evenly arranged on the top of the conveying device. The limiting devices are arranged on the four edges of the top of each of the carrying plates. The limiting devices comprise a vertical plate, a connecting shaft and a limiting wheel. The vertical plates are arranged on the four edges of the top of each of the carrying plates. Connecting shafts are arranged on both sides of the long sides of the vertical plates. The limiting wheels are arranged on the connecting shafts. A plurality of circular holes are evenly arranged on the bottom of the carrying plates, and the circular holes are located between the four limiting wheels between two adjacent limiting devices. The bottom of the rotating sphere contacts the conveying device through the circular holes. The conveying device is used to drive the rotating sphere to rotate. The arc-shaped positioning plates are arranged on the vertical plates corresponding to the rotating spheres. The conveying device is connected to the controller.

[0010] The adjustable shooting device includes a horizontal slide rail, a vertical telescopic rod, a cover, a camera and a fill light. The horizontal slide rail is set on the top of the grape position adjustment device through a bracket, the bottom of the horizontal slide rail is slidably connected to the vertical telescopic rod, the bottom of the vertical telescopic rod is connected to the cover, the camera is set on the inner top of the cover, the fill lights are set on both sides of the inside of the cover, the lower side of the cover corresponds to the opening of the supporting plate, and the camera is connected to the controller.

[0011] Preferably, the conveying device includes an active roller, a driven roller, a driving motor and a conveyor belt, the driving motor is connected to the controller, and the conveyor belt and the rotating sphere are both made of soft rubber material.

[0012] The present invention also provides a method for predicting the harvesting time of table grapes based on a deep learning model, comprising:

[0013] Step 100: Acquire hyperspectral images of table grapes at different maturity stages and physical and chemical index data representing the maturity degree of table grapes;

[0014] Step 200: performing data preprocessing operations on the hyperspectral image data, performing correlation analysis and screening on the physical and chemical index data, and establishing a table grape maturity data set;

[0015] Step 300: performing unsupervised classification on the physical and chemical index data representing the maturity degree of table grapes in the maturity data set based on a clustering algorithm, so as to realize the classification of grape maturity stages based on multiple indicators;

[0016] Step 400: Establishing a regression prediction model based on deep learning, and training it based on a table grape maturity data set, fitting the mapping relationship between spectral data and physical and chemical index data, and realizing regression prediction of physical and chemical indicators representing maturity through spectral data;

[0017] Step 500: Calculate the distance between the sample and the cluster center based on the sample physical and chemical indicators predicted by the regression prediction model, determine the maturity of the sample according to the principle of shortest distance, and determine the harvesting time of table grapes.

[0018] Preferably, in step 200, data preprocessing operations are performed on the hyperspectral image data, correlation analysis and screening are performed on the physical and chemical index data, and a table grape maturity data set is established, specifically:

[0019] Step 201: performing data preprocessing operations on the hyperspectral image data;

[0020] Perform black and white correction on the acquired hyperspectral image;

[0021] For the corrected hyperspectral image, the region of interest is selected and the average spectrum of the entire surface area of ​​each fruit sample is calculated;

[0022] Multivariate scattering correction and smoothing processing are performed on the grape spectrum information;

[0023] The first-order differential derivative of the smoothed spectral data is taken, and the spectral data is further normalized to ensure that spectra in different bands are on the same scale;

[0024] Step 202: performing correlation analysis and screening on the physical and chemical index data;

[0025] Grape physical and chemical indicators include soluble solids, pH value, color difference and hardness, and correlation analysis was conducted on the four indicators;

[0026] Step 203: Establishing a table grape maturity data set;

[0027] The preprocessed spectral data and physical and chemical index data are matched to form variable pairs and constitute the basic data set, which is then divided into a training set and a test set.

[0028] Preferably, in step 300, unsupervised classification is performed on the physical and chemical index data representing the maturity of table grapes in the maturity data set based on a clustering algorithm to achieve the classification of grape maturity stages based on multiple indicators, specifically:

[0029] Based on the data of four physical and chemical indicators, a clustering algorithm is used to obtain the best sample cluster division. Combined with the clustering result evaluation index, it is divided into four maturity stages, and the cluster center of each maturity stage is obtained to complete the division of the maturity stage of the grape samples.

[0030] Preferably, in step 400, a regression prediction model based on deep learning is established, and is trained based on a table grape maturity data set, and a mapping relationship between spectral data and physicochemical index data is fitted to achieve regression prediction of physicochemical indexes characterizing maturity through spectral data, specifically:

[0031] A regression prediction model based on deep learning is established, wherein the regression prediction model includes two one-dimensional convolutional layers, a skip connection module, two long short-term memory network layers, a self-attention mechanism layer and a fully connected regression prediction layer, wherein the two one-dimensional convolutional layers are used to extract local features of grape hyperspectral image data, and a skip connection is used after the second one-dimensional convolutional layer extracts features, and the outputs of the two convolutional layers are added and input into the first long short-term memory network layer to learn the long-term dependency between features, and the output of the first long short-term memory network layer is input into the second long short-term memory network layer to further capture the connection between features, and the output result of the second long short-term memory network layer is input into the self-attention mechanism layer, and the input features are processed using LKA and deformable convolution, and the long-distance dependency and deep features are further captured through the self-attention mechanism, and finally the learned features are input into the fully connected regression prediction layer, and a regression prediction model of features and predicted values ​​is established, and the grape spectral image data in the training set is used as input, and the physical and chemical index data is used as output to train the regression prediction model.

[0032] Preferably, the training loss function of the regression prediction model is:

[0033]

[0034] In the formula, y i is the true value, is the model prediction value, N is the number of samples, α is a parameter between 0 and 1, and λ is the regularization parameter. is the regularization term, j represents the weight index of different layers in the model, and θ represents the parameters of the model.

[0035] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0036] The present invention provides a system and method for predicting the harvesting time of table grapes based on a deep learning model, which has the following advantages:

[0037] (1) The present invention proposes a prediction method for the harvesting time of fresh grapes based on a deep learning model. Through the correspondence between spectral data and physical and chemical index data characterizing the maturity of grapes, a regression prediction model for fresh grape maturity index data based on deep learning is established. The collected grape hyperspectral image data is used to predict the true value using a reliable prediction model. A clustering algorithm is used to perform unsupervised classification on the physical and chemical index data characterizing the maturity of fresh grapes in the maturity data set, so as to realize the division of grape maturity stages based on multiple indicators; a loss function is constructed to train the model and update the prediction model. The loss function effectively prevents the model from overfitting by adding a regularization term, improves the generalization ability of the model, and makes its performance on new data more robust. In addition, the values ​​of and can be adjusted according to specific problems and data characteristics to adapt to different task requirements and data distributions. It has strong adaptability and improves the performance of the model in deep learning analysis of one-dimensional spectral data, so that it can better fit the data and improve the accuracy of prediction.

[0038] (2) The present invention proposes a prediction system for the harvesting time of fresh grapes based on a deep learning model. The system includes a hyperspectral image shooting device, which can obtain hyperspectral image information of fresh grapes at different maturity stages, thereby improving the accuracy of spectral information data.

[0039] (3) The image capturing device provided by the present invention includes a grape position adjusting device, wherein the grape position adjusting device adjusts the angle of the grapes located on top of the plurality of rotating spheres through the cooperation between the transmission device, the carrying plate, and the rotating spheres, thereby obtaining multi-angle spectral data of the same grape. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 This is a schematic diagram of the structure of a hyperspectral image shooting device;

[0042] Figure 2 It is a schematic diagram of the local structure of the hyperspectral image shooting device;

[0043] Figure 3 This is a schematic diagram of a method flow of an embodiment of the present invention;

[0044] Figure 4 It is a schematic diagram of the structure of the regression prediction model of the ripeness characterization index of fresh grapes of the present invention.

[0045] Figure numerals: 1. conveying device; 2. carrying plate; 3. vertical plate; 4. connecting shaft; 5. limiting wheel; 6. rotating sphere; 7. grapes; 8. arc-shaped positioning plate; 9. cover; 10. camera; 11. fill light; 12. vertical telescopic rod. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] The present invention provides a prediction system for the harvesting time of table grapes based on a deep learning model, comprising:

[0049] A hyperspectral image capturing device, a data acquisition device, a controller and a data processor, wherein the hyperspectral image capturing device is used to obtain hyperspectral image information of fresh grapes at different maturity stages; the data acquisition device is used to obtain physical and chemical index data of maturity of fresh grapes at different maturity stages; the controller is used to control the hyperspectral image capturing device and store the data acquired during the execution of the hyperspectral image capturing device and the data acquisition device in the memory of the controller; and the data processor is used to process the data in the memory.

[0050] like Figure 1 and Figure 2 As shown, the hyperspectral image shooting device includes a grape position adjustment device and an adjustable shooting device;

[0051] The grape position adjustment device includes a conveying device 1, a carrying plate 2, a limiting device and a rotating sphere 6. A plurality of the carrying plates 2 are evenly arranged on the top of the conveying device 1. The limiting devices are arranged on the four edges of the top of each of the carrying plates 2. The limiting devices include a vertical plate 3, a connecting shaft 4 and a limiting wheel 5. The vertical plate 3 is arranged on the four edges of the top of each of the carrying plates 2. The connecting shaft 4 is arranged on both sides of the long side of the vertical plate 3. The limiting wheel 5 is arranged on the connecting shaft 4. A plurality of circular holes are evenly arranged on the bottom of the carrying plate 2, and the circular holes are located between the four limiting wheels 5 between two adjacent limiting devices. The bottom of the rotating sphere 6 passes through the circular holes and contacts with the conveying device 1. The conveying device 1 is used to drive the rotating sphere 6 to rotate. The arc positioning plate 8 is arranged on the vertical plate 3 corresponding to the rotating sphere 6. The cooperation between the limiting wheel 5 and the arc positioning plate 8 can prevent the rotating sphere 6 from falling off. Grapes 7 are arranged on the top of the four rotating spheres 6. The conveying device 1 is connected to the controller.

[0052] The adjustable shooting device includes a horizontal slide rail, a vertical telescopic rod 12, a cover 9, a camera 10 and a fill light 11. The top of the grape position adjustment device is provided with the horizontal slide rail through a bracket, the bottom of the horizontal slide rail is slidably connected to the vertical telescopic rod 12, the bottom of the vertical telescopic rod 12 is connected to the cover 9, the camera 10 is arranged on the inner top of the cover 9, the fill light 11 is arranged on both sides of the inside of the cover 9, the lower side of the cover 9 corresponds to the opening of the supporting plate 2, the camera 10 is connected to the controller, the horizontal slide rail and the vertical telescopic rod are not limited in this application, and it is sufficient to be able to realize the functions of horizontal sliding and vertical telescopic.

[0053] The conveying device 1 includes an active roller, a driven roller, a driving motor and a conveyor belt. The driving motor is connected to the controller. The conveyor belt and the rotating sphere 6 are both made of soft rubber material. When in use, the driving motor drives the conveyor belt to rotate. The conveyor belt contacts the part of the rotating sphere that passes through the circular hole, driving the rotating sphere to rotate, thereby realizing the adjustment of the angle of the grapes 7 placed on the top of the multiple rotating spheres 6.

[0054] The grapes 7 with the stems removed are placed in a space surrounded by four rotating spheres 6, and the rotating spheres 6 are driven by a conveyor belt to rotate to adjust the angle of the grapes, and the camera 10 is used to capture multi-angle hyperspectral images of the grapes.

[0055] like Figure 3 As shown, the present invention also provides a method for predicting the harvesting time of table grapes based on a deep learning model, comprising:

[0056] Step 100: Acquire hyperspectral images of table grapes at different maturity stages and physical and chemical index data representing the maturity degree of table grapes;

[0057] Step 200: performing data preprocessing operations on the hyperspectral image data, performing correlation analysis and screening on the physical and chemical index data, and establishing a table grape maturity data set;

[0058] Step 300: performing unsupervised classification on the physical and chemical index data representing the maturity degree of table grapes in the maturity data set based on a clustering algorithm, so as to realize the classification of grape maturity stages based on multiple indicators;

[0059] Step 400: Establishing a regression prediction model based on deep learning, and training it based on a table grape maturity data set, fitting the mapping relationship between spectral data and physical and chemical index data, and realizing regression prediction of physical and chemical indicators representing maturity through spectral data;

[0060] Step 500: Calculate the distance between the sample and the cluster center based on the sample physical and chemical indicators predicted by the regression prediction model, determine the maturity of the sample according to the principle of shortest distance, and determine the harvesting time of table grapes.

[0061] The steps of the present invention are introduced in conjunction with specific embodiments:

[0062] In step 100, hyperspectral images of table grapes at different maturity stages and physical and chemical index data characterizing their maturity are obtained, specifically:

[0063] First, samples of table grapes from the color change period to the fully mature stage are obtained, which specifically includes: selecting grape trees to be sampled evenly and equidistantly in the vineyard, and evenly marking the grape bunches to be sampled at different parts of each sampling tree, which should include grape bunches under different lighting conditions; selecting 10 complete grape berries from the upper, middle and lower parts of each grape bunch, removing the stalks, and numbering each berry, and collecting the corresponding spectral data based on the hyperspectral imaging device;

[0064] The physical and chemical index data characterizing the degree of maturity are collected, specifically including: using a handheld colorimeter to measure the LAB value of three measuring points 180 degrees around the equator of each grape berry, and taking the average value after calculating the color difference; after obtaining the color difference data, the grape sample is subjected to a TPA test using a texture analyzer to obtain the surface pressure of the grape puncture as the grape hardness; the sample grape berries are squeezed into a test tube, and the pH value is obtained using a glass membrane electrode pH meter; after obtaining the pH value, the soluble solids value of the grape sample is measured using a handheld sugar meter.

[0065] In step 200, data preprocessing operations are performed on the hyperspectral image data, correlation analysis and screening are performed on the physical and chemical index data, and a fresh grape maturity data set is established, specifically:

[0066] Step 201: performing data preprocessing operations on the hyperspectral image data;

[0067] The acquired hyperspectral image is subjected to black and white correction to eliminate the noise caused by instrument current interference. The calculation formula is:

[0068]

[0069] Where R(x,λ) is the hyperspectral image after black and white correction, GI(x,λ) is the hyperspectral image before correction, DI(x,λ) is the black calibration plate data, and SI(x,λ) is the white calibration plate data;

[0070] For the corrected hyperspectral image, the region of interest was selected on the sample grape berries in the hyperspectral image to avoid unnecessary errors caused by uneven grape skin color during the color change period due to single-point acquisition, and the average spectral reflectance of each sample was calculated. Specifically, when collecting spectral data, the hyperspectral camera used a wavelength of 386.12-1026nm, and took 641 spectral bands between 450nm-950nm;

[0071] Multivariate scattering correction and smoothing are performed on the grape spectral information to eliminate the spectral differences caused by different scattering levels due to uneven sample surfaces and correct the baseline shift and offset of spectral data;

[0072] The first-order differential derivative of the smoothed spectral data is taken, and the spectral data is further normalized to ensure that spectra of different bands are on the same scale. Specifically:

[0073] The average value of all spectral data is taken as the baseline, and the calculation formula is as follows:

[0074]

[0075] Perform a univariate linear regression on the spectrum of each sample and the average spectrum, and solve the least squares problem to obtain the baseline shift and offset of each sample. The calculation formula is as follows:

[0076]

[0077] Correct each sample spectrum by subtracting the obtained baseline shift and dividing it by the offset to obtain the corrected spectrum. The calculation formula is as follows:

[0078]

[0079] The spectral data processed by multivariate scattering correction is subjected to SG smoothing to improve the smoothness of the spectrum and reduce the interference of noise. The calculation formula is as follows:

[0080]

[0081] The smoothed spectral data is subjected to first-order derivative processing to reduce low-frequency noise, amplify high-frequency noise, highlight the change details, and make the spectral features more significant. The calculation formula is as follows:

[0082]

[0083] Where g is the differential window width; x is the spectrum value before derivation; f'(x i ) is the spectral value after derivative;

[0084] The spectral data after derivative processing is normalized to the maximum and minimum values ​​to ensure that spectra of different bands are on the same scale in order to improve the data analysis effect. The calculation formula is as follows:

[0085]

[0086] In the formula, x normalized is the normalized spectral value, x i is the spectral value of the ith band of the spectral sequence, X max and X min are the maximum and minimum values ​​in the spectral sequence, respectively;

[0087] Step 202: performing correlation analysis and screening on the physical and chemical index data;

[0088] Grape physical and chemical indicators include soluble solids, pH value, color difference and hardness. In order to judge the maturity of fresh grapes, the four indicators are analyzed using the Spearson correlation analysis method to reveal the correlation between the indicators, eliminate the redundant indicators with strong correlation, and overcome the multicollinearity between the indicators. Specifically, they include:

[0089] The Spearman rank correlation coefficients were calculated for each of the four physical and chemical index values ​​obtained, and the calculation formula is as follows:

[0090]

[0091] Where ρ is the Spearman correlation coefficient; d i is the difference in the ranks of the corresponding variables, that is, the difference in the position of the variables when the two variables are sorted in pairs; n is the number of calculated variables;

[0092] The value range of the Spearman correlation coefficient is between -1 and 1. If the coefficient is close to 1, it means that there is a positive monotonic relationship between the variables; if the coefficient is close to -1, it means that there is a negative monotonic relationship between the variables; if the coefficient is close to 0, it means that there is no obvious monotonic relationship between the variables;

[0093] The calculated correlation coefficients between soluble solids and pH, hardness and color difference were the highest, which were 0.9, -0.81 and -0.96 respectively, with the greatest correlation. The other indicators also had high correlations, but there were still some differences.

[0094] Step 203: Establishing a table grape maturity data set;

[0095] The preprocessed spectral data and physical and chemical index data are matched to form variable pairs and constitute the basic data set, which is then divided into a training set and a test set in a ratio of 7:3.

[0096] In step 300, unsupervised classification is performed on the physical and chemical index data representing the maturity of table grapes in the maturity data set based on a clustering algorithm to achieve the classification of grape maturity stages based on multiple indicators, specifically:

[0097] Based on the four physical and chemical index data, the clustering algorithm is used to obtain the best sample cluster division. Combined with the clustering result evaluation index, it is divided into four maturity stages, and the cluster center of each maturity stage is obtained to complete the division of the maturity stage of the grape samples. Detailed introduction:

[0098] Clustering method is used to divide the grape samples in the table grape dataset into maturity stages. The calculation process is as follows:

[0099] The input is the table grape sample set D = {y1,y2,…,y m}The number of clusters k, where y i =(SSC i ,PH i ,hardness i , color difference i ), where i represents the i-th sample, m is the number of samples, and the sample index values ​​are preprocessed to ensure that each index is on the same scale. The standardization formula is as follows:

[0100]

[0101] In the formula, y ij is the jth index of the i-th sample, μ ij is the mean of the jth index, σ j is the standard deviation of the jth indicator. The standardized data is recorded as z ij ; Maximum number of iterations N, the output is the sample clusters of different maturity after division C = {C1, C2, ..., C k}, randomly select k samples from the data set D as the initial k centroid vectors: {c1,c2,…c k}, for n = 1, 2, ..., N, the clusters are divided into C t =φ,t=1,2,…k;

[0102] For i = 1, 2, ..., m, calculate the grape sample y i and the individual centroid vectors;

[0103] c j =(c j1 ,c j2 ,c j3 ,c j4 ),j=1,2,…,k:

[0104] Calculate the distance component of SSC: (z i1 -c j1 ), calculate the distance component of PH: (z i2 -c j2 ), calculate the distance component of hardness: (z i3 -c j3 ), calculate the distance component of the color difference: (z i4 -c j4 ), combine the distance components of the four indicators to obtain the total Euclidean distance sum of squares:

[0105]

[0106] Through this calculation method, the four indicators are comprehensively utilized to calculate the distance from each sample point to the cluster center. i The smallest one is d(y i ,c k ) corresponds to the category: λ i ,renew

[0107] For j = 1, 2, ... k, for C j Recalculate the new centroid for all sample points in

[0108] Until all k centroid vectors remain unchanged, the output cluster partition C = {C1, C2, ..., C k}.

[0109] The cluster analysis method uses the Calinski-Harabasz index to evaluate the clustering effect. In the fresh grape dataset, the clustering evaluation index value is higher when the grape samples are divided into four categories. The division result is more in line with production practice and more reasonably completes the division of maturity of comprehensive fresh grape multi-index data.

[0110] According to the sample characterization indicators of each category and combined with actual production, the four categories of samples are divided into four stages: early color change, mid-color change, early maturity and maturity.

[0111] In step 400, a regression prediction model based on deep learning is established, and it is trained based on the table grape maturity data set, and the mapping relationship between spectral data and physical and chemical index data is fitted to achieve regression prediction of physical and chemical indicators representing maturity through spectral data, specifically:

[0112] Establish a regression prediction model based on deep learning, where Figure 4 As shown, the regression prediction model includes 2 one-dimensional convolutional layers, 1 skip connection module, 2 long short-term memory network layers, 1 self-attention mechanism layer and 1 fully connected regression prediction layer. The two one-dimensional convolutional layers are used to extract local features of grape hyperspectral image data. After the second one-dimensional convolutional layer extracts features, a skip connection is used. The outputs of the two convolutional layers are added and input into the first long short-term memory network layer to learn the long-term dependency between features. The output of the first long short-term memory network layer is input into the second long short-term memory network layer to further capture the connection between features. The output of the second long short-term memory network layer is input into the self-attention mechanism layer. LKA and variable The convolution is processed, and the long-distance dependency and deep features are further captured through the self-attention mechanism. Finally, the learned features are input into the fully connected regression prediction layer to establish a regression prediction model of features and predicted values. The grape spectral image data in the training set is used as input, and the physical and chemical index data is used as output to train the regression prediction model. The entire modeling process is described in detail as follows: based on the convolutional neural network and long short-term memory network combined model, and combined with the actual situation of the spectrum of fresh grapes, by adding jump connections to the model to reduce the loss of data information in the extraction process, adding a self-attention layer to achieve more effective extraction of global features, the established model has high prediction ability and robustness;

[0113] Assume that X0 is an n×p dimensional spectral independent variable matrix:

[0114] X1=[x1,x2,…,x p ]

[0115] Y0 is an n-dimensional independent variable vector:

[0116] Y0=[y1 y2 … y n ] T

[0117] In the formula, p is the number of wavelengths of grape spectral data, and the total number of samples is 2155 fresh grape samples;

[0118] Create a data loader to load data in batches. Model training uses batch processing, and the batch size is set to 64.

[0119] The deep learning-based regression prediction model for table grape maturity characterization indicators includes 2 convolutional layers, 1 skip connection module, 2 long short-term memory network layers, 1 self-attention mechanism layer and 1 fully connected regression prediction layer. The details of the prediction model are as follows:

[0120] The model input layer accepts 64×641 table grape spectral data, representing 649 spectral bands of 64 grape samples in each batch;

[0121] After the input layer comes the first part of the model, which consists of two consecutive one-dimensional convolutional layers, a batch normalization layer, an activation function RELU layer, and a maximum pooling layer MaxPool1D. The parameters set for the two convolutional layers are as follows:

[0122] The first convolutional module: The first convolutional layer structure has an input channel of 1, an output channel of 16, and a kernel size of kernel_size = 2;

[0123] The input one-dimensional data is convoluted without padding to extract the one-dimensional features of the grape spectrum;

[0124] The parameters of the first batch layer are: the output channel is 16; batch normalization is performed on the first convolutional layer;

[0125] The first activation function layer: introduces the ReLU activation function for nonlinear activation to increase the model's expressiveness; the first Maxpool layer, kernel size kernel_size = 2, reduces the feature map scale;

[0126] The second convolution module further extracts features based on the extracted local features of the grape spectrum: the first convolution layer structure has 16 input channels, 32 output channels, and kernel size kernel_size = 18;

[0127] The output of the first convolution module is convolved; the parameters of the second batch layer are: the output channel is 32;

[0128] Apply batch normalization to the second convolutional layer;

[0129] The second activation function layer: introduces the ReLU activation function for nonlinear activation to increase the model's expressiveness; the second Maxpool layer, kernel size kernel_size = 2, reduces the feature map scale;

[0130] Adding a skip connection layer with 32 input and output channels and a convolution kernel size of 1, performing a jump transformation on the input of the second layer and adding it to the output features allows the model to learn the residual of some spectral information lost by the second layer of convolution and achieve optimized feature extraction.

[0131] After the convolution module extracts the local spectral data features, the features are input into the long short-term memory network layer. This module consists of two layers of LSTM to further realize the hidden features related to the characterization index data between the local features of the table grape spectrum, as follows:

[0132] The input size of the first layer LSTM is 149, the size of the second layer convolution output is 149, input_size=149, that is, the hidden neuron size is 17, a bidirectional long short-term memory network (bidirectional=True) is used to enhance learning ability, and the batch size batch_size=True is set to ensure the correct dimension of the input data.

[0133] The number of features output by LSTM is batch normalized, and the channel parameter is 32;

[0134] Input the output of the previous layer into the second LSTM layer;

[0135] The second layer LSTM input size input_size = 34, hidden_size = 17, the number of hidden neurons is 17, a bidirectional long short-term memory network (bidirectional = True) is used to enhance learning ability, and the batch size batch_size = True is set to ensure the correct dimension of the input data.

[0136] Add a dropout layer with a dropout rate of 0.2 to reduce some neurons to reduce overfitting and improve the generalization ability of the model;

[0137] The third part is the self-attention module, which introduces LKA and variable convolution to expand the receptive field, better capture the global features and complex dependencies in the spectral data of fresh grapes, and mine the factors causing the changes in the relevant spectra during the ripening period of grapes, thereby improving the prediction accuracy of the characterization indicators:

[0138] The input parameter is the output 34 of the previous layer, which accepts the input tensor to represent the input features. The features are first processed by the LKA module:

[0139] The input features undergo a convolution operation: F = Conv(X), where Conv is a depth-wise separable convolution.

[0140] Use depth-wise separable convolution and 1×1 convolution to construct a large convolution kernel, calculated as follows:

[0141] F LKA =Conv 1×1 (Conv DDW-D (Conv DDW (F)))

[0142] In the formula, Conv DDW-D and Conv DDW They are depth-wise separable dilated convolution and depth-wise separable convolution, Conv 1×1 is a 1×1 convolution. The deformable convolution adjusts the sampling grid by learning the offset so that it can better adapt to the shape of the data. When calculating the attention weight, the deformable convolution is used to adjust the feature F LKA To process:

[0143] Attention=Conv deform (F LKA )

[0144] In the formula, Conv deform is a deformable convolution operation. The offset of the deformable convolution can be learned from the features through additional convolutional layers, so that the convolution kernel can be adaptively adjusted according to the shape of the data and better capture the characteristics of the data.

[0145] Through query_layer, key_layer and value_layer, the processed features are linearly transformed to obtain query vector, key vector and value vector. The calculation method is as follows:

[0146] Q=(Attention)·W q +b q

[0147] K=(Attention)·W k +b k

[0148] V=(Attention)·W V +b V

[0149] In the formula, Attention = Conv deform (F LKA ) is the feature after LKA and deformable convolution processing, W q , W k and W V is the learned weight matrix, corresponding to the linear transformation of Query, Key and Value, respectively, q 、b k and b v is the bias term, Q, K and V represent the obtained Query, Key and Value respectively;

[0150] The attention score is obtained by multiplying the transpose of the query and the key. The calculation formula is as follows:

[0151] attention_scores = Q·KT

[0152] Where Q and K T Represents Query and Key, K T represents the transpose of the key, and the scaled attention score is obtained by dividing the attention score by the square root of the number of feature channels. The calculation formula is as follows:

[0153]

[0154] Where in_channels is the number of channels of the input feature. This value is used to scale the attention score and then calculate the attention weight. The calculation formula is as follows:

[0155] attention_weights=softmax(attention_scores_scaled)

[0156] The scaled attention scores are normalized using the softmax function to obtain the attention weights.

[0157] Applying dropout to the attention weights helps reduce overfitting and improve the generalization ability of the model, with a dropout rate of 0.5.

[0158] Furthermore, the attention weight is used to perform weighted summation on the value vector value to obtain the self-attention output attention_output. The calculation formula is as follows:

[0159] attention_output=attention_weights·value

[0160] Where attention_weights is the attention weight tensor of size (N, T, T'), which represents the attention weight of each position to all positions, and value is the value vector tensor of size (N, T', C), which represents the features of each position;

[0161] N represents the batch size, T represents the length of the input sequence, T' represents the length of the output sequence, and C represents the feature dimension of each position;

[0162] Furthermore, the important features output by the self-attention layer are obtained.

[0163] Furthermore, the output is input to batch_norm2 with 32 channels in batch normalization layer, where the parameter channel number is 32 for normalization to improve the model stability.

[0164] Furthermore, the output is transformed and flattened into a one-dimensional vector for input into the fully connected layer;

[0165] Furthermore, the dropout operation is applied to the flattened features with a dropout rate of 0.2, which reduces some neurons to reduce overfitting and improve the generalization ability of the model.

[0166] The last part is the output layer. The output feature vector is input into the fully connected layer with an input parameter of 1088. The output vector y is transformed to a dimension of (64, 1), which represents the model output of each batch of samples.

[0167] The training loss function of the regression prediction model is:

[0168]

[0169] In the formula, y i is the true value, is the model prediction value, N is the number of samples, α is a parameter between 0 and 1, and λ is the regularization parameter. is the regularization term, j represents the weight index of different layers in the model, and θ represents the parameters of the model.

[0170] This loss function measures the difference between the predicted value and the true value more comprehensively. By adding regularization terms, the model can be effectively prevented from overfitting, the generalization ability of the model can be improved, and its performance on new data can be more robust. And the values ​​of and can be adjusted according to specific problems and data characteristics to adapt to different task requirements and data distributions, which has strong adaptability. The advantages of the two loss functions are comprehensively utilized to improve the performance of the model in the deep learning analysis of one-dimensional spectral data, so that it can better fit the data and improve the accuracy of prediction.

[0171] Adam was used as the parameter optimizer during model training, with the initial learning rate set to 0.005;

[0172] The learning rate scheduler ReduceLROnPlateau is used to adjust the learning rate, where factor = 0.5, indicating that each adjustment is reduced to 0.5 times the original value, and eps = 1e-06 to prevent the learning rate from being too small, thereby effectively updating the model parameters and controlling the change of the learning rate during training.

[0173] Furthermore, after the model was trained on the training set, evaluation indicators were used to evaluate the model's prediction results on the training set and test set.

[0174] Specifically, in this embodiment, in order to measure the prediction accuracy of the model, the root mean square error RMSE is used to measure the average difference between the model prediction value and the true value. The calculation formula of this indicator is as follows:

[0175]

[0176] Where n is the number of samples, y i is the true value, y i It is the model prediction value. The smaller the RMSE value, the better the model prediction effect. It is used as a loss function to improve the fitting ability.

[0177] Specifically, when evaluating the robustness of the model, the average difference between the model prediction value and the actual value is measured, and the relative standard deviation RPD is introduced. The calculation method of this indicator is as follows:

[0178]

[0179] Where X1 and X2 are the average values ​​of two groups of data, and n is the number of test data samples. The model effect is relatively robust when RPD≥2.5.

[0180] Specifically, in terms of evaluating the goodness of fit of the model, R is introduced 2 , which corresponds to:

[0181]

[0182] In the formula, y i is the true value, y i is the model prediction value, y is the average value of the target value, and n is the number of samples;

[0183] In this example, the prediction of soluble solids in the test set has an RMSE as low as 0.6, R 2 It reaches 0.96 and RPD can reach 5, indicating that the model prediction effect is very good.

[0184] In step 500, the distance between the sample and the cluster center is calculated according to the physical and chemical indicators of the sample predicted by the regression prediction model, and the maturity of the sample is determined according to the principle of shortest distance, so as to determine the harvesting time of the table grapes, specifically:

[0185] The four fresh grape maturity characterization indicators predicted by the deep learning model regression for the tested samples, and then the distances to the four cluster centers are calculated.

[0186] According to the principle of shortest distance, the ripe stage of the fresh grapes with test samples is determined, and then the harvesting time of fresh grapes is further determined according to actual application needs. If the grapes are in the ripe stage, it can be determined that it is time to harvest, and the grapes can be harvested.

[0187] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0188] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A prediction system for harvesting time of table grapes based on a deep learning model, characterized in that: include: A hyperspectral image capturing device, a data acquisition device, a controller and a data processor, wherein the hyperspectral image capturing device is used to obtain hyperspectral image information of fresh grapes at different maturity stages; the data acquisition device is used to obtain physical and chemical index data of maturity of fresh grapes at different maturity stages; the controller is used to control the hyperspectral image capturing device and store the data acquired during the execution of the hyperspectral image capturing device and the data acquisition device in the memory of the controller; and the data processor is used to process the data in the memory.

2. The system according to claim 1, characterized in that The hyperspectral image shooting device includes a grape position adjustment device and an adjustable shooting device; The grape position adjustment device comprises a conveying device, a carrying plate, a limiting device and a rotating sphere. A plurality of the carrying plates are evenly arranged on the top of the conveying device. The limiting devices are arranged on the four edges of the top of each of the carrying plates. The limiting devices comprise a vertical plate, a connecting shaft and a limiting wheel. The vertical plates are arranged on the four edges of the top of each of the carrying plates. Connecting shafts are arranged on both sides of the long sides of the vertical plates. The limiting wheels are arranged on the connecting shafts. A plurality of circular holes are evenly arranged on the bottom of the carrying plates, and the circular holes are located between the four limiting wheels between two adjacent limiting devices. The bottom of the rotating sphere contacts the conveying device through the circular holes. The conveying device is used to drive the rotating sphere to rotate. The arc-shaped positioning plates are arranged on the vertical plates corresponding to the rotating spheres. The conveying device is connected to the controller. The adjustable shooting device includes a horizontal slide rail, a vertical telescopic rod, a cover, a camera and a fill light. The horizontal slide rail is set on the top of the grape position adjustment device through a bracket, the bottom of the horizontal slide rail is slidably connected to the vertical telescopic rod, the bottom of the vertical telescopic rod is connected to the cover, the camera is set on the inner top of the cover, the fill lights are set on both sides of the inside of the cover, the lower side of the cover corresponds to the opening of the supporting plate, and the camera is connected to the controller.

3. The system according to claim 2, characterized in that The conveying device comprises an active roller, a driven roller, a driving motor and a conveying belt. The driving motor is connected to the controller. The conveying belt and the rotating sphere are both made of soft rubber material.

4. A method for predicting the harvesting time of table grapes based on a deep learning model, characterized in that: include: Step 100: Acquire hyperspectral images of table grapes at different maturity stages and physical and chemical index data representing the maturity degree of table grapes; Step 200: performing data preprocessing operations on the hyperspectral image data, performing correlation analysis and screening on the physical and chemical index data, and establishing a table grape maturity data set; Step 300: performing unsupervised classification on the physical and chemical index data representing the maturity degree of table grapes in the maturity data set based on a clustering algorithm, so as to realize the classification of grape maturity stages based on multiple indicators; Step 400: Establishing a regression prediction model based on deep learning, and training it based on a table grape maturity data set, fitting the mapping relationship between spectral data and physical and chemical index data, and realizing regression prediction of physical and chemical indicators representing maturity through spectral data; Step 500: Calculate the distance between the sample and the cluster center based on the sample physical and chemical indicators predicted by the regression prediction model, determine the maturity of the sample according to the principle of shortest distance, and determine the harvesting time of table grapes.

5. The method according to claim 4, characterized in that In step 200, data preprocessing operations are performed on the hyperspectral image data, correlation analysis and screening are performed on the physical and chemical index data, and a fresh grape maturity data set is established, specifically: Step 201: performing data preprocessing operations on the hyperspectral image data; Perform black and white correction on the acquired hyperspectral image; For the corrected hyperspectral image, the region of interest is selected to calculate the average spectrum of the entire surface area of ​​each fruit sample; Multivariate scattering correction and smoothing processing are performed on the grape spectrum information; The first-order differential derivative of the smoothed spectral data is taken, and the spectral data is further normalized to ensure that spectra in different bands are on the same scale; Step 202: performing correlation analysis and screening on the physical and chemical index data; Grape physical and chemical indicators include soluble solids, pH value, color difference and hardness, and correlation analysis was conducted on the four indicators; Step 203: Establishing a table grape maturity data set; The preprocessed spectral data and physical and chemical index data are matched to form variable pairs and constitute the basic data set, which is then divided into a training set and a test set.

6. The method according to claim 5, characterized in that In step 300, unsupervised classification is performed on the physical and chemical index data representing the maturity of table grapes in the maturity data set based on a clustering algorithm to achieve the classification of grape maturity stages based on multiple indicators, specifically: Based on the data of four physical and chemical indicators, a clustering algorithm is used to obtain the best sample cluster division. Combined with the clustering result evaluation index, it is divided into four maturity stages, and the cluster center of each maturity stage is obtained to complete the division of the maturity stage of the grape samples.

7. The method according to claim 5, characterized in that In step 400, a regression prediction model based on deep learning is established, and it is trained based on the table grape maturity data set, and the mapping relationship between spectral data and physical and chemical index data is fitted to achieve regression prediction of physical and chemical indicators representing maturity through spectral data, specifically: A regression prediction model based on deep learning is established, wherein the regression prediction model includes two one-dimensional convolutional layers, a skip connection module, two long short-term memory network layers, a self-attention mechanism layer and a fully connected regression prediction layer, wherein the two one-dimensional convolutional layers are used to extract local features of grape hyperspectral image data, and a skip connection is used after the second one-dimensional convolutional layer extracts features, and the outputs of the two convolutional layers are added and input into the first long short-term memory network layer to learn the long-term dependency between features, and the output of the first long short-term memory network layer is input into the second long short-term memory network layer to further capture the connection between features, and the output result of the second long short-term memory network layer is input into the self-attention mechanism layer, and the input features are processed using LKA and deformable convolution, and the long-distance dependency and deep features are further captured through the self-attention mechanism, and finally the learned features are input into the fully connected regression prediction layer, and a regression prediction model of features and predicted values ​​is established, and the grape spectral image data in the training set is used as input, and the physical and chemical index data is used as output to train the regression prediction model.

8. The method according to claim 7, characterized in that The training loss function of the regression prediction model is: In the formula, y i is the true value, is the model prediction value, N is the number of samples, α is a parameter between 0 and 1, and λ is the regularization parameter. is the regularization term, j represents the weight index of different layers in the model, and θ represents the parameters of the model.

Images