Multi-source and multi-level data fusion method applied to crop phenotypic parameter inversion

Abstract

The present invention provides a multi-source and multi-level data fusion method applied to the inversion of crop phenotypic parameters. The specific steps of the multi-source and multi-level data fusion method are as follows: Step 5: Extract the characteristic information of crop leaves and canopy scales using hyperspectral data to construct a feature vector for subsequent fusion modeling; Step 7: Extract the characteristic information of crop leaves and canopy scales using chlorophyll fluorescence data to construct a feature vector for subsequent fusion modeling; The present invention extracts the feature vectors of the information obtained by different sensors, and uses a three-layer improved fusion method to substitute the data into different machine learning algorithms for training and verification. Finally, the ensemble learning algorithm is used to output the final result, which can not only ensure the accuracy but also prevent overfitting, greatly improving the stability and accuracy of crop phenotypic parameter monitoring and diagnosis, providing agronomic monitoring services for farmers, and providing technical support for crop disaster prevention and control and production cultivation.

Description

Technical Field

[0001] The present invention belongs to the technical field of crop growth detection, and relates to a multi-source and multi-level data fusion method applied to the inversion of crop phenotypic parameters. Background Art

[0002] The traditional method for monitoring crop-related phenotypic parameters will consume a large amount of time and energy. In addition to causing chemical pollution, it will also cause devastating damage to crops, which is not conducive to plant growth and long-term repetitive monitoring, and it is difficult to meet the real-time, fast and reliable monitoring standards required by precision agriculture.

[0003] Therefore, when inverting crop phenotypic parameters, there is an urgent need for a technology that can make up for the deficiencies of traditional monitoring methods and at the same time effectively and accurately diagnose the physiological and biochemical status of crops. In recent years, the development of RGB images, hyperspectral remote sensing technology and chlorophyll fluorescence kinetics technology has provided a direction for this; accurately obtaining the status of crop phenotypic parameters through convenient and fast means has become the key research direction of relevant researchers in recent years. The corresponding features of the image can be extracted to discover the subtle changes in crop phenotypes for inversion; hyperspectral remote sensing technology has the characteristics of high resolution and strong continuity between bands, so it can detect many information that cannot be detected on the surface; compared with hyperspectral reflection signals, chlorophyll fluorescence parameters are closely related to the physiological and ecological changes and photosynthesis of crops. Existing research has shown that the fluorescence signal has occurred before the chlorophyll content decreases, which can indicate the growth status of crops in advance, can essentially explain the changes in crop physiology and biochemistry, and can better monitor the growth status of crops, especially the state of crops under environmental stress. In summary, combining RGB images, hyperspectral remote sensing and chlorophyll fluorescence monitoring technology to achieve accurate and rapid acquisition and monitoring of crop phenotypic parameters is the key to solving the efficient use and reasonable input of fertilizers, thereby reducing cultivation costs and increasing crop yields. The present invention takes field crops as the research object, combines the spatio-temporal distribution laws of crop-related phenotypic parameters at different leaf positions and different growth stages, explores the response mechanisms of hyperspectral, RGB image and chlorophyll fluorescence characteristics, and establishes a multi-source and multi-level data fusion method applied to the inversion of crop phenotypic parameters to further improve the timeliness and accuracy of monitoring and diagnosing the physiological and biochemical parameters of field crops. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-source and multi-level data fusion method applied to the inversion of crop phenotypic parameters to solve the problems raised in the above background art.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A multi-source and multi-level data fusion method applied to the inversion of crop phenotypic parameters, and the specific steps of the multi-source and multi-level data fusion method are as follows:

[0007] Step 1: Obtain digital images at the scale of crop leaves and canopy;

[0008] Step 2: Segment the region of interest in the digital images to separate the leaves and canopy areas to be monitored from the background;

[0009] Step 3: Convert the digital images into different color spaces, and extract color, texture, and morphological features at the scale of crop leaves and canopy to construct feature vectors for subsequent fusion modeling;

[0010] Step 4: Obtain hyperspectral data at the scale of crop leaves and canopy;

[0011] Step 5: Use the hyperspectral data to extract feature information at the scale of crop leaves and canopy to construct feature vectors for subsequent fusion modeling;

[0012] Step 6: Obtain chlorophyll fluorescence data at the scale of crop leaves and canopy;

[0013] Step 7: Use the chlorophyll fluorescence data to extract feature information at the scale of crop leaves and canopy to construct feature vectors for subsequent fusion modeling;

[0014] Step 8: Substitute the feature vectors extracted from images, spectra, and fluorescence into the multi-source and multi-level data fusion method to accurately invert crop phenotypic parameters.

[0015] In the above multi-source and multi-level data fusion method applied to the inversion of crop phenotypic parameters, in the first step, the specific operation of obtaining digital images at the scale of crop leaves and canopy is as follows:

[0016] Apply mobile phones and digital cameras. When shooting, set the camera to auto white balance. Under sunny weather, shoot samples at a fixed height from the crop leaves and canopy and at a 90° angle to the ground. It is necessary to assist with a standard color card for color correction in order to minimize the influence of light and the model of the shooting equipment on the images to the greatest extent.

[0017] In the above multi-source and multi-level data fusion method applied to the inversion of crop phenotypic parameters, the specific operation of the third step includes the following steps:

[0018] Step 1: Image color space conversion;

[0019] Step 2: Extract the color features of the image;

[0020] Step 3: Extract the texture features of the image;

[0021] Step 4: Extract the morphological features of the image;

[0022] Step 5: Screen the image-based feature vectors for subsequent modeling.

[0023] In the above-mentioned multi-source and multi-level data fusion method for crop phenotypic parameter inversion, the specific operations of the fourth step are as follows:

[0024] A portable ground feature spectrometer was used to collect crop hyperspectral data. The spectrometer's built-in leaf clip and built-in light source were used for data collection. The average value of three repeated measurements was taken for each leaf or canopy measurement point as the spectral value of that point. Whiteboard calibration was performed before measuring different leaves or canopies.

[0025] In the above-mentioned multi-source and multi-level data fusion method for inversion of crop phenotypic parameters, in the seventh step, the method for screening characteristic chlorophyll fluorescence parameters is: analyzing the leaf or canopy chlorophyll fluorescence parameter data and the phenotypic parameters of the crop to be tested, using the Duncan method to analyze the significance of the differences, and then performing correlation regression analysis on the chlorophyll fluorescence parameters and the parameters to be tested, and selecting the fluorescence parameters with good correlation with the crop phenotypic parameters.

[0026] Compared with the existing technology, the advantages of the multi-source and multi-level data fusion method applied to crop phenotypic parameter inversion in the present invention are:

[0027] 1. Currently, crop monitoring still commonly uses single digital images, single reflectance spectral data, or single chlorophyll fluorescence data to invert crop phenotypic parameters. This invention addresses the problems of traditional crop phenotypic monitoring and diagnosis methods, such as high destructiveness, limited information acquisition from single sensor monitoring, and low inversion accuracy. It proposes an improved multi-source and multi-level data fusion method for accurately inverting crop phenotypic parameters using digital images, fluorescence, and spectral feature information. This method can effectively improve the accuracy of inverting phenotypic parameters at the plant leaf or canopy scale, laying the foundation for the development of precision agriculture.

[0028] 2. Most of the existing fusion technologies are multi-feature fusion, which is a low-level fusion method. It only cascades different data together for modeling and prediction. Although the fusion effect is improved, the fusion data is redundant and the modeling effect is not ideal. The patent of this invention extracts feature vectors from the information obtained by different sensors, and uses a three-layer improved fusion method to substitute the data into different machine learning algorithms for training and verification. Finally, an integrated learning algorithm is used to output the final result, which can not only ensure accuracy but also prevent overfitting, greatly improving the stability and accuracy of crop phenotypic parameter monitoring and diagnosis, providing farmers with agricultural monitoring services, and providing technical support for disaster prevention and control and production cultivation of crops. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the flow chart of feature-level fusion of this method;

[0030] Figure 2 This is the flow chart of decision-level fusion of this method;

[0031] Figure 3 This is the flowchart of the mixing and fusion of this method. Specific implementation mode

[0032] The following are specific embodiments of the present invention and, in combination with the accompanying drawings, the technical solutions of the present invention are further described, but the present invention is not limited to these embodiments.

[0033] The first step: Obtain digital images at the scale of crop leaves or canopy.

[0034] Use a mobile phone or digital camera, and set the camera to automatic white balance during shooting. Under sunny weather, shoot and sample at a fixed height from the crop leaves or canopy and at a 90° angle to the ground; it is necessary to assist with a standard color card for color correction in order to minimize the influence of light and the model of the shooting equipment on the image to the greatest extent.

[0035] The second step: Segment the region of interest in the digital image, and segment the leaf or canopy area to be monitored from the background.

[0036] There are mainly the following several methods for segmenting the leaf or canopy area of the digital image, and the optimal selection can be made according to the actual application situation, in combination with the segmentation accuracy and running time.

[0037] (1) Otus algorithm: The maximum inter-class variance method, also known as the Otsu method, is a method for automatically obtaining the threshold that is self-suitable for the bimodal situation. This method divides the image into a background and an object according to the gray level characteristics of the image. The greater the difference between the background and the object, the greater the difference between the two parts that make up the image. Suppose the gray level of the gray image is H, then the gray level range is [0, H-1]. The optimal segmentation threshold of the image is calculated by the maximum inter-class variance method, and the expression is as follows:

[0038] t = Max[W0(t) * (u0(t) - u) 2 + w1(t) * (u1(t) - u) 2

[0039] Among them, t represents the optimal segmentation threshold, W0 represents the background ratio of the image, w1 represents the foreground ratio of the image, u0 represents the background mean value, and u1 represents the foreground mean value.

[0040] ​(2) K-means clustering algorithm: The k-means clustering algorithm is an iterative clustering analysis algorithm. Its steps are as follows: divide the prediction data into k groups, then randomly select k objects as the initial clustering centers, and then calculate the distance between each object and each seed clustering center, and assign each object to the clustering center closest to it. The basic idea of this algorithm is: perform clustering with k points in space as the centers. The value of k is generally determined artificially according to the specific research content. After the value of k is determined, classify the objects closest to them, and then through an iterative method, gradually update the values of each clustering center until the most ideal clustering result is obtained.

[0041] (3) Excess Green algorithm: Researchers utilized the spectral characteristics of green crops, which have a high reflectance in the green channel and low reflectance in the red and blue channels in the visible light band, to construct a series of vegetation indices such as the Excess Red Index (EXR), Normalized Difference Vegetation Index (NDI), Excess Green Index (EXG), and Vegetation Index Combination (COM) based on R, G, and B, to enhance the difference between the crop itself and its background and effectively separate the green crops from their background. In this study, the Excess Green Index (EXG) was used for cotton leaf image segmentation, and its formula is shown as follows.

[0042] EXG = 2G - R - B

[0043] (4) Deep learning algorithm: The image segmentation steps based on deep learning mainly include: making the dataset, calibrating the region of interest, building a convolutional neural network, training the image data, and testing the results.

[0044] The third step: Convert the digital image into different color spaces, and extract the color, texture, and morphological features at the crop leaf or canopy scale to construct a feature vector for subsequent fusion modeling.

[0045] 1. Image color space conversion

[0046] Convert the RGB image of the leaf or canopy into HSV, La*b*, YCrCb, and YIQ color space models. The color space conversion formulas are shown in Table 1.

[0047] Table 1 Review of color spaces involved in this method

[0048]

[0049]

[0050] 2. Extract the color features of the image

[0051] A Stricker and M Orengo proposed the method of color moments, which includes the first-order moment (mean), the second-order moment (variance), and the third-order moment (skewness). Since color information is mainly distributed in the low-order moments, the first-order, second-order, and third-order moments are sufficient to express the distribution of color information in the target image.

[0052] The calculation methods of three types of color moments are shown in Table 2:

[0053] Table 2 Summary of Calculation Methods of Three Types of Color Moments

[0054]

[0055] Among them, p i，j represents the i-th color component of the j-th pixel in the leaf or canopy image, and N represents the number of pixels in the image. Taking the HSV color space as an example, the color feature vector extracted from the image by this method is a 9-dimensional histogram vector composed of the first three color moments, which is expressed as follows:

[0056]

[0057] Extract the 9-dimensional histogram vector of each color space as the color feature vector of the image.

[0058] 3. Extract the texture features of the image

[0059] Four texture features calculated from four different angles (0°, 45°, 90°, and 135°) based on the gray-level co-occurrence matrix (GLMC) were selected from the obtained leaf or canopy images, and the mean (mean) and variance (sd) of each angle of the four texture features were calculated. The three bands of the RGB image were calculated as gray values before calculating the texture features (Table 3). The meanings of each texture feature are as follows:

[0060] (1) Second-order moment: Represents the change in the image energy value, reflecting the uniformity of the image gray value distribution and the texture fineness. When all pixel gray values in the image are the same, the energy value is 1:

[0061] (2) Entropy: Reflects the complexity of the gray value distribution in the image. The larger the Ent value, the more complex the pixel distribution in the image and the more dispersed the distribution of the same elements:

[0062] (3) Contrast: Reflects the clarity and texture depth of the image. The deeper the texture, the larger the Con, the clearer the image, and the greater the change in gray values between pixels:

[0063] (4) Autocorrelation: Reflects the predictable linear relationship between the gray values of two adjacent pixels within the window. The larger the Cor, the greater the predictability between pixels and the more uniform the gray values:

[0064] Table 3 Extracted texture features and extraction angles

[0065]

[0066] 4. Extract morphological features of the image

[0067] Morphological features are another important feature in images. Unlike low-level features such as color and texture, the description of morphological features requires the segmentation of objects or regions within the image. Morphological feature representation methods can be divided into two categories: one based on contour features, typically the Fourier descriptor method; the other based on region features, typically the shape-independent moment method. Contour features only use the object's boundaries, while region features consider the entire shape region. The leaf area of a crop leaf measured on this basis should also be considered an important morphological feature.

[0068] 5. Screening image-based feature vectors for subsequent modeling

[0069] Correlates the color, texture, and morphological feature vectors extracted in steps 2, 3, and 4 with the crop's surface parameters to be diagnosed. Significantly correlated variables are selected for subsequent modeling. Alternatively, select features using SelectFromModel, which includes L1-based feature selection, random forest model feature selection, and tree-based feature selection. Alternatively, integrate the feature selection process into the pipeline.

[0070] Step 4: Obtain hyperspectral data at the crop leaf or canopy scale.

[0071] A portable spectrometer was used to collect crop hyperspectral data using the spectrometer's built-in leaf clip and built-in light source. The spectral value for each leaf or canopy measurement point was calculated as the average of three replicates. Whiteboard calibration was performed before measuring different leaves or canopies.

[0072] Step 5: Use hyperspectral data to extract characteristic information at the crop leaf or canopy scale to construct feature vectors for subsequent fusion modeling.

[0073] The homogeneity and redundancy of full-band spectral data are not conducive to the extraction of crop nitrogen characteristics and affect the accuracy of model monitoring. In order to effectively improve the representativeness of spectral data, the extracted characteristic parameters must have a strong correlation with crop nitrogen. The method for screening characteristic spectral parameters is as follows:

[0074] (1) The Stochastic Frogs Leaping Algorithm (SFLA) sets a normal distribution space for variables and pre-selects a group of initial variables. Variables within the initial variable set are continuously selected to form a candidate subset. The number of variables in the candidate subset repeats with the increase or decrease in the number of times. Depending on the number of loops each time, the higher the correlation between variables, the greater the likelihood of their appearance in the candidate subset. The bands corresponding to the variable numbers selected according to the sample frequency sorting are chosen as the characteristic wavelengths.

[0075] (2) For the Successive Projections Algorithm (SPA), first, assume the maximum number of variables are selected from all the original variables. Then, through vector projection, non-collinear vectors and high-projection vectors are continuously selected as the starting vectors. Finally, confidence analysis is performed on the selected vector set, and the value at which the RMSE stabilizes and is minimized is the optimal number of selected variables. The characteristic wavelengths are selected according to the output results.

[0076] (3) Grey relational analysis (GRA) is an evaluation method in grey system theory. It evaluates the target through the correlation between indicators. Its essence is to linearly interpolate discrete data to make the data continuous, and then compare the geometric characteristics of the data. The more similar the geometric characteristics, the higher the correlation between the two. Finally, the degree of correlation is evaluated.

[0077] (4) Competitive Adaptive Reweighted Sampling (CARS) is a characteristic wavelength selection algorithm based on the partial least squares regression (PLSR) coefficient and Monte Carlo sampling. CARS first selects calibration set samples through Monte Carlo sampling, establishes the corresponding PLSR model, calculates the absolute weights of the wavelength regression coefficients for this sampling, and excludes the wavelength variables with smaller absolute values. On this basis, adaptive reweighted sampling is performed on the remaining wavelengths to select wavelengths, and the corresponding machine learning model is established. The selected characteristic wavelengths correspond to the machine learning model with the lowest wavelength RMSE CV of the machine learning model.

[0078] Step 6: Obtain chlorophyll fluorescence data at the leaf or canopy scale of crops.

[0079] Leaf scale: Use the MultispeQ, a multi-functional plant measuring instrument, to measure the fluorescence parameters of crop leaves while acquiring spectral data. The measurement time is selected as 12:00 - 16:00 Beijing time (light adaptation) and 00:00 - 05:00 (dark adaptation) on sunny and cloudless days. The measured leaf position is the typical functional leaf. Parameters such as Fm', qP, qN, Fm, and Fv / Fm are selected from the measurement results for data processing (Table 4).

[0080] Table 4 Chlorophyll fluorescence parameters

[0081]

[0082]

[0083] Canopy scale: An index representing the comprehensive chlorophyll fluorescence of the canopy was constructed according to the calculation method of canopy chlorophyll, and the relationship with the canopy reflectance hyperspectrum was analyzed using this index, significantly improving the accuracy of vegetation index in monitoring canopy chlorophyll fluorescence. The calculation formula for the canopy chlorophyll fluorescence kinetic parameters is as follows:

[0084] CCFC = 1stLD × 1stLCFC + 2ndLD × 2ndLCFC + 3rdLD × 3rdLCFC +....

[0085]

[0086] CCFD is the canopy chlorophyll fluorescence density; 1stLD, 2ndLD, 3rdLD, etc. are the weights of the first leaf, the second leaf, and the third leaf of the typical leaf, etc., with the unit of kg / m2; 1stLCFC, 2ndLCFC, 3rdLCFC, etc. correspond to Fv / Fm of different typical leaves respectively.

[0087] Step 7: Use the chlorophyll fluorescence data to extract the characteristic information at the crop leaf or canopy scale to construct a feature vector for subsequent fusion modeling.

[0088] There are numerous chlorophyll fluorescence parameters, which leads to a large amount of analysis work, making it difficult to select parameters with good correlation and affecting the accuracy of model monitoring. The method for screening characteristic chlorophyll fluorescence parameters is as follows: Analyze the data of leaf or canopy chlorophyll fluorescence parameters and the phenotypic parameters of the crop to be measured, use the Duncan method to analyze the significance of differences, and then conduct a correlation regression analysis between the chlorophyll fluorescence parameters and the parameters to be measured, and select the fluorescence parameters with good correlation with the crop phenotypic parameters.

[0089] Step 8: Substitute the feature vectors extracted from the image, spectrum, and fluorescence into the multi-source and multi-level data fusion method for accurately inverting the crop phenotypic parameters.

[0090] 1. Inverting crop phenotypic parameters by multi-source feature-level fusion

[0091] Reference Figure 1 , Feature-level fusion is carried out at the feature level, that is, features are extracted from different data sources. The data sources of this method are digital images, chlorophyll fluorescence, and hyperspectral information. The feature vectors extracted from the above data sources are feature-connected before the final regression prediction and fused using the cascading method to obtain the final inversion result. Among them, the image features, chlorophyll fluorescence features, and hyperspectral features that have been extracted are fused using the cascading fusion function, and the features of each sensor are fused and stacked. Finally, a suitable machine learning regression model is selected for prediction analysis; thus, the final results of the corresponding crop phenotypic parameters are obtained.

[0092] 2. Multi-source decision-level fusion inversion of crop phenotypic parameters

[0093] Reference Figure 2 , the decision-level fusion module uses digital image features, spectral features, and chlorophyll fluorescence features to separately model the results in three dimensions with the crop phenotypic parameters to be measured, and makes decisions on the results, realizing the matching from picture data to text data.,

[0094] First, the features extracted from the image, chlorophyll fluorescence, and hyperspectral data are respectively input into the fully connected layer and the classification layer to separately conduct the modeling prediction evaluation of the initial machine learning regression model. The prediction results of the three modalities are substituted into the ensemble learning algorithm for regression prediction again, and the final inversion result of the crop phenotypic parameters is obtained.

[0095] 3. Multi-source hybrid fusion inversion of crop phenotypic parameters

[0096] Reference Figure 3 , the hybrid fusion is similar to the feature-level fusion. It inputs the feature-level fusion result and the decision-level fusion result into the ensemble learning algorithm for secondary regression prediction to obtain the final prediction result of the crop phenotypic parameters of the hybrid fusion.

[0097] The content not described in detail in this specification belongs to the prior art well-known to those skilled in the art. The specific embodiments described herein are only illustrative of the spirit of the present invention. Those skilled in the art of the present invention can make various modifications or supplements to the described specific embodiments or use similar methods to replace them, but will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A multi-source and multi-level data fusion method applied to the inversion of crop phenotypic parameters, characterized in that The specific steps of the multi-source and multi-level data fusion method are as follows: The first step: Obtain digital images at the scale of crop leaves and canopy; The second step: Segment the region of interest in the digital images to separate the leaves and canopy areas to be monitored from the background; The third step: Convert the digital images into different color spaces, and extract the color, texture, and morphological features at the scale of crop leaves and canopy to construct feature vectors for subsequent fusion modeling; The fourth step: Obtain hyperspectral data at the scale of crop leaves and canopy; The fifth step: Use the hyperspectral data to extract the feature information at the scale of crop leaves and canopy to construct feature vectors for subsequent fusion modeling; The sixth step: Obtain chlorophyll fluorescence data at the scale of crop leaves and canopy; The seventh step: Use the chlorophyll fluorescence data to extract the feature information at the scale of crop leaves and canopy to construct feature vectors for subsequent fusion modeling; The eighth step: Substitute the feature vectors extracted from images, spectra, and fluorescence into the multi-source and multi-level data fusion method to accurately invert crop phenotype parameters; The multi-source and multi-level data fusion method is feature-level fusion, decision-level fusion, and hybrid fusion.

2. The multi-source and multi-level data fusion method applied to crop phenotypic parameter inversion according to claim 1, wherein In the first step, the specific operation of obtaining digital images at the scale of crop leaves and canopy is as follows: Use mobile phones and digital cameras. When shooting, the camera is set to automatic white balance. Under sunny weather, shoot samples at a fixed height from the crop leaves and canopy and at a 90° angle to the ground. It is necessary to assist with a standard color card for color correction to minimize the influence of light and the model of the shooting equipment on the images to the greatest extent.

3. The multi-source and multi-level data fusion method applied to the inversion of crop phenotypic parameters according to claim 1, characterized in that, The specific operation of the third step includes the following steps: Step one: Image color space conversion; Step two: Extract the color features of the image; Step three: Extract the texture features of the image; Step four: Extract the morphological features of the image; Step five: Screen the feature vectors based on the image for subsequent modeling.

4. The multi-source and multi-level data fusion method applied to crop phenotypic parameter inversion according to claim 1, wherein The specific operation of the fourth step is as follows: Use a portable ground object spectrometer to collect crop hyperspectral data. When collecting, collect through the leaf clip and built-in light source of the spectrometer. Take the average value of 3 repetitions at each leaf or canopy measurement point as the spectral value of this point. Perform whiteboard correction before measuring different leaves or canopies.

5. The multi-source and multi-level data fusion method applied to crop phenotypic parameter inversion according to claim 1, characterized in that In the seventh step, the method for screening characteristic chlorophyll fluorescence parameters is: Analyze the chlorophyll fluorescence parameter data of the leaves or canopy and the crop phenotype parameters to be measured. Use the Duncan method to analyze the significance of the differences, and then perform a correlation regression analysis on the chlorophyll fluorescence parameters and the parameters to be measured, and select the fluorescence parameters with good correlation with the crop phenotype parameters.

Images