Lettuce phenotype parameter estimation method based on deep learning fusion of multiple source images

Abstract

The present application relates to a kind of based on deep learning fusion multi-source image's lettuce phenotype parameter estimation method, including the following steps: obtaining RGB image and Depth image, manually measure the phenotype parameter of each lettuce sample, form image dataset;Using GrabCut algorithm carries out lettuce image foreground segmentation, using Z-Score method to the normalization of lettuce image data;Construction and training lettuce phenotype parameter estimation deep learning multi-source data fusion model.The beneficial effects of the present application are: the method of the present application adopts deep learning technology, fuses visible light image and depth image features, utilizes lettuce phenotype parameter estimation deep learning multi-source data fusion model to accurately estimate lettuce phenotype parameter by multi-source image information;As can be seen from the experimental results, the present application can successfully fuse two-dimensional RGB image and Depth image, excellent performance, and has important application value for high-throughput growth monitoring and yield estimation of protected vegetable.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital agriculture, and particularly relates to a lettuce phenotype parameter estimation method based on deep learning and multi-source image fusion. BACKGROUND

[0002] The external phenotype parameters of lettuce, such as dry weight, wet weight, plant height, diameter and leaf area, are directly related to the growth potential and yield of lettuce, and have important biological significance in agricultural production and research. The traditional quantitative measurement method of lettuce phenotype parameters consumes manpower and is low in efficiency; more seriously, in the dry and wet weight measurement, if the traditional destructive weighing method is used for soil culture or substrate culture lettuce, the lettuce cannot continue to grow, and the dry and wet weight of the specific plant during the growth process cannot be continuously collected. Non-destructive and high-precision estimation of the external phenotype parameters of lettuce can provide a technical basis for all-weather growth monitoring and plant growth process optimization, and help to realize the digitalization, automation and intelligentization of facility vegetable cultivation.

[0003] With the development of machine vision related software and hardware technologies, machine vision and image processing technology based on visible light has become a research hotspot for non-destructive estimation of phenotype parameters. However, most of the research on non-destructive estimation of lettuce phenotype parameters based on machine vision uses manually designed and manually extracted image features, which is difficult to cope with environmental interference and has low model generalization performance. The research based on deep learning method only relies on two-dimensional visible light (RGB) images, and does not consider the influence of three-dimensional morphology of plants on phenotype parameter estimation, so the estimation performance needs to be improved. Therefore, it is particularly important to develop a lettuce phenotype parameter estimation method combining deep learning and multi-source data fusion. SUMMARY

[0004] The purpose of the present application is to overcome the deficiencies in the prior art and provide a lettuce phenotype parameter estimation method based on deep learning and multi-source image fusion.

[0005] The lettuce phenotype parameter estimation method based on deep learning and multi-source image fusion comprises the following steps:

[0006] Step 1: Collect RGB images and Depth images, and manually measure the phenotype parameters of each lettuce sample to form an image dataset;

[0007] Step 2: Perform lettuce image foreground segmentation using the GrabCut algorithm, and normalize the lettuce image data using the Z-Score method;

[0008] Step 3: Construct and train a lettuce phenotype parameter estimation deep learning multi-source data fusion model; the lettuce phenotype parameter estimation deep learning multi-source data fusion model comprises a feature extraction part, a multi-source feature fusion part and a regression network part;

[0009] Step 4, input the RGB image and the depth image obtained in step 1 into the lettuce phenotype parameter estimation deep learning multi-source data fusion model obtained in step 3 to estimate the lettuce phenotype parameter vector y = (y1, y2, y3, y4, y5) of the sample to be estimated; wherein y1, y2, y3, y4 and y5 represent wet weight, dry weight, plant height, diameter and leaf area respectively.

[0010] As preferred, step 1 specifically comprises the following steps:

[0011] Step 1.1, a computer drives an RGBD camera located on an image acquisition platform to acquire RGBD overhead images of lettuce at each growth stage from planting to harvesting, and the RGBD camera outputs an RGB image (visible light image) and a depth image (depth map image);

[0012] Step 1.2, after the image acquisition of the lettuce sample is completed in step 1.1, the phenotype parameters of each lettuce sample are measured manually to form an image dataset.

[0013] As preferred in step 1.1, the RGB image contains leaf size, color and shape features, and the depth image records the distance information between the photographed object and the camera lens, and the depth image provides spatial depth information for plant phenotype parameter estimation.

[0014] As preferred in step 1.2, the phenotype parameters of the lettuce sample include wet weight, dry weight, plant height, maximum crown diameter and crown leaf area; wherein the plant height and the maximum crown diameter are measured by a vernier caliper, the plant height represents the straight-line distance from the highest point of the crown leaf to the root, and the maximum crown diameter represents the maximum straight-line distance of the crown leaf tip; the crown leaf area is measured by a leaf area meter; after removing the root of each lettuce sample, the wet weight of each lettuce sample is measured by an electronic balance; and then each lettuce sample is placed in a paper envelope and dried, and the dry weight of the lettuce after drying is measured by an electronic balance.

[0015] As preferred, step 2 specifically comprises the following steps:

[0016] Step 2.1, based on the lettuce RGBD overhead image, use the GrabCut algorithm to perform lettuce image foreground segmentation and extract the lettuce foreground image;

[0017] For each variety of lettuce sample, a lettuce sample is taken in advance as a standard image sample of the variety, the lettuce RGBD overhead image is divided, and the normalized color histogram p = (p0, p1,..., p 255 ) of each divided overhead image region is calculated, and the similarity between the normalized color histogram of each divided overhead image region and the normalized color histogram q = (q0, q1,..., q 255 ) of the standard image sample is calculated, p0, p1,..., p255 represents the color distribution characteristics of pixel values from 0 to 255 in the normalized color histogram of the overhead view region; q0, q1,..., q255represent the color distribution characteristics of pixel values from 0 to 255 in the normalized color histogram of the overhead view region; q0, q1,..., q255 255 represents the color distribution characteristics of pixel values from 0 to 255 in the normalized color histogram of the standard image sample; the KL divergence (Kullback-Leibler divergence) is used to measure the similarity between the histogram q and the histogram p, and the calculation formula is:

[0018] In the above formula, k represents the index value from 0 to 255; p k represents the number of points with pixel value k in the normalized color histogram of the overhead view region; q k represents the number of points with pixel value k in the normalized color histogram of the standard image sample; a threshold value a is set, if the KL divergence is lower than the threshold value a, it is determined that the color of the divided overhead view region is similar to the color of the lettuce; if the color of the region with a size of a set value is similar to the color of the lettuce, and the distances corresponding to the depth channels of the set number of pixels in the middle of the region are all less than a set value, the region is marked as foreground T F ; a preset rectangle in the middle of the picture in the lettuce image set is set as T r ; the rest of the lettuce image picture except T r is marked as background According to the marked T F and T B , the GrabCut algorithm is used for image segmentation;

[0019] Step 2.2, the Z-Score method is used to normalize the lettuce image data; in the lettuce image after foreground segmentation, the ith pixel point pixel c，i of the cth channel (c=0, 1, 2, i=0, 1, 2..., 369664) is converted into pixel' through Z-Score normalization: c，i

[0020]

[0021] In the above formula, μ c is the mean value of the pixel value of the cth channel image, σ c is the standard deviation of the pixel value of the cth channel image, and pixel c，i represents the ith pixel point of the cth channel.

[0022] As preferred, in step 2.1, the threshold value a=4.5; if the color of the region with a size of 15x15 is similar to the color of the lettuce, and the distances corresponding to the depth channels of the 5x5 pixels in the middle of the region are all less than 90cm, the region is marked as foreground T​F In step 2.2, in the foreground segmented lettuce image, the RGB image has 3 channels, and the Depth image has only 1 channel.

[0023] As preferred, step 3 specifically comprises the following steps:

[0024] Step 3.1, constructing a lettuce phenotype parameter estimation deep learning multi-source data fusion model, and setting an error function of the lettuce phenotype parameter estimation deep learning multi-source data fusion model; the output y of the lettuce phenotype parameter estimation deep learning multi-source data fusion model is a vector of length 5 (y1, y2, y3, y4, y5), wherein y1, y2, y3, y4 and y5 represent wet weight, dry weight, plant height, diameter and leaf area, respectively;

[0025] The feature extraction part is a ResNet50 network, and the number 50 means that the ResNet network (Residual Neural Network) contains a total of 50 layers; the ResNet50 network is used for learning and extracting lettuce effective features of RGB images and Depth images; the input of the ResNet50 network is the RGB image I rgb and the Depth image I Depth ; the output of the ResNet50 network is the feature vector f rgb and f depth :

[0026] f rgb = ResNet50(I rgb )

[0027] f depth = ResNet50(I depth )

[0028] The multi-source feature fusion part is used for splicing the RGB image feature vector f rgb and the Depth image feature vector f depth to obtain a lettuce phenotype parameter feature vector F:

[0029]

[0030] The regression network part is composed of multiple FC layers (full connection layers) and an activation function LeakyReLU, and the regression network part is used for phenotype parameter regression fitting of the fused lettuce phenotype parameter feature vector F; the activation function LeakyReLU is specifically:

[0031]

[0032] In the above formula, a is a coefficient set to a value; x is an input variable of the activation function LeakyReLU;

[0033] The lettuce phenotype parameter estimation belongs to a regression problem, and a normalized mean square error (NMSE) is used as an error function of the lettuce phenotype parameter estimation deep learning multi-source data fusion model.

[0034]

[0035] In the above formula, y represents a true value vector measured manually, and y' represents an estimation result vector; y i represents the i-th true value vector measured manually, y' i represents the i-th estimation result vector.

[0036] Step 3.2, training the lettuce phenotype parameter estimation deep learning multi-source data fusion model: setting the dropout rate of the FC layer of the regression network part, the training round, the batch size and the initial value of the learning rate; using the Adam optimizer; fine-tuning the ResNet50 network, and setting the learning rate of the corresponding ResNet50 network; after setting the number of times of fine-tuning the ResNet50 network, reducing the learning rate of the FC layer and the ResNet50 network.

[0037] As preferred, the value of the inner coefficient a of the activation function LeakyReLU in step 3.1 is 0.01; the FC layer comprises three layers of an input layer FC1, a hidden layer FC2 and an output layer FC3; and the calculation formula of the regression network part is:

[0038] F1 = LeakyReLU (FC1 (F))

[0039] F2 = LeakyReLU (FC2 (F1))

[0040] y = FC3 (F2)

[0041] The regression network part is optimized by adjusting the number of hidden layers and the number of neurons in the hidden layer.

[0042] The present application has the advantages that: the present application is a crop growth monitoring method, which adopts deep learning technology, fuses visible light image and depth image features, and accurately estimates lettuce phenotype parameters by using a lettuce phenotype parameter estimation deep learning multi-source data fusion model through multi-source image information; as can be seen from the experimental results, the present application can successfully fuse two-dimensional RGB images and Depth images, has excellent performance, and has important application value for high-throughput growth monitoring and yield estimation of facility vegetables. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of the estimation method of the present application;

[0044] Figure 2 is a schematic diagram of the lettuce phenotype parameter image dataset acquisition process of the present application;

[0045] Figure 3 A deep learning multi-source data fusion model diagram is estimated for the lettuce phenotype parameters of the application. DETAILED DESCRIPTION

[0046] The application will be further described below in conjunction with examples. The following examples are only used to help understand the application. It should be pointed out that for ordinary people in the technical field, some modifications can be made without departing from the principles of the application, and these improvements and modifications also fall within the scope of protection of the claims of the application.

[0047] Example 1

[0048] The application example 1 provides a lettuce phenotype parameter estimation method based on deep learning fusion of multi-source images in a greenhouse environment, which is divided into two stages: Figure 1

[0049] 1、Model construction and training stage

[0050] 1.1、Lettuce phenotype parameter-image data set collection

[0051] 1.1.1、As shown in Figure 2 , an RGBD camera is used for image data collection of lettuce phenotype parameters. The RGBD camera is a camera that can measure the color and depth information of a pixel point at the same time, and outputs a visible light image (hereinafter referred to as an RGB image) and a depth map image (hereinafter referred to as a Depth image). The RGB image contains some basic features visible to the naked eye, such as leaf size, color, shape, etc., which can provide relatively basic plant features. The depth image can record the distance information of the object being photographed from the camera lens, and can reflect the three-dimensional features of the plant, providing spatial depth information for plant phenotype parameter estimation. On the image collection platform, the RGBD camera is installed at a viewing angle of looking down at the lettuce, suspended about 0.90 meters above the lettuce. The collected image format is PNG format, the original image resolution is 1920x1080 pixels, the RGB image bit depth is 8, and the Depth image bit depth is 16. The computer drives the RGBD camera to collect images, and records the RGBD overhead images of the lettuce at each growth stage from planting to harvesting;

[0052] 1.1.2、After each lettuce sample completes image collection, its phenotype parameters are measured manually, including wet weight, dry weight, plant height, diameter, and leaf area. A vernier caliper is used to measure the plant height and the maximum diameter of the crown layer, wherein the plant height is the straight-line distance from the highest point of the crown layer to the root, and the maximum diameter of the crown layer is the maximum straight-line distance of the crown layer leaf tip; a leaf area meter is used to measure the crown layer leaf area; an electronic balance is used to measure the fresh weight of the sample after removing the root; then the sample is placed in a paper envelope and dried at 90℃ for 48h, and an electronic balance is used to measure the dry weight.​

[0053] 1.2, foreground segmentation and data normalization of lettuce image

[0054] 1.2.1, foreground image extraction of lettuce based on RGBD image

[0055] For each variety, a standard image sample is taken in advance. When segmenting the lettuce image, the image is divided into several 15x15 regions, and then the normalized color histogram p = (p0, p1,..., p 255 ) of each region is calculated, and its similarity q = (q0, q1,..., q 255 ) with the normalized color histogram of the standard image sample is calculated. The KL divergence (Kullback-Leibler divergence) is used to measure the similarity of the histograms, and the calculation formula is:

[0056]

[0057] As can be seen from the formula, the KL divergence does not allow the input to contain zero elements, so the normalized result is added by 0.001 before calculation. According to the experimental results, the threshold value is set to a = 4.5, and the region with KL divergence lower than the threshold value can be considered as similar in color to the lettuce color. If the color of a 15x15 region is similar to the lettuce color, and the distance corresponding to the depth channel of the central 5x5 pixels is less than 90cm, then the region is marked as foreground T F .

[0058] On the other hand, the lettuce image is concentrated in the middle of the picture, and will not exceed a preset rectangle T r , and the rest of the picture except T r is marked as background According to the marked T F and T B , the GrabCut algorithm is used for image segmentation;

[0059] 1.2.2, data normalization of lettuce image

[0060] The Z-Score method is used for data normalization of the image. In the segmented image, the ith pixel point pixel c，i (c = 0, 1, 2, i = 0, 1, 2,..., 369664) of the cth channel is converted to pixel′ c，i by Z-Score normalization. The formula is:

[0061]

[0062] In the formula, μ c is the mean value of the pixel value of the cth channel image, and σc is the standard deviation of the c-channel image pixel values, pixel c，i represents the i-th pixel point of the c-channel. The RGB image has 3 channels, and the Depth image has only 1 channel;

[0063] 1.3, Constructing and training a deep learning multi-source data fusion model for lettuce phenotype parameter estimation

[0064] 1.3.1, Construction of a deep learning multi-source data fusion model for lettuce phenotype parameter estimation

[0065] As shown in Figure 3 , the deep learning multi-source data fusion model for lettuce phenotype parameter estimation in this embodiment is composed of a feature extraction part, a multi-source feature fusion part and a regression network part.

[0066] The feature extraction part uses a residual neural network (Residual neural network, ResNet) to automatically learn and extract effective features of lettuce for RGB images and Depth images respectively. ResNet50 (the meaning of 50 is that the network contains a total of 50 layers) is used as the feature extraction network of the deep learning multi-source data fusion model for lettuce phenotype parameter estimation.

[0067] The result of ResNet50 feature extraction is a feature vector with a length of 2048. The RGB image I rgb and the Depth image I Depth are input into two ResNet50s for feature extraction, respectively, to obtain two feature vectors f rgb and f depth with a length of 2048:

[0068] f rgb = ResNet50(I rgb )

[0069] f depth = ResNet50(I depth )

[0070] The multi-source feature fusion part splices the RGB image feature vector f rgb and the Depth image feature vector f depth to obtain a lettuce phenotype parameter feature vector F with a length of 4096:

[0071]

[0072] The fused lettuce feature vector is input into the regression network part for phenotype parameter regression fitting. The regression network is composed of multiple layers of fully connected (FC) layers and activation functions. By adjusting the number of hidden layers and the number of neurons in the hidden layer, the regression model can be optimized. In the final lettuce phenotype parameter estimation deep learning multi-source data fusion model, the number of FC layers is three (including one input layer, one hidden layer and one output layer), the first layer FC1 contains 1024 neurons, the second layer FC2 contains 512 neurons, and the third layer FC3 contains 5 neurons. The activation function selected is LeakyReLU, which is calculated as follows:

[0073]

[0074] wherein the coefficient a is 0.01; x is the input variable of the activation function LeakyReLU.

[0075] The calculation formula of the regression network is as follows:

[0076] F1 = LeakyReLU(FC1(F))

[0077] F2 = LeakyReLU(FC2(F1))

[0078] y = FC3(F2)

[0079] The output y of the lettuce phenotype parameter estimation deep learning multi-source data fusion model is a vector of length 5, y1, y2, y3, y4, y5 representing wet weight, dry weight, plant height, diameter, and leaf area, respectively.

[0080] The lettuce phenotype parameter estimation of the present embodiment belongs to a regression problem, and the normalized mean square error (NMSE) is used as the error function, which is calculated as follows:

[0081]

[0082] wherein y' represents the true value vector measured manually, and y represents the estimation result vector, and the same below.

[0083] 1.3.2, lettuce phenotype parameter estimation deep learning multi-source data fusion model training

[0084] The FC layer in the model uses the same dropout rate of 0.3, is trained for 100 rounds, uses the Adam optimizer, and the batch size is 16. When initialized, the learning rate of the FC layer is set to 1x10 -3 The ResNet50 pre-trained model is fine-tuned, and the learning rate is set to 2x10 -4; After 50 rounds of training, the learning rate is reduced, and the learning rate of the FC layer is 2x10 -4 , and the learning rate of the ResNet50 pre-training model is 4x10 -5 ;

[0085] 2. Phenotype parameter estimation stage

[0086] 2.1. Take the sample to be estimated on the image acquisition platform, install the RGBD camera at a perspective of looking down at the lettuce, and hang it about 0.90 meters above the lettuce to ensure that the outer contour of the lettuce does not exceed or approach the image boundary. The computer drives the RGBD camera to acquire images, obtains the RGB image I rgb and the Depth image I Depth , and after image foreground extraction and data normalization using the method described in step 1.2 in stage 1, inputs into the lettuce phenotype parameter estimation deep learning multi-source data fusion model for calculation to obtain the output y = (y1, y2, y3, y4, y5), which is the phenotype parameter vector of the sample to be estimated, wherein y1, y2, y3, y4, and y5 represent the estimated wet weight, dry weight, plant height, diameter, and leaf area.

[0087] Example Two

[0088] Based on example one, the application example two provides a result verification method of the estimation method in example one:

[0089] In order to verify the effect of the method in example one, this example collects a data set in the intelligent plant factory laboratory of Zhejiang University City College, and the test material is 4 varieties of lettuce, including green, purple and red crinkled lettuce, covering all stages of lettuce growth from the seedling stage to maturity, a total of 388 lettuce samples. The data set is collected according to the method described in stage one. Using the cross-validation method, the data set is divided into four subsets S1, S2, S3, S4, each with 97 samples. Take S1 as the test set, S2, S3, and S4 as the training set to train the model I, and perform estimation experiments on S1 to obtain the estimation results of S1; take S2 as the test set, S1, S3, and S4 as the training set to train the model II, and perform estimation experiments on S2 to obtain the estimation results of S2; and so on to obtain the estimation results of the four subsets, which are combined as the final estimation results.

[0090] The prediction results are shown in Table 1 below. It can be seen that the phenotype parameter estimation method performs excellently in estimating the wet weight, dry weight, plant height, diameter, and leaf area of the lettuce, and the R 2 index is higher than 0.94, indicating that the estimation results of the method in example one have good correlation with the true values; on the other hand, the MAPE index is less than 8%, indicating that the proposed phenotype parameter estimation method has potential application value.

[0091] Table 1 Phenotype parameter estimation depth learning multi-source data fusion model estimation experimental results table

[0092]

[0093] The ablation experiment results are shown in Table 2 below. The ablation experiment found that, compared with the method of using only single-source data, the phenotype parameter estimation method of fusing multi-source data according to the present application has performance improvement in various phenotype parameters to different degrees, which shows that the three-dimensional morphological information of the plant provided by the Depth image helps the lettuce phenotype parameter estimation depth learning multi-source data fusion model to make more accurate estimation of the phenotype parameters.

[0094] Table 2 Comparison of results of different lettuce phenotype parameter estimation methods

[0095]

Claims

1. A method for estimating lettuce phenotypic parameters based on deep learning and fusion of multi-source images, characterized in that, Includes the following steps: Step 1: Collect RGB and Depth images, manually measure the phenotypic parameters of each lettuce sample, and form an image dataset; Step 2: Use the GrabCut algorithm to segment the foreground of the lettuce image and use the Z-Score method to normalize the lettuce image data; Step 3: Construct and train a deep learning multi-source data fusion model for estimating lettuce phenotypic parameters; the deep learning multi-source data fusion model for estimating lettuce phenotypic parameters includes a feature extraction part, a multi-source feature fusion part, and a regression network part; Step 3 specifically includes the following steps: Step 3.1: Construct a deep learning multi-source data fusion model for estimating lettuce phenotypic parameters, and set the error function of the deep learning multi-source data fusion model for estimating lettuce phenotypic parameters; the output y of the deep learning multi-source data fusion model for estimating lettuce phenotypic parameters is a vector (y1, y2, y3, y4, y5) of length 5, where y1, y2, y3, y4 and y5 represent wet weight, dry weight, plant height, diameter and leaf area, respectively; The feature extraction part uses a ResNet50 network, where 50 indicates that the ResNet network contains 50 layers. The ResNet50 network is used to extract effective features from RGB and depth images of lettuce. The input to the ResNet50 network is the RGB image I... rgb and Depth image I Depth The output of the ResNet50 network is the feature vector f. rgb and f depth : f rgb =ResNet50(I rgb ) f depth =ResNet50(I depth ) The multi-source feature fusion part is used to combine the RGB image feature vector f rgb and Depth image feature vector f depth By concatenating the vectors, we obtain the phenotypic feature vector F of lettuce: The regression network consists of multiple fully connected (FC) layers and the LeakyReLU activation function. This network is used to perform phenotypic parameter regression fitting on the fused lettuce phenotypic parameter feature vector F. The LeakyReLU activation function is specifically: In the above formula, 'a' is a coefficient with a set value; 'x' is the input variable of the activation function LeakyReLU. The normalized mean squared error (NMSE) is used as the error function of the deep learning multi-source data fusion model for estimating lettuce phenotypic parameters. In the above formula, y represents the vector of true values ​​measured manually. ′ Represents the estimation result vector; y i Let y′ represent the true value vector of the i-th manual measurement. i This represents the vector of the i-th estimation result; Step 3.2: Train the deep learning multi-source data fusion model for estimating lettuce phenotypic parameters: Set the dropout rate, training epochs, batch size, and initial learning rate of the FC layer in the regression network. Use the Adam optimizer; fine-tune the ResNet50 network and set the corresponding learning rate for the ResNet50 network; after fine-tuning the ResNet50 network a set number of times, reduce the learning rate of the FC layer and the ResNet50 network; Step 4: Input the RGB image and Depth image obtained in Step 1 into the deep learning multi-source data fusion model for estimating lettuce phenotypic parameters obtained in Step 3 to estimate the vector of various lettuce phenotypic parameters of the sample to be estimated, y = (y1, y2, y3, y4, y5); where y1, y2, y3, y4 and y5 represent wet weight, dry weight, plant height, diameter and leaf area, respectively.

2. The method for estimating lettuce phenotypic parameters based on deep learning fusion of multi-source images according to claim 1, characterized in that, Step 1 specifically includes the following steps: Step 1.1: The computer drives the RGBD camera located on the image acquisition platform to acquire RGBD top-view images of lettuce at each growth stage from planting to harvest. The RGBD camera outputs RGB images and depth images. Step 1.2: After image acquisition is completed in step 1.1, the phenotypic parameters of each lettuce sample are manually measured to form an image dataset.

3. The method for estimating lettuce phenotypic parameters based on deep learning fusion of multi-source images according to claim 2, characterized in that, In step 1.1: the RGB image contains leaf size, color and shape features, the depth image records the distance information between the photographed object and the camera lens, and the depth image provides spatial depth information for estimating plant phenotypic parameters.

4. The method for estimating lettuce phenotypic parameters based on deep learning fusion of multi-source images according to claim 2, characterized in that, In step 1.2: the phenotypic parameters of the lettuce samples include wet weight, dry weight, plant height, maximum canopy diameter, and canopy leaf area; among which, plant height and maximum canopy diameter are measured by vernier calipers, plant height represents the straight-line distance from the highest point of the canopy leaf to the root, and maximum canopy diameter represents the maximum straight-line distance from the tip of the canopy leaf; the canopy leaf area is measured by a leaf area meter; after removing the root of each lettuce sample, the wet weight of each lettuce sample is measured by an electronic balance; then each lettuce sample is placed in a paper envelope and dried, and the dry weight of the dried lettuce is measured by an electronic balance.

5. The method for estimating lettuce phenotypic parameters based on deep learning fusion of multi-source images according to claim 2, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Based on the RGBD top view image of lettuce, use the GrabCut algorithm to segment the foreground of the lettuce image and extract the foreground image of lettuce; For each variety of lettuce sample, a standard image sample of lettuce is pre-selected, and the lettuce is divided into RGBD top view regions. Then, a normalized color histogram p = (p0, p1, ..., p) is calculated for each divided top view region. 255 ), calculate the normalized color histogram of each divided top view region and the normalized color histogram of the standard image sample, q = (q0, q1, ..., q 255 The similarity between p0, p1, ..., p 255 To find the color distribution characteristics of pixel values ​​from 0 to 255 in the normalized color histogram of the top-view region; q0, q1, ..., q 255 The normalized color histogram represents the color distribution characteristics of pixel values ​​from 0 to 255 in the image; the KL divergence is used to measure the similarity between histogram q and histogram p, and the calculation formula is: In the above formula, k represents the subscript value ranging from 0 to 255; p k This represents the number of pixels with pixel value k in the normalized color histogram of the top-view area; q k This represents the number of pixels with pixel value k in the normalized color histogram of the standard image sample. A threshold α is set; if the KL divergence is below the threshold α, the color of the divided top-view region is determined to be similar to the color of lettuce. If the color of a region of a set size is similar to the color of lettuce, and the distances corresponding to the depth channels of a set number of pixels in the center of that region are all less than a set value, then that region is marked as foreground T. F Let T be the preset rectangle in which the lettuce images are concentrated in the center of the image. r Remove T from the lettuce image. r The rest of the area is marked as background. According to the marked T F and T B Image segmentation is performed using the GrabCut algorithm; Step 2.2: Normalize the lettuce image data using the Z-Score method; in the foreground segmented lettuce image, the i-th pixel of the c-th channel is... c,i ， c = 0, 1, 2, i = 0, 1, 2, ..., 369664, are converted to pixels using Z-score normalization. c,i : In the above formula, μ c σ is the mean of the pixel values ​​in the c-channel image. c The standard deviation of the pixel values ​​in the c-channel image, in pixels. c,i This represents the i-th pixel in channel c.

6. The method for estimating lettuce phenotypic parameters based on deep learning fusion of multi-source images according to claim 5, characterized in that: In step 2.1, the threshold α = 4.5; if the color of a 15×15 region is similar to that of lettuce, and the distance between the depth channels of the 5×5 pixels in the center of this region is less than 90cm, then this region is marked as the foreground T. F ; In step 2.2, in the foreground segmented lettuce image, the RGB image has 3 channels, while the Depth image has only 1 channel.

7. The method for estimating lettuce phenotypic parameters based on deep learning fusion of multi-source images according to claim 5, characterized in that: In step 3.1, the inner coefficient 'a' of the activation function LeakyReLU is set to 0.01; the FC layer consists of three layers: input layer FC1, hidden layer FC2, and output layer FC3; the calculation formula for the regression network part is: F1 = LeakyReLU(FC1(F)) F2 = LeakyReLU(FC2(F1)) y = FC3(F2) The regression network part is optimized by adjusting the number of hidden layers and the number of neurons in the hidden layers.

Images