Plant early stress monitoring method based on nanometer sensing infrared thermal image recognition

Abstract

The invention discloses a plant early stress monitoring method based on nano sensing infrared thermal image recognition, which comprises the following steps: capturing a biological signal of a plant by in-situ synthesizing a nano biosensor on the plant, converting absorbed near-infrared laser into heat by the biosensor after the plant is stressed, and making a data set by utilizing a thermal imaging picture of the plant; expanding the data set, dividing the data set into a training set, a test set and a verification set, preprocessing pictures of the training set and the verification set, inputting the pictures into a deep learning model for training, and according to a training result, evaluating the model and optimizing parameters; and finally, inputting the test set into the deep learning model after parameter optimization to obtain a final classification result. According to the method, the nano biosensor, the thermal imaging technology and the deep learning algorithm are fused, the stress of the plant in the early stage can be monitored and warned, the accuracy of monitoring the early stress of the plant is effectively improved, and plant damage and significant output reduction are avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of plant growth monitoring, and particularly relates to a method for monitoring early plant stress based on nano-sensing infrared thermal image recognition. Background Art

[0002] When a plant is stressed, chemical changes such as changes in hormone levels or surges in signaling molecules begin to occur in its body before any physical changes take place. However, with the continuous action of stress factors, by the time the plant has undergone visible external changes, it is often too late to intervene in its growth condition.

[0003] There are many existing methods for detecting plant stress, but the effects are not satisfactory. Traditional biochemical means can accurately detect plant stress by analyzing plant signaling molecules and hormones, but the detection process is often destructive and cannot achieve non-destructive detection; deep learning technology can accurately identify stressed plants without damage through model training, but training the model with RGB images has too much noise, and training the model with hyperspectral data has data redundancy, both of which will affect the accuracy of classification results; while obtaining more detailed information on plant health through thermal imaging technology can improve the recognition accuracy, but data such as stem water potential and stomatal conductance used in the research are difficult to obtain.

[0004] Combining the above viewpoints, the present invention proposes a non-destructive, accurate, and simple method for early detection of plant stress, which can monitor the life activities and physiological states of plants in real time and accurately, take timely measures to reduce losses, and improve agricultural production efficiency. Summary of the Invention

[0005] In view of the problems existing in the above background art, the present invention provides a method for monitoring early plant stress based on nano-sensing infrared thermal image recognition, which is convenient for making a diagnosis at the early stage of plant stress occurrence, thereby reducing the losses caused by plant stress.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for monitoring early plant stress based on nano-sensing infrared thermal image recognition, comprising the following steps:

[0008] S1. In-situ synthesize a nano-biosensor on the plant to be measured, ensuring good contact between the nano-biosensor and the plant, and being able to accurately capture the biological signals of the plant;

[0009] S2. Stress the plant to be measured;

[0010] S3. Use an infrared thermal imager to take a thermal image of the part of the plant to be measured where the nano-biosensor is attached;

[0011] S4. Screen the captured thermal imaging pictures, select the pictures that meet the conditions for classification, improve the picture labels, and make them into a dataset;

[0012] S5. Expand the made dataset by means of data augmentation to prevent overfitting of the model due to small data volume and affect the output results; divide the expanded dataset into a training set, a test set and a validation set, and preprocess the pictures in the training set and the validation set to make them meet the requirements of the input of the deep learning model;

[0013] S6. Input the preprocessed pictures into the deep learning model, start training the deep learning model, and evaluate and optimize the parameters of the deep learning model according to the training results of the deep learning model;

[0014] S7. Input the test set into the deep learning model with optimized parameters to obtain the final classification result.

[0015] Furthermore, in step S1, the method for attaching the nano-biosensor to the plant to be measured is as follows:

[0016] S11. Clean the surface of the plant to be measured to ensure that the nano-biosensor can be attached to it;

[0017] S12. Mix ABTS, HRP and 2-mim and incubate for 10 minutes. The final concentrations of ABTS, HRP and 2-mim are 2 mg / mL -1 , 15 μg / mL -1 and 70 mM respectively;

[0018] S13. Spray the incubated mixture on the leaves of the plant to be measured;

[0019] S14. After waiting for a certain time, continue to spray 40 mM ZnAc 2 solution on the leaves of the plant to be measured sprayed with the mixture and let it stand for 5 minutes to generate a nano-biosensor attached to the plant to be measured.

[0020] Furthermore, in step S2, the types of the stress include one or more of drought, salinity, high temperature, low temperature, and mechanical damage.

[0021] Furthermore, the conditions of the stress are as follows:

[0022] The condition of drought stress is: maintain the relative water content (RWC) of the soil of the plant to be measured at 45% for 2 days;

[0023] The condition of salinity stress is: irrigate the plant to be measured with a sodium chloride solution with a concentration of 20 mM for 2 days;

[0024] The conditions for high-temperature stress are as follows: Place the plant to be tested in an environment of 40°C to 45°C for 10 minutes;

[0025] The conditions for low-temperature stress are as follows: Place the plant to be tested in an environment of 2°C to 8°C for 10 minutes;

[0026] The conditions for mechanical stress are as follows: Punch holes on the leaf surface of the plant to be tested.

[0027] Furthermore, in step S5, the method for expanding the data set is as follows:

[0028] S51. Random horizontal flipping;

[0029] S52. Random vertical flipping;

[0030] S53. Random rotation, with the angle range between (-180°, 180°);

[0031] S54. Random translation, with the translation parameters randomly offsetting (-10, 10) pixels respectively in the width and height of the image;

[0032] S55. Random cropping, while keeping the output image size unchanged;

[0033] S56. Random deformation, with the deformation parameter being that the length of the image becomes 60% of the original with a probability of 70% while ensuring the image size remains unchanged.

[0034] Furthermore, in step S5, the division ratio of the training set, test set, and validation set is 8:1:1.

[0035] Furthermore, in step S5, the preprocessing method for the images in the training set is as follows:

[0036] S511. Randomly crop the image to a size of 224*224 pixels to enhance the generalization ability of the model;

[0037] S512. Randomly horizontally flip the image to further increase data diversity;

[0038] S513. Convert the image into Tensor representation;

[0039] S514. Perform standardization processing on the Tensor using predefined mean and standard deviation;

[0040] The preprocessing method for the images in the validation set is as follows:

[0041] S521. Resize the image to 256*256, and then crop 224*224 from it;

[0042] S522. Crop the center of the image to a size of 224*224 pixels;

[0043] S523. Convert the image into a Tensor representation;

[0044] S524. Normalize the Tensor using predefined mean and standard deviation.

[0045] Furthermore, in step S6, the deep learning model is VGGNet or ResNet.

[0046] Furthermore, in step S6, during the training of the deep learning model, transfer learning is adopted. The convolutional layers of the model are frozen, and only the part after the convolutional layers is trained. The pre-trained weights used are the official pre-trained weights of the ImageNet dataset corresponding to the model.

[0047] Compared with the disadvantages and deficiencies of the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention uses a nano-biosensor to amplify the response of plants after being stressed, can respond more sensitively to stress, and conduct early diagnosis on plants;

[0049] 2. The present invention uses a thermal imaging device to capture the response of plants stressed by the nano-biosensor. The captured pictures are less interfered by irrelevant details, with more prominent features, and achieve non-destructive sampling;

[0050] 3. The present invention uses a deep learning model to identify the pictures taken by the thermal imaging device, improves the classification accuracy and working efficiency, and provides an intelligent and efficient solution for plant stress monitoring and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of a method for early plant stress monitoring based on nano-sensing infrared thermal image recognition provided by an embodiment of the present invention;

[0052] Figure 2 is a sample diagram of thermal imaging pictures (a) of plants in a healthy state and (b) of plants in a stressed state taken by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] A method for early plant stress monitoring based on nano-sensing infrared thermal image recognition, the flowchart refers to Figure 1 , and includes the following steps:

[0055] S1. In-situ synthesize a nano-biosensor on the plant to be tested, the method is as follows:

[0056] S11. Clean the surface of the plant to be tested to ensure that the nano-biosensor can adhere to it and accurately capture the biological signals of the plant;

[0057] S12. Mix ABTS, HRP and 2-mim, and incubate for 10 minutes. The final concentrations of ABTS, HRP and 2-mim are 2 mg / mL -1 , 15 μg / mL -1 and 70 mM respectively;

[0058] S13. Load the incubated mixture into a sprayer and spray it on the leaves of the plant to be tested for 3 seconds;

[0059] S14. After waiting for 3 minutes, continue to spray 40 mM ZnAc 2 solution on the leaves of the plant to be tested sprayed with the mixture and let it stand for 5 minutes to generate a nano-biosensor attached to the plant to be tested.

[0060] S2. Apply stress to the plant to be tested; the types of stress include one or more of drought, salinity, high temperature, low temperature, and mechanical damage; the conditions of stress are as follows:

[0061] The condition of drought stress is: maintain the relative water content (RWC) of the soil of the plant to be tested at 45% for 2 days; the calculation formula of the soil relative water content is:

[0062]

[0063] where FW is the fresh weight of the soil, DW is the dry weight of the soil, and IW is the initial weight of the soil at the start of drought treatment.

[0064] The condition of salinity stress is: irrigate the plant to be tested with a sodium chloride solution with a concentration of 20 mM for 2 days;

[0065] The condition of high temperature stress is: place the plant to be tested in an environment of 40°C to 45°C for 10 minutes;

[0066] The condition of low temperature stress is: place the plant to be tested in an environment of 2°C to 8°C for 10 minutes;

[0067] The condition of mechanical stress is: punch holes on the leaf surface of the plant to be tested.

[0068] S3. Use an infrared thermal imager to take a thermal imaging picture of the part of the plant to be tested attached with the nano-biosensor. The shooting operation is as follows:

[0069] S31. Adjust the imaging mode of the infrared thermal imager to "Rainbow", and hide the center point, the highest temperature point, and the lowest temperature point to ensure that the captured images have no other interferences.

[0070] S32. The parts of the plants to be photographed are leaves or stems. Take one photo every 1 minute for healthy plants and plants after stress at different angles, and the shooting time is 10 minutes.

[0071] S33. The shooting is carried out under an 808nm laser emitter, and the only variable controlled in the experiment is whether there is stress.

[0072] S4. Screen the captured thermal imaging pictures, select the qualified pictures for classification. The criteria for the selected thermal imaging pictures are: the shooting parts are neat and complete, the background is single, the difference between healthy plants and plants after stress is obvious. Refer to Figure 2 , uniformly name each category of the captured pictures, and classify them in different folders, improve the picture tags, and make them into a dataset.

[0073] S5. Expand the made dataset by means of data augmentation to prevent overfitting of the model due to small data volume and affect the output results. The methods for expanding the dataset are as follows:

[0074] S51. Random horizontal flipping;

[0075] S52. Random vertical flipping;

[0076] S53. Random rotation, and the angle range is between (-180°, 180°);

[0077] S54. Random translation, and the translation parameters are to randomly offset (-10, 10) pixels respectively in the width and height of the image;

[0078] S55. Random cropping, and keep the size of the output image unchanged;

[0079] S56. Random deformation, and under the condition of ensuring the unchanged size of the picture, the deformation parameter is to make the length of the picture become 60% of the original with a probability of 70%.

[0080] Divide the expanded dataset into a training set, a test set and a validation set, and preprocess the pictures in the training set and the validation set to make them meet the requirements of the input of the deep learning model. The preprocessing method for the pictures in the training set is:

[0081] S511. Randomly crop the image to a size of 224 * 224 pixels to increase the generalization ability of the model;

[0082] S512. Randomly horizontally flip the image to further increase data diversity;

[0083] S513. Convert the image into a Tensor representation;

[0084] S514. Normalize the Tensor using predefined mean and standard deviation;

[0085] The preprocessing method for the images in the validation set is as follows:

[0086] S521. Resize the image to 256*256, and then crop out 224*224 from it;

[0087] S522. Crop the center of the picture to a size of 224*224 pixels;

[0088] S523. Convert the image into a Tensor representation;

[0089] S524. Normalize the Tensor using predefined mean and standard deviation.

[0090] The augmented dataset is divided into a training set, a test set, and a validation set in the ratio of 8:1:1. The training set is used for training the model, adjusting the model's parameters to minimize the prediction error on the training set; the validation set is used to evaluate the model's performance during training to help select the best model parameters or structure; the test set is used to finally evaluate the model's generalization ability after the model selection and training are completed.

[0091] Since the official weights of the ImageNet dataset are used for training the model, the predefined mean and standard deviation in S614 and S624 are the mean and standard deviation of the ImageNet dataset, which are [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225] respectively. The formula for the mean is:

[0092]

[0093] where n is the total number of samples, x is the i-th sample, and i is the sample index;

[0094] The formula for the standard deviation is:

[0095]

[0096] S6. Input the preprocessed pictures into a deep learning model, where the deep learning model is VGGNet or ResNet, and start training the deep learning model. The hyperparameters set for model training are: learning rate, batch size, and number of iterations; during model training, the transfer learning method is adopted, freezing the convolutional layers of the model, and only training the part after the convolutional layers. The pre-trained weights used are the official pre-trained weights of the ImageNet dataset corresponding to the model.

[0097] Among them:

[0098] The model structure of VGGNet is successively: input layer, convolutional layer, pooling layer, convolutional layer, pooling layer, convolutional layer, pooling layer, convolutional layer, pooling layer, convolutional layer, pooling layer, fully connected layer, fully connected layer, fully connected layer, Softmax; this model structure is shown in Table 1:

[0099] Table 1 Model Structure of VGGNet

[0100]

[0101]

[0102] The dimension of the input layer of the VGGNet model is 224*224*3; the convolutional kernel size of the convolutional layer is 3*3, the stride is 1, the padding is 1, the activation function is the ReLU function, and the Dropout is set to 0.5; the formula for convolution is:

[0103]

[0104] Among them, x is a two-dimensional vector with the receptive field size of A*B, A and B are the length and width of the two-dimensional vector of the image respectively, w is a convolutional kernel with length j and width i, c is the bias added to the output feature map, defaulting to 1, y conv is the result output after the convolution operation, f is the activation function in the neural network, and a and b are the indices of the current position in the output feature map;

[0105] The formula for pooling is:

[0106] f pool = Max(x m,n , x m+1,n , x m,n+1 , x m+1,n+1 )

[0107] Among them, f pool represents the result after max pooling;

[0108] The formula for the ReLU activation function is:

[0109] f(x) = max(0, x)

[0110] The loss function in Softmax is:

[0111]

[0112] Among them, m is the total number of samples, θ = (θ1, θ2,..., θk) are the training parameters of the model, j (j ∈ {1, 2,..., k}) is the sample label, y(i) (y (i) ∈ {1, 2, …, k}) is the predicted sample label, and γ represents the Dirichlet function: if y (i) = j is true, then γ takes the value of 1; otherwise, it takes the value of 0.

[0113] The model structure of ResNet is shown in Table 2:

[0114] Table 2 Model Structure of ResNet

[0115]

[0116] The first part (Conv1): First, perform a convolution operation on the input feature matrix with a size of 224*224. The size of the convolution kernel is 7, the number is 64, and the stride is 2, outputting a feature matrix with a size of 112x112. Then, perform a max pooling operation with a convolution kernel size of 3 and a stride of 2, outputting a feature matrix with a size of 56*56;

[0117] The second part (Conv2_x): Process the input feature matrix with a size of 56*56 through a complex residual block structure, and the output size remains unchanged. Among them, the ResNet18 and ResNet34 models use 2 and 3 basic residual block structures respectively, and the ResNet50 and ResNet101 models use 3 bottleneck residual block structures;

[0118] The third part (Conv3_x): Process the input feature matrix with a size of 56*56 through a complex residual block structure, and output a feature matrix with a size of 28*28. Among them, the ResNet18 and ResNet34 models use 2 and 4 basic residual block structures respectively, and the ResNet50 and ResNet101 models use 4 bottleneck residual block structures;

[0119] The fourth part (Conv4_x): Process the input feature matrix with a size of 28*28 through a complex residual block structure, and output a feature matrix with a size of 14*14. Among them, the ResNet18 and ResNet34 models use 2 and 6 basic residual block structures respectively, and the ResNet50 and ResNet101 models use 6 and 23 bottleneck residual block structures respectively;

[0120] Part 5 (Conv5x): The feature matrix with a size of 14*14 is processed through a complex residual block structure, and a feature matrix with a size of 7*7 is output. Among them, the ResNet18 and ResNet34 models use 2 and 3 basic residual block structures respectively, and the ResNet50 and ResNet101 models use 3 bottleneck residual block structures;

[0121] Part 6: This part is the output of the ResNet model. First, an average pooling operation is performed on the input feature matrix with a size of 7x7 to achieve dimensionality reduction. Then, through a fully connected layer, a feature vector with a size of 1000*1*1 is obtained. Finally, classification tasks can be performed through a Softmax classifier.

[0122] According to the training results of the deep learning model, evaluate the deep learning model and optimize the parameters; the metrics for evaluating the model are respectively:

[0123] Accuracy: The proportion of the number of samples correctly classified by the model to the total number of samples;

[0124] Precision: Among the samples predicted as positive by the model, the proportion that is truly positive;

[0125] Recall: Among all real positive samples, the proportion correctly identified by the model;

[0126] F1-Score: The harmonic mean of precision and recall.

[0127] The formulas for the evaluation metrics of the deep learning model are respectively:

[0128]

[0129]

[0130]

[0131]

[0132] Among them, TP represents the positive sample with a positive prediction result, TN represents the negative sample with a negative prediction result, FP represents the negative sample with a positive prediction result, and FN represents the positive sample with a negative prediction result.

[0133] S7. Input the test set into the deep learning model with optimized parameters, and the final classification results are shown in Table 3:

[0134] Table 3 Classification results of different VGGNet and ResNet models with the same parameters

[0135]

[0136] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for monitoring early plant stress based on nanosensing infrared thermal image recognition, characterized in that: The following steps are involved: S1. In situ synthesis of nanobiosensors on the plants to be tested, ensuring that the nanobiosensors are in good contact with the plants and can accurately capture the biological signals of the plants; S2, stressing the plants to be tested; S3, using an infrared thermal imager to take thermal imaging pictures of the parts of the plant to be tested where the nanobiosensor is attached; S4, screening the captured thermal imaging images, selecting the images that meet the conditions for classification, improving the image labels, and making a data set; S5. Expand the prepared data set by means of data enhancement to prevent the model from overfitting and affecting the output results due to the small amount of data; divide the expanded data set into a training set, a test set, and a validation set, and preprocess the images in the training set and the validation set to make them meet the requirements of the deep learning model input; S6. Input the preprocessed image into the deep learning model, start training the deep learning model, and evaluate the deep learning model and optimize parameters according to the training results of the deep learning model; S7. Input the test set into the deep learning model with optimized parameters to obtain the final classification result.

2. The method for monitoring early plant stress based on nanosensor infrared thermal image recognition according to claim 1, characterized in that: In step S1, the method of attaching the nanobiosensor to the plant to be tested is as follows: S11, cleaning the surface of the plant to be tested to ensure that the nanobiosensor can be attached to it; S12. Mix ABTS, HRP and 2-mim and incubate for 10 min. The final concentrations of ABTS, HRP and 2-mim are 2 mg / mL -1 , 15 μg mL -1 and 70 mM; S13, spraying the incubated mixture on leaves of the plants to be tested; S14. After waiting for a certain period of time, continue to spray 40 mM ZnAc2 solution on the leaves of the plant to be tested sprayed with the mixture and let it stand for 5 minutes to generate nanobiosensors attached to the plant to be tested.

3. The method for monitoring early plant stress based on nanosensor infrared thermal image recognition according to claim 1, characterized in that: In step S2, the type of stress includes one or more of drought, salinity, high temperature, low temperature, and mechanical damage.

4. The method for monitoring early plant stress based on nanosensor infrared thermal image recognition according to claim 3, characterized in that: The conditions of the coercion are as follows: The conditions of drought stress were as follows: the relative water content (RWC) of the soil of the tested plants was maintained at 45% for 2 days; The conditions of saline-alkali stress were as follows: watering the plants to be tested with a 20 mM sodium chloride solution for 2 days; The conditions of high temperature stress were as follows: the plants to be tested were placed in an environment of 40°C to 45°C for 10 minutes; The conditions of low temperature stress are as follows: the plants to be tested are placed in an environment of 2°C to 8°C for 10 minutes; The condition of mechanical stress is: holes are punched on the leaf surface of the plant to be tested.

5. The method for monitoring early plant stress based on nanosensor infrared thermal image recognition according to claim 1, characterized in that: In step S5, the method for expanding the data set is as follows: S51, random horizontal flip; S52, random vertical flip; S53, random rotation, the angle range is between (-180°, 180°); S54, random translation, the translation parameter is to randomly shift (-10, 10) pixels in the width and height of the image respectively; S55, randomly cropping and keeping the output image size unchanged; S56, random deformation, under the condition of ensuring that the image size remains unchanged, the deformation parameter is to make the image length become 60% of the original length with a probability of 70%.

6. The method for monitoring early plant stress based on nanosensor infrared thermal image recognition according to claim 1, characterized in that: The division ratio of the training set, test set and validation set is 8:1:

1.

7. The method for monitoring early plant stress based on nanosensor infrared thermal image recognition according to claim 1, characterized in that: In step S5, the preprocessing method of the images in the training set is: S511, randomly crop the image to 224*224 pixels to increase the generalization ability of the model; S512, randomly flipping the image horizontally to further increase data diversity; S513, converting the image into a Tensor representation; S514, standardize the Tensor using the predefined mean and standard deviation; The preprocessing method for the images in the verification set is: S521, resize the image to 256*256, and then crop 224*224 from it; S522, crop the center of the image to a size of 224*224 pixels; S523, converting the image into a Tensor representation; S524. Standardize the Tensor using a predefined mean and standard deviation.

8. The method for monitoring early plant stress based on nanosensor infrared thermal image recognition according to claim 1, characterized in that: In step S6, the deep learning model is VGGNet or ResNet.

9. The method for monitoring early plant stress based on nanosensor infrared thermal image recognition according to claim 8, characterized in that: In the deep learning model training, a transfer learning method is adopted to freeze the convolutional layer of the model, and only train the part after the convolutional layer. The pre-training weights used are the official pre-training weights of the ImageNet dataset of the corresponding model.

Images