Plant micro-change image data sequence prediction method based on deep learning

Abstract

The invention belongs to the technical field of plant image prediction, and particularly relates to a plant micro-change image data sequence prediction method based on deep learning, and the method comprises the following steps: S1, data preparation, S2, data analysis, S3, building a PMD-PM model, S4, training the PMD-PM model, and storing the PMD-PM model, S5, carrying out the hyper-parameter adjustment of a selected network, and S6, carrying out the parameter fine adjustment of the PMD-PM model. The invention provides a general framework for predicting a plant micro-change image sequence, not only can process data of plants in a growth process, but also can process micro-change prediction of fruits and the like in an aging process, provides a solution for real-time detection of an agricultural whole industrial chain, and has certain practicability.

Description

Technical Field

[0001] The present invention belongs to the technical field of plant image prediction, and in particular relates to a method for predicting plant micro-change image data sequence based on deep learning. Background Art

[0002] Plants grow slowly naturally. Predicting their growth and development and analyzing the corresponding plant phenotypes in advance can help agricultural experts obtain plant growth and development, storage and aging conditions earlier, and then conduct statistical analysis of phenotypic variation, correlation between different traits, and localization of genetic loci for regulating target phenotype compilation, thereby enhancing the timeliness of detection in the entire agricultural industry chain. The latest advances in artificial intelligence (AI) technology have opened the door to nonlinear modeling methods in the field of plants. In order to improve the speed, reliability and scale of plant phenotyping procedures, many researchers have adopted deep learning methods to estimate phenotypic information in plant and crop images. Therefore, this has become an extremely important key technology and scientific means in the field of modern "AI+Agriculture", and also provides technical support for further guiding and implementing the national agricultural revitalization strategy.

[0003] In the past few years, deep learning methods, especially convolutional neural networks, have been mostly used in plant phenotyping, feature extraction, and classification research based on individual static images of different species of plants. However, simply using temporal dependencies can only predict changes in a single phenotype, and it is difficult to complete a more friendly visualization behavior of plant growth and development. In contrast, video frame prediction technology can effectively predict the dynamic changes of future frames by capturing long-term dynamic dependencies and spatial correlations in the frame sequence in the video. However, there are still many challenges in directly applying this technology to plant growth prediction research. First, the time interval between adjacent plant images is significantly longer than that of traditional video prediction tasks. The video frame interval is usually within tens of milliseconds, while the interval between plant images ranges from several hours to even a day. Therefore, the movement of leaves is more dynamic. Second, the shape, size, and direction of plant organs will appear, disappear, stack, rotate, and other random changes over time. These types of motion are different from objects often observed in natural videos. Finally, in the video, the speed can be regarded as a steady process, while the growth rate of plants at a fixed sampling rate is still inconsistent. This must be affected by higher-order local feature relationships and long-term dependencies, and it is difficult to find motion vectors in plant images. Summary of the invention

[0004] The purpose of the present invention is to provide a plant micro-change image data sequence prediction method based on deep learning to solve the problems existing in the background technology.

[0005] like Figure 1 As shown, a method for predicting plant micro-change image data sequence based on deep learning includes the following steps:

[0006] S1: Data preparation: First, preprocess the raw data, identify abnormal data and remove or correct it from the raw data; second, perform background removal and correction on the image sequence; finally, divide the training set, validation set and test set;

[0007] S2: Data analysis: ESTDA (Exploratory Spatial Temporal Data Analysis) is used to examine the temporal and spatial trends of the training data and determine whether there are large-scale trends and non-stationary features in the image sequence of plant micro-change processes. If not, the training data is a stationary process. Otherwise, a deep learning network is needed to remove these features before prediction.

[0008] S3: Building PMD-PM (Plant Microvariation Dynamic Prediction Model, PMD-PM) by stacking deep temporal C-LSTM on the original input layer L unit, followed by a gradient connection unit; D-LSTM is stacked at a high level L Unit, uses spatiotemporal long short-term memory network to learn spatiotemporal features, and expands the differential memory mechanism in the spatiotemporal recurrent network to eliminate the non-stationary characteristics of the sequence;

[0009] S4: training and saving the PMD-PM model: using the training set of the plant micro-change image sequence data set in step S1 to train the PMD-PM model, saving the best model parameters and weight coefficients during the training process, and calculating the prediction accuracy of the prediction model on the validation set;

[0010] S5: Adjust the hyperparameters of the selected network: select error evaluation indicators, image quality evaluation indicators, and phenotypic feature evaluation indicators for performance evaluation;

[0011] S6: Fine-tuning of PMD-PM model parameters: Fine-tuning is performed based on the model prediction quality in step S4 as the basis for adjusting the model parameters.

[0012] Preferably, in S2, by visualizing the trend of the spatial distribution of data on a large scale, if the spatiotemporal series has a stationary property, it needs to satisfy:

[0013]

[0014] where z i (t) is the spatiotemporal data sequence, is the sample mean, is the variance, k is for time delay, and h is for space delay;

[0015] The ADF (Augmented Dickey-Fuller test) sequence stationarity verification method is used to verify the high-order trend of the data; when performing the ADF test, the value of the ADF statistic is compared with the critical value to determine whether to reject the null hypothesis; if the value of the ADF statistic is less than the critical value, the null hypothesis can be rejected and the sequence is considered to be stationary; otherwise, the null hypothesis cannot be rejected and the sequence is considered to be non-stationary; the formula is as follows:

[0016]

[0017] where Δy t is a difference time series, t represents the time variable, α, γ, ρ are the parameters of the model, ε t represents the white noise error term.

[0018] Preferably, in S3, the following steps are specifically included:

[0019] 3.1PMD-PM model stacks deep temporal C-LSTM on the original input layer L , where X 1 ,X 2 ,...,X t An image representing a plant phenotypic feature sequence at time t, through C-LSTM L Increasing the recurrent depth in time makes it easier for the memory unit to store features of nearby frames.

[0020]

[0021] 3.2 A GHU unit is adopted and effectively modeled at the original feature input stage. The formula is as follows:

[0022] Z t =GHU(H l,t ,Z t-1 ,S t )

[0023] Among them, H l,t Represents the hidden state of the current layer at the current time, S t Represents a switch gate used to control the previous state Z t-1 and the depth transition feature H l,t proportion;

[0024] 3.3 Stacking D-LSTM in high-level L L Perform high-dimensional feature learning so that the system can remember the spatial appearance in a unified memory pool and time changes By doing this, the memory state Flowing through the entire network in a zigzag direction;

[0025]

[0026] O t Represents the output representation after feature fusion in time and space. The final prediction result is obtained by fusing all the previous features in the last layer and controlling the network shape through 2D convolution. The goal of the PMD-PM model is to minimize the MSE Loss (Mean Squared Error Loss) of the validation set for each sample sequence after model training, where i represents each sample sequence (i=1,2,...,n), j represents the window sequence in each sample sequence (j=1,2,...,m), and θ represents the network parameter space. The formula is as follows:

[0027]

[0028] Preferably, in 3.1, C-LSTM L The following operations are available:

[0029] Based on the LSTM (Long Short Term Memory Networks) minimum unit, more nonlinear layers are added in the loop transition to increase the network depth from one state to the next. This module increases the number of loop states from input to output by adding an identical unit horizontally on the basis of the original minimum unit, and increases the depth of information flow in time. l,t and are two temporary memory units, where the subscript t represents the time step and the superscript represents the lth hidden layer. The current memory C l,t directly depends on its previous state C l,t-1 , and through the forget gate f t , input gate i t and input modulation gate g t Control, current memory Depends on the depth conversion path The first layer of C-LSTM L The update equation can be expressed as follows:

[0030]

[0031] C l,t =f t □C l,t-1 +i t □g t

[0032]

[0033] Where * is convolution, is element-by-element multiplication, σ is the element-by-element Sigmoid function, [] represents the concatenation of tensors, () represents a system of equations, σ is the convolution filter, where w 3 and w 5 It is a 1×1 convolution filter, which is used to change the number of filters and finally output H l,t By dual memory state C l,t and The recursive depth of information flow along the spatiotemporal transformation path increases significantly, and each pixel in the final generated frame will have a larger input volume receptive field at each time step, which makes the prediction model more capable of modeling short-term video dynamics and sudden changes.

[0034] Preferably, in 3.3, D-LSTM L The following operations are available:

[0035] The traditional forget gate is used to capture the non-stationary features and stationary features in the sequence with two cascaded LSTM minimum units, and the previous layer hidden representation H at the current time step in the network is l-1,t and the previous hidden representation H at the previous time step l-1,t-1 By performing differentiation, the network information flow can not only be transmitted horizontally in time steps and vertically in space, but also can achieve leapfrogging transmission from different levels of convolutional layers. The formula is as follows:

[0036] Δ=H l-1,t -H l-1,t-1

[0037] The difference unit combines the difference result Δ with the internal memory state N of the previous time step l,t-1 As input, multiple differences are used to eliminate the non-stationary characteristics of the sequence, so that the forget gate f l, ' t A portion of long-term memory can be retained, and the input gate i l ' ,t Control the current memory state N l,t The amount of information that needs to be saved, output gate o l ' ,t Control the internal state N at the current moment l,t Output to external state D l,t The amount of information, w u ′、w i ′、w′ f 、b u ′、b i ′, b′ f are the network weight and bias respectively, and the formula is as follows:

[0038] u l ' ,t =tanh(w u ′*[Δ,N l,t-1 ]+b u ′)

[0039] i l ' ,t =σ(w i ′*[Δ,N l,t-1 ]+b i ′)

[0040] f l, ' t =σ(w′ f *[Δ,N l,t-1 ]+b′ f )

[0041]

[0042] o l ' ,t =σ(w o ′*[Δ,N l,t ]+b o ′)

[0043]

[0044] The connecting unit will be differential state D l,t 、External time memory C l,t-1 And the internal memory unit S of this unit at the previous moment l,t-1 As input, it captures the stationary characteristics of the sequence T l,t , w u ″、w″、w″ f 、b u ″、b i ″、b″ f are network weights and biases respectively. By fusing two cascade modules, a differential memory mechanism is established to eliminate the non-stationary characteristics of the sequence. The formula is as follows:

[0045] u l ″ ,t =tanh(w u ″*[D l,t ,C l,t-1 ]+b u ″)

[0046] i l ″ ,t =σ(w i ″*[D l,t ,C l,t-1 ]+bi ″)

[0047] f l, ″ t =σ(w″ f *[D l,t ,C l,t-1 ]+b″ f )

[0048]

[0049] o l ″ ,t =σ(w o *[D l,t ,C l,t-1 ,S l,t ]+b o ″)

[0050]

[0051] Preferably, in S4, forward propagation is performed layer by layer through the PMD-PM model, and the weights and biases of neurons in each layer are adjusted in combination with the Adam optimizer and the onecycle learning rate scheduler until the set number of iterations is reached.

[0052] Preferably, in S5, the error evaluation index, the image quality evaluation index, and the phenotype feature evaluation index are as follows:

[0053] A. Standard error indicator

[0054] (1)RMSE(root-mean-squared error)

[0055]

[0056] (2) MSE (mean squared error)

[0057]

[0058] (3) MAE (mean absolute error)

[0059]

[0060] Among them, y i and are the predicted value and the true value respectively;

[0061] B. Objective evaluation indicators of image quality with real reference: SSIM, PSNR and LPIPS:

[0062] (1)SSIM (structural similarity)

[0063] Structural similarity index (SSIM) is an index used to quantify the structural similarity between two images. The formula is:

[0064]

[0065] where μ A is the mean value of A, is the variance of A, μ B is the mean of B, is the variance of B, σ AB is the covariance of A and B is constant. 1 =(k 1 L) 2 , C 2 =(k 2 L) 2 are two constants used to maintain stability, and L represents the pixel range of the image.

[0066] (2)PSNR (peak signal-to-noise ratio)

[0067] Peak Signal to Noise Ratio (PSNR) is a metric for evaluating image quality.

[0068]

[0069] Where S is the maximum possible pixel value in the image, MSE stands for mean square error,

[0070] (3) Learned Perceptual Image Patch Similarity (LPIPS)

[0071] Learned Perceptual Image Patch Similarity (LPIPS) is used to measure the difference between two images. It quantifies the perceptual similarity between image patches into Euclidean distance to evaluate the visual quality of the image.

[0072]

[0073] It is the feature vector extracted from the middle layer l after two image blocks pass through a pre-trained deep neural network Alex, the feature vector is unit normalized and a weight vector is used To weight the average of these feature vectors, and calculate the L2 distance to obtain their perceptual similarity, C. Phenotypic feature indicators

[0074] For leaves: the root mean square error (RMSE) of plant type (canopy area, canopy length, canopy width, canopy angle) and the error image describe the prediction accuracy of phenotypic micro-changes in plant growth image sequences, where the canopy area is binarized using a grayscale image, and the Otsu method is used to automatically select the optimal threshold to calculate the number of pixels in the format; the canopy length and width are the length and width of the minimum circumscribed rectangle of the segmented image, respectively; the error image is binarized using the subtract method of the predicted image and the original image, and is obtained by statistical color channel threshold segmentation.

[0075] For fruits: appearance and texture characteristics, homogeneity, describes the uniformity of the image, and characterizes the uniform distribution of the fruit color or texture; dissimilarity, describes the differences and changes between different pixels in the image, and characterizes the clarity or roughness of the texture; second-order moment ASM, describes the average value of pixel intensity, and characterizes the brightness and color distribution; total information entropy, describes the complexity and disorder of image information, and characterizes the complexity (disorder) and simplicity (order) of the texture.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] (1) The present invention provides a method for predicting plant micro-change image data sequences based on deep learning. Different from the traditional method for predicting image frames, the present invention takes into account the characteristics of image sequences formed by micro-change characteristics of plants at various stages in their life cycle, including the sampling interval of plant image sequences, short-term violent oscillations between frames, and trend non-stationarity. The method uses an increased network time depth and differential memory mechanism to better capture the complex dynamics and time dependence of plant growth. This will help to more accurately predict the growth status of plants and provide more scientific and efficient guidance for agricultural production.

[0078] (2) The present invention provides a general framework for predicting plant micro-change image sequences, including a rich type of evaluation indicators for network structures and plant bodies. It can not only process data on plants during their growth, but also process the prediction of micro-changes in fruits during the aging process, providing a solution for real-time detection of the entire agricultural industry chain, and has certain practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is the D-LSTM in the present invention L Schematic diagram of the unit structure;

[0080] Figure 2 It is the C-LSTM in the present invention L Schematic diagram of the unit structure;

[0081] Figure 3 It is the PMD-PM flow chart of the present invention

[0082] Figure 4 This is the short-term prediction result diagram of Arabidopsis thaliana in Example 1;

[0083] Figure 5 This is the long-term prediction result diagram of Arabidopsis thaliana in Example 1;

[0084] Figure 6 This is the short-term prediction result diagram of melon in Example 2;

[0085] Figure 7 This is a diagram of the long-term prediction results for melon in Example 2. DETAILED DESCRIPTION

[0086] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings and preferred embodiments.

[0087] like Figure 1 As shown, a method for predicting plant micro-change image data sequence based on deep learning includes the following steps:

[0088] 1) Data preparation: First, preprocess the raw data, identify abnormal data and remove or correct it; second, perform background removal and correction on the image sequence; finally, divide the image into training set, validation set and test set.

[0089] 2) Data analysis: ESTDA (Exploratory Spatial Temporal Data Analysis) is used to examine the temporal and spatial trends of the training data and determine whether there are large-scale trends and non-stationary features in the image sequence of plant micro-change processes. If not, the training data is a stationary process. Otherwise, a deep learning network is needed to remove these features before making predictions.

[0090] 2.1 Visualize the trend of spatial distribution of data on a large scale, that is, the "first-order" effect of spatiotemporal heterogeneous sequences. Mainly examine whether statistical characteristics such as mean, variance, covariance, etc. do not change with the evolution of time and the difference in spatial location. If the spatiotemporal series has a stationary property, it needs to meet the following requirements:

[0091]

[0092] where z i (t) is the spatiotemporal data sequence, is the sample mean, is the variance, k is the time delay, and h is the space delay.

[0093] 2.2 The ADF (Augmented Dickey-Fuller test) sequence stationarity verification method is used to verify the high-order trend of the data. When performing the ADF test, the value of the ADF statistic is compared with the critical value to determine whether to reject the null hypothesis. If the value of the ADF statistic is less than the critical value, the null hypothesis can be rejected and the sequence is considered to be stationary; otherwise, the null hypothesis cannot be rejected and the sequence is considered to be non-stationary. The formula is as follows:

[0094]

[0095] where Δy t is a difference time series, t represents the time variable, α, γ, ρ are the parameters of the model, ε t represents the white noise error term.

[0096] 3) Construct PMD-PM model: PMD-PM is divided into two parts, one of which is C-LSTM L , the other part is D-LSTM L , both use LSTM (Long Short-Term Memory Network) as the smallest unit, C-LSTM L For capturing short-term feature dependencies, D-LSTM L Used to handle high-order non-stationary features and long-term feature dependencies, a GHU unit is used between them to effectively transfer gradients in very deep feedforward networks to prevent long-term gradients from disappearing quickly.

[0097] Stacking Deep Temporal C-LSTM on the original input layer L Unit, such as Figure 2 As shown in the figure, followed by a gradient connection unit, which is used to build a time-depth module to learn short-term dynamic processes and prevent gradient disappearance; D-LSTM is stacked at a high level L Unit, such as Figure 3 As shown in the figure, the spatiotemporal long short-term memory network is used to learn the spatiotemporal features, and the differential memory mechanism is extended in the spatiotemporal recurrent network to eliminate the non-stationary characteristics of the sequence. The specific steps are as follows:

[0098] 3.1PMD-PM model stacks deep temporal C-LSTM on the original input layer L Where X 1 ,X 2 ,...,Xt An image representing a plant phenotypic feature sequence at time t, through C-LSTM L Increasing the recurrent depth in time makes it easier for the memory unit to store features of nearby frames.

[0099]

[0100] 3.2 Since deep networks often suffer from the difficulty of gradient backpropagation for a long time, the temporal memory C l,t The appearance of outdated frames may be forgotten due to the long transition. In order to maintain a stable gradient flow while increasing the depth, an information channel needs to be added between the interval frames to learn the frame skipping relationship. A GHU unit can effectively transfer gradients in very deep feedforward networks to prevent long-term gradients from disappearing quickly and effectively model them at the original feature input stage. The formula is as follows:

[0101] Z t =GHU(H l,t ,Z t-1 ,S t )

[0102] Among them, H l,t Represents the hidden state of the current layer at the current time, S t Represents a switch gate used to control the previous state Z t-1 and the depth transition feature H l,t ratio, thereby achieving adaptive learning of long-term and short-term framework relationships.

[0103] 3.3 Stacking D-LSTM in high-level L L High-dimensional feature learning is performed on data with spatiotemporal characteristics, allowing the system to remember spatial appearance in a unified memory pool and time changes By doing this, the memory state Flows through the entire network in a zigzag direction.

[0104]

[0105] O t Represents the output representation after feature fusion in time and space. The final prediction result is obtained by fusing all the previous features in the last layer and controlling the network shape through 2D convolution. The goal of the PMD-PM model is to minimize the MSE Loss (Mean Squared Error Loss) of the validation set for each sample sequence after model training, where i represents each sample sequence (i=1,2,...,n), j represents the window sequence in each sample sequence (j=1,2,...,m), and θ represents the network parameter space. The formula is as follows:

[0106]

[0107] 4) Training and saving the PMD-PM model: using the training set of the plant micro-change image sequence data set in step 1) to train the PMD-PM model, saving the best model parameters and weight coefficients during the training process, and calculating the prediction accuracy of the prediction model on the validation set.

[0108] 5) Adjust the hyperparameters of the selected network, and select error evaluation indicators, image quality evaluation indicators, and phenotypic feature evaluation indicators for performance evaluation.

[0109] 6) Fine-tuning of PMD-PM model parameters: Fine-tuning is performed based on the model prediction quality in step 4) as the basis for adjusting the model parameters.

[0110] Furthermore, in step 3.1, C-LSTM L The following operations are available:

[0111] On the basis of the LSTM minimum unit, more nonlinear layers are added in the loop transition to increase the network depth from one state to the next. This module increases the number of loop states from input to output and increases the depth of information flow in time by adding an identical unit horizontally on the basis of the original minimum unit. l,t and are two temporary memory units, where the subscript t represents the time step and the superscript represents the lth hidden layer. The current memory C l,t directly depends on its previous state C l,t-1 , and through the forget gate f t , input gate i t and input modulation gate g t Control. Current memory Depends on the depth conversion path The first layer of C-LSTM L The update equation can be expressed as follows:

[0112]

[0113]

[0114] Where * is convolution, is the element-by-element multiplication, σ is the element-by-element Sigmoid function, [] represents the concatenation of tensors, and () represents a system of equations. σ is the convolution filter, where w 3 and w 5 is a 1×1 convolution filter, which is used to change the number of filters. The final output H l,t By dual memory state Cl,t and The recursive depth of information flow along the spatiotemporal transformation path increases significantly. Each pixel in the final generated frame will have a larger receptive field of the input volume at each time step.

[0115] Furthermore, in step 3.3, D-LSTM L The following operations are available:

[0116] We use two cascaded LSTM minimum units to capture the non-stationary and stationary features in the sequence respectively. The previous layer hidden representation H at the current time step in the network l-1,t and the previous hidden representation H at the previous time step l-1,t-1 By performing differentiation, the network information flow can not only be transmitted horizontally in time steps and vertically in space, but also can achieve leapfrogging transmission from different levels of convolutional layers. The formula is as follows:

[0117] Δ=H l-1,t -H l-1,t-1

[0118] The difference unit combines the difference result Δ with the internal memory state N of the previous time step l,t-1 As input, multiple differences are used to eliminate the non-stationary characteristics of the sequence, so that the forget gate f l, ' t A portion of long-term memory can be retained, and the input gate i l ' ,t Control the current memory state N l,t The amount of information that needs to be saved, output gate o l ' ,t Control the internal state N at the current moment l,t Output to external state D l,t The amount of information, w u ′、w i ′、w′ f 、b u ′、b i ′, b′ f are the network weight and bias respectively, and the formula is as follows:

[0119] u l ' ,t =tanh(w u ′*[Δ,N l,t-1 ]+b u ′)

[0120] i l ' ,t =σ(w i ′*[Δ,N l,t-1 ]+b i′)

[0121] f l, ' t =σ(w′ f *[Δ,N l,t-1 ]+b′ f )

[0122]

[0123] o l ' ,t =σ(w o ′*[Δ,N l,t ]+b o ′)

[0124] D l,t =o t □tanh(N l,t )

[0125] The connecting unit will be differential state D l,t 、External time memory C l,t-1 And the internal memory unit S of this unit at the previous moment l,t-1 As input, it captures the stationary characteristics of the sequence T l,t , w u ″、w″、w″ f ″、b u ″、b i ″、b″ f ″ are the network weight and bias respectively, and the formula is as follows. A differential memory mechanism is established by fusing two cascade modules to eliminate the non-stationary characteristics of the sequence.

[0126] u l ″ ,t =tanh(w u ″*[D l,t ,C l,t-1 ]+b u ″)

[0127] i l ″ ,t =σ(w i ″*[D l,t ,C l,t-1 ]+b i ″)

[0128] f l, ″ t =σ(w″ f *[D l,t ,C l,t-1 ]+b″ f )

[0129]

[0130] o l ″ ,t =σ(w o *[D l,t ,C l,t-1 ,S l,t ]+b o ″)

[0131]

[0132] Furthermore, in step 3, the input image sequence is converted into a feature matrix and then processed by C-LSTM. L and D-LSTM L After that, a layer of Conv2d is connected to convert the last hidden state into a frame representation with the appropriate number of channels, and these frames are added to the next_frames list. This process is iterated between time steps to generate a series of consecutive frames to obtain the final prediction result

[0133] Furthermore, in step 4, forward propagation is performed layer by layer through the PMD-PM model, and the weights and biases of neurons in each layer are adjusted in combination with the Adam optimizer and the onecycle learning rate scheduler until the set number of iterations is reached.

[0134] Furthermore, in step 5, the error evaluation index, image quality evaluation index, and phenotype feature evaluation index are as follows:

[0135] A. Standard error indicator

[0136] (1)RMSE(root-mean-squared error)

[0137]

[0138] (2) MSE (mean squared error)

[0139]

[0140] (3) MAE (mean absolute error)

[0141]

[0142] Among them, y i and are the predicted value and the true value respectively.

[0143] B. Objective evaluation indicators of image quality with real reference: SSIM, PSNR and LPIPS:

[0144] (1)SSIM (structural similarity)

[0145] Structural similarity index (SSIM) is an indicator used to quantify the structural similarity between two images. The formula is

[0146]

[0147] where μ A is the mean value of A, is the variance of A, μ B is the mean of B, is the variance of B, σ AB is the covariance of A and B is constant. 1 =(k 1 L) 2 , C 2 =(k 2 L) 2 are two constants used to maintain stability, and L represents the pixel range of the image.

[0148] (2)PSNR (peak signal-to-noise ratio)

[0149] Peak Signal to Noise Ratio (PSNR) is a metric for evaluating image quality.

[0150]

[0151] Where S is the maximum possible pixel value in the image and MSE stands for mean squared error.

[0152] (3) Learned Perceptual Image Patch Similarity (LPIPS)

[0153] Learned Perceptual Image Patch Similarity (LPIPS) is used to measure the difference between two images. The perceptual similarity between image patches is quantified as Euclidean distance to evaluate the visual quality of the image.

[0154]

[0155] After the two image blocks pass through a pre-trained deep neural network Alex, from the middle layerl The feature vector is normalized to unit and a weight vector is used To weighted average these feature vectors, and calculate the L2 distance to obtain their perceptual similarity.

[0156] C. Phenotypic characteristic indicators

[0157] For leaves: the root mean square error (RMSE) of plant type (canopy area, canopy length, canopy width, canopy angle) and the error image describe the prediction accuracy of phenotypic micro-changes in plant growth image sequences, where the canopy area is binarized using a grayscale image, and the Otsu method is used to automatically select the optimal threshold to calculate the number of pixels in the format; the canopy length and width are the length and width of the minimum circumscribed rectangle of the segmented image, respectively; the error image is binarized using the subtract method of the predicted image and the original image, and is obtained by statistical color channel threshold segmentation.

[0158] For fruits: appearance and texture characteristics, homogeneity, describes the uniformity of the image, and characterizes the uniform distribution of the fruit color or texture; dissimilarity, describes the differences and changes between different pixels in the image, and characterizes the clarity or roughness of the texture; second-order moment ASM, describes the average value of pixel intensity, and characterizes the brightness and color distribution; total information entropy, describes the complexity and disorder of image information, and characterizes the complexity (disorder) and simplicity (order) of the texture.

[0159] Example 1

[0160] Prepare the dataset according to step 1), which is a time series of 22 overhead images of four different varieties of Arabidopsis thaliana. Each image is a single pot image, cropped from 160 pot images taken by a SLR camera. Before cropping and processing, the full-resolution images are color corrected, distortion-free, and damage-free. Continuous images are recorded at 12:00 noon every day. Together, they constitute the Arabidopsis thaliana growth micro-change image sequence dataset.

[0161] The steps for background removal are as follows:

[0162] First, in order to better distinguish plants from the background, the image is converted from RGB to L*a*b color space, and the a and b channels are selected for better distinction. Then, for each a and b component, the Otsu method is used to binarize the image and generate two binary masks to highlight the candidate foreground and background points of each channel. Using the hybrid GrabCut algorithm, the above binary masks are first combined into masks with three intensity levels, and the Gaussian mixture model (GMM) is used to model the background and foreground, and the pixels marked as possible foreground or background are coarsely segmented; secondly, the image is enhanced with interactive rectangular boxes to perform fine segmentation. It ensures that the plant will not be misjudged as noise in the early stage of growth. The Lab model effectively makes up for the lack of yellow and other colors in the RGB model, so the yellow leaves will not be misjudged as the background, and the edges of the leaves and soil will not be misjudged as the foreground. Next, we use morphology to expand the mask and remove small holes.

[0163] According to step 2), the spatiotemporal characteristics of the image sequence are checked. Through analysis, it is found that some features of the Arabidopsis image sequence have definite trends and high-order non-stationary characteristics on a large scale, so the PMD-PM method is used for prediction.

[0164] According to step 3), the data of the training set, validation set, and test set are sent to the PMD-PM model entrance, and the prediction analysis is carried out in the form of short-term prediction and long-term prediction. Figure 4 As shown, the short-term prediction takes the images at time t-3, t-2, and t-1 as input and the image at time t as output to generate a sequence of images; Figure 5 As shown in the figure, the long-term prediction takes the images at time t-6, t-5, t-4, t-3, t-2, and t-1 as input, and the images at time t, t+1, t+2, and t+3 as output to generate a sequence of images. The model prediction results are that in the short-term prediction, the cumulative mean square error (MSE) of the sample pixels and the mean of the sample structural similarity index (SSIM) are 62.7364 and 0.9050, respectively, and in the long-term prediction, they are 121.4618 and 0.8402, respectively.

[0165] Example 2

[0166] Prepare the dataset according to step 1), which is a dataset of micro-changes in the aging process of melon. The acquisition device is the MERCURY2 series industrial camera of China Daheng Image Company, equipped with Sony IMX250 global exposure CMOS sensor chip. The lens used was Computar's M0824-MPW2, which had a target size of 2 / 3", a focal length of 8mm, and an aperture range of F2.4 to F16.0. A black light-absorbing cloth was placed behind the fruit to prevent interference from external light. A light-emitting diode strip light source was used to adjust the background light. The camera was placed at a distance of 23 cm from the object, and the center of the lens had a height of 21 cm. The image data of the melons and fruits were collected using a low-speed turntable and transferred to the 3.0 data interface on the computer through a USB interface. The pictures were saved in JPG format with a size of 1944×1944 pixels. A fixed number of 5 melons were selected for image collection each time, and the image collection was performed once a day. The melon image data were recorded according to the vertical placement position. During the recording process, the fruit was placed on the low-speed turntable axis of 60s / rad, and the photography frequency was 1 / 24rad, forming a fixed data set.

[0167] According to step 2), the spatiotemporal characteristics of the image sequence are checked. Through analysis, it is found that some features of the melon aging micro-change image sequence have definite trends and high-order non-stationary characteristics on a large scale, so the PMD-PM method is used for prediction.

[0168] According to step 3), the data of the training set, validation set, and test set are sent to the PMD-PM model entrance, and the prediction analysis is carried out in the form of short-term prediction and long-term prediction. Figure 6 As shown, the short-term prediction takes the images at time t-6, t-5, t-4, t-3, t-2, and t-1 as input, and the images at time t, t+1, t+2, and t+3 as output to generate a sequence of images; Figure 7 As shown in the figure, the long-term prediction is in the form of 10 inputs and 10 outputs, that is, images at time t-10, t-9, ..., t-1 are used as inputs, and images at time t, t+1, ..., t+9 are used as outputs to generate sequential images. The model prediction results are that in the short-term prediction, the accumulated mean square error (MSE) of the sample pixels and the mean of the sample structural similarity index (SSIM) are 212.2062 and 0.7457 respectively, and in the long-term prediction, they are 211.1035 and 0.7465 respectively.

[0169] It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for predicting plant micro-change image data sequence based on deep learning, characterized in that : Includes the following steps: S1: Data preparation: First, preprocess the raw data, identify abnormal data and remove or correct it from the raw data; second, perform background removal and correction on the image sequence; finally, divide the training set, validation set and test set; S2: Data analysis: ESTDA (Exploratory Spatial Temporal Data Analysis) is used to examine the temporal and spatial trends of the training data and determine whether there are large-scale trends and non-stationary features in the image sequence of plant micro-change processes. If not, the training data is a stationary process. Otherwise, a deep learning network is needed to remove these features before prediction. S3: Building a PMD-PM model: stacking deep temporal C-LSTM on the original input layer L unit, followed by a gradient connection unit; D-LSTM is stacked at a high level L Unit, uses spatiotemporal long short-term memory network to learn spatiotemporal features, and expands the differential memory mechanism in the spatiotemporal recurrent network to eliminate the non-stationary characteristics of the sequence; S4: training and saving the PMD-PM model: using the training set of the plant micro-change image sequence data set in step S1 to train the PMD-PM model, saving the best model parameters and weight coefficients during the training process, and calculating the prediction accuracy of the prediction model on the validation set; S5: Adjust the hyperparameters of the selected network: select error evaluation indicators, image quality evaluation indicators, and phenotypic feature evaluation indicators for performance evaluation; S6: Fine-tuning of PMD-PM model parameters: Fine-tuning is performed based on the model prediction quality in step S4 as the basis for adjusting the model parameters.

2. The method for predicting plant micro-change image data sequence based on deep learning according to claim 1, characterized in that: In S2, by visualizing the trend of the spatial distribution of data on a large scale, if the spatiotemporal series has a stationary property, it needs to satisfy: where z i (t) is the spatiotemporal data sequence, is the sample mean, is the variance, k is for time delay, and h is for space delay; The ADF (Augmented Dickey-Fuller test) sequence stationarity verification method is used to verify the high-order trend of the data; when performing the ADF test, the value of the ADF statistic is compared with the critical value to determine whether to reject the null hypothesis; if the value of the ADF statistic is less than the critical value, the null hypothesis can be rejected and the sequence is considered to be stationary; otherwise, the null hypothesis cannot be rejected and the sequence is considered to be non-stationary; the formula is as follows: where Δy t is a difference time series, t represents the time variable, α, γ, ρ are the parameters of the model, ε t represents the white noise error term.

3. The method for predicting plant micro-change image data sequence based on deep learning according to claim 1, characterized in that: In S3, the following steps are specifically included: 3.1PMD-PM model stacks deep temporal C-LSTM on the original input layer L , where X1,X2,...,X t An image representing a plant phenotypic feature sequence at time t, through C-LSTM L Increasing the recurrent depth in time makes it easier for the memory unit to store features of nearby frames. 3.2 A GHU unit is adopted and effectively modeled at the original feature input stage. The formula is as follows: With t =GHU(H l,t ,With t-1 ,S t ) Among them, H l,t Represents the hidden state of the current layer at the current time, S t Represents a switch gate used to control the previous state Z t-1 and the depth transition feature H l,t proportion; 3.3 Stacking D-LSTM in high-level L L High-dimensional feature learning is performed on data with spatiotemporal characteristics, allowing the system to remember spatial appearance in a unified memory pool and time changes By doing this, the memory state Flowing through the entire network in a zigzag direction; O t Represents the output representation after feature fusion in time and space. The final prediction result is obtained by fusing all the previous features in the last layer and controlling the network shape through 2D convolution. The goal of the PMD-PM model is to minimize the MSE Loss (Mean Squared Error Loss) of the validation set for each sample sequence after model training, where i represents each sample sequence (i=1,2,...,n), j represents the window sequence in each sample sequence (j=1,2,...,m), and θ represents the network parameter space. The formula is as follows:

4. The method for predicting plant micro-change image data sequence based on deep learning according to claim 3, characterized in that: In 3.1, C-LSTM L The following operations are available: On the basis of the LSTM minimum unit, more nonlinear layers are added in the loop transition to increase the network depth from one state to the next. This module increases the number of loop states from input to output and increases the depth of information flow in time by adding an identical unit horizontally on the basis of the original minimum unit. l,t and are two temporary memory units, where the subscript t represents the time step and the superscript represents the lth hidden layer. The current memory C l,t directly depends on its previous state C l,t-1 , and through the forget gate f t , input gate i t and input modulation gate g t Control, current memory Depends on the depth conversion path The first layer of C-LSTM L The update equation can be expressed as follows: Where * is convolution, is element-by-element multiplication, σ is the element-by-element Sigmoid function, [] represents the concatenation of tensors, () represents a system of equations, σ is a convolution filter, where w3 and w5 are 1×1 convolution filters used to change the number of filters, and the final output is H l,t By dual memory state C l,t and The recursive depth of information flow along the spatiotemporal transformation path increases significantly, and each pixel in the final generated frame will have a larger input volume receptive field at each time step, which makes the prediction model more capable of modeling short-term video dynamics and sudden changes.

5. The method for predicting plant micro-change image data sequence based on deep learning according to claim 3, characterized in that: In 3.3, D-LSTM L The following operations are available: The traditional forget gate is used to capture the non-stationary features and stationary features in the sequence with two cascaded LSTM minimum units, and the previous layer hidden representation H at the current time step in the network is l-1,t and the previous hidden representation H at the previous time step l-1,t-1 By performing differentiation, the network information flow can not only be transmitted horizontally in time steps and vertically in space, but also can achieve leapfrogging transmission from different levels of convolutional layers. The formula is as follows: Δ=H l-1，t -H l-1，t-1 The difference unit combines the difference result Δ with the internal memory state N of the previous time step l，t-1 As input, multiple differences are used to eliminate the non-stationary characteristics of the sequence, so that the forget gate f′ l，t A portion of long-term memory can be retained, and the input gate i′ l，t Control the current memory state N l，t The amount of information that needs to be saved, output gate o′ l，t Control the internal state N at the current moment l，t Output to external state D l，t The amount of information, w′ u , w′ i , w′ f , b′ u , b′ i , b′ f are the network weight and bias respectively, and the formula is as follows: u′ l，t =tanh(w′ u *[Δ,N l，t-1 ]+b′ u ) i′ l，t =σ(w′ i *[D,N l，t-1 ]+b′ i ) f′ l，t =σ(w′ f *[D,N l，t-1 ]+b′ f ) oh l，t =σ(w′ o *[D,N l，t ]+b′ o ) The connecting unit will be differential state D l，t 、External time memory C l，t-1 And the internal memory unit S of this unit at the previous moment l，t-1 As input, it captures the stationary characteristics of the sequence T l，t ,w″ u ,w″,w″ f , b″ u , b″ i , b″ f are network weights and biases respectively. By fusing two cascade modules, a differential memory mechanism of spatiotemporal cyclic network extension is established to eliminate the non-stationary characteristics of the sequence. The formula is as follows: u″ l，t =tanh(w″ u *[D l，t C l，t-1 ]+b″ u ) i″ l，t =σ(w″ i *[D l，t ,C l，t-1 ]+b″ i ) f″ l，t =σ(w″ f *[D l，t ,C l，t-1 ]+b″ f ) oh l，t =σ(w o *[D l，t ,C l，t-1 ,S l，t ]+b″ o ) 6. The method for predicting plant micro-change image data sequence based on deep learning according to claim 1, characterized in that: In S4, the PMD-PM model is used to perform forward propagation layer by layer, and the weights and biases of neurons in each layer are adjusted in combination with the Adam optimizer and the onecycle learning rate scheduler until the set number of iterations is reached.

7. The method for predicting plant micro-change image data sequence based on deep learning according to claim 1, characterized in that: In S5, the error evaluation index, image quality evaluation index, and phenotype feature evaluation index are as follows: A. Standard error indicator (1)RMSE(root-mean-squared error) (2) MSE (mean squared error) (3) MAE (mean absolute error) Among them, y i and are the predicted value and the true value respectively; B. Objective evaluation indicators of image quality with real reference: SSIM, PSNR and LPIPS: (1)SSIM (structural similarity) Structural similarity index (SSIM) is an index used to quantify the structural similarity between two images. The formula is: where μ A is the mean of A, is the variance of A, μ B is the mean of B, is the variance of B, σ AB The covariance of A and B is constant. C1=(k1L) 2 , C2=(k2L) 2 are two constants used to maintain stability, and L represents the pixel range of the image. (2)PSNR (peak signal-to-noise ratio) Peak Signal to Noise Ratio (PSNR) is a metric for evaluating image quality. Where S is the maximum possible pixel value in the image, MSE stands for mean square error, (3) Learned Perceptual Image Patch Similarity (LPIPS) Learned Perceptual Image Patch Similarity (LPIPS) is used to measure the difference between two images. It quantifies the perceptual similarity between image patches into Euclidean distance to evaluate the visual quality of the image. It is the feature vector extracted from the middle layer l after two image blocks pass through a pre-trained deep neural network Alex, the feature vector is unit normalized and a weight vector is used To weight the average of these feature vectors, and calculate the L2 distance to obtain their perceptual similarity, C. Phenotypic feature indicators For leaves: the root mean square error (RMSE) of plant type (canopy area, canopy length, canopy width, canopy angle) and error image describe the prediction accuracy of phenotypic micro-changes in plant growth image sequence, where the canopy area is binarized using grayscale images, and the Otsu method is used to automatically select the optimal threshold to calculate the number of pixels in the format; the canopy length and width are the length and width of the minimum circumscribed rectangle of the segmented image, respectively; the error image is binarized using the subtract method of the predicted image and the original image, and is obtained by statistical color channel threshold segmentation; For fruits: appearance and texture characteristics, homogeneity, describes the uniformity of the image, and characterizes the uniform distribution of the fruit's color or texture; dissimilarity, describes the differences and changes between different pixels in the image, and characterizes the clarity or roughness of the texture; second-order moment ASM, describes the average value of pixel intensity, and characterizes the brightness and color distribution; total information entropy, describes the complexity and confusion of image information, and characterizes the complexity and simplicity of texture.

Images