A two-stage crop growth prediction method based on multi-modal information

Abstract

The application discloses a two-stage crop growth prediction method based on multi-modal information, which comprises the following steps: collecting video images of the growth state of crops and recording the shooting time of the video images; extracting a key area in the video images to obtain a feature map; identifying and training the feature map based on a 3D convolutional neural network recognition model to obtain complete crop growth state data; collecting real-time data of the growth environment around the crops through a sensor, performing correlation analysis and sorting on different real-time data and complete crop growth cycle data to obtain a correlation feature list; and predicting the crop growth condition based on the correlation feature list and a prediction crop growth model. The application solves the problem that the existing agricultural images are separated from the real application scene, the results cannot be migrated to the real crop planting, the personalized factor is strong, the stability is not high, and the single modal crop data is analyzed and predicted, so that the crop growth prediction accuracy is not high.

Description

Technical Field

[0001] This invention relates to the field of crop growth monitoring technology, and in particular to a two-stage crop growth prediction method based on multimodal information. Background Technology

[0002] Many crops are now grown on a large scale and are making significant strides towards intelligent manufacturing, greatly reducing manpower in crop monitoring and management. The use of artificial intelligence has been explored in areas such as seedling selection and greenhouse crop cultivation. Smart agriculture is moving towards goals of high quality, high yield, low energy consumption, and automation, covering the entire stage of agricultural production.

[0003] Computer vision image recognition and sensor information acquisition are important components of smart agriculture technology, and monitoring and predicting crop growth is a key application scenario of this invention. Currently, many crop growing bases are equipped with cameras, but the number and types of images collected are limited, which leads to a decrease in the final performance of data-driven deep learning algorithms. However, existing open-source agricultural image datasets are often detached from real-world application scenarios, and results obtained using such datasets cannot be transferred to actual crop cultivation. Self-built agricultural image data has strong individual factors and low stability. Furthermore, existing crop growth models often use data from a single modality for analysis and prediction, resulting in low accuracy in crop growth prediction. Summary of the Invention

[0004] This invention provides a two-stage crop growth prediction method based on multimodal information to overcome the problems that existing open-source agricultural image datasets are often detached from real application scenarios, and the results obtained using such datasets cannot be transferred to real crop planting. Furthermore, self-built agricultural image data has strong personalized factors and low stability. At the same time, existing crop growth models often use data from a single modality for analysis and prediction, resulting in low accuracy in crop growth prediction.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A two-stage crop growth prediction method based on multimodal information includes:

[0007] Step S1: Periodically collect video images of the growth status and planting area of ​​crops, and record the shooting time of the video images. The video images are a sequence of images containing time characteristics.

[0008] Step S2: Extract key regions from the video image and obtain feature maps, wherein the feature maps are images containing crops;

[0009] Step S3: Based on the 3D convolutional neural network recognition model, the feature map is trained to obtain the spatial and temporal features of the crop, so as to obtain complete state data of the crop growth cycle; the spatial features include the number of leaves, leaf area and plant height of the crop, and the temporal features are the growth stages of the crop.

[0010] Step S4: Periodically collect real-time data of the crop's surrounding environment using sensors. Perform correlation analysis on different real-time data and the state data of the complete crop growth cycle, and sort them to obtain a correlation feature list L. rank ;

[0011] The real-time data includes air temperature, air humidity, light intensity, CO2 concentration, soil temperature, soil humidity, soil nitrogen content, soil phosphorus content, and soil potassium content.

[0012] Step S5: Establish a crop growth prediction model, which is based on the correlation feature list L rank Predicting crop growth.

[0013] Furthermore, the acquisition of the feature map in step S2 specifically involves:

[0014] Step S2.1: Extract video images according to a preset fixed time period, and cut several original images from the video images. The video images are obtained by programming to obtain coarse-grained area images containing crops.

[0015] Step S2.2: The coarse-grained region image is processed by Haar-like feature extraction to obtain rectangular features of crop growth status. The Haar-like features include edge features, linear features, center features, and diagonal features.

[0016] Step S2.3: Apply the integral image calculation method to obtain the rectangular feature value of the rectangular feature, wherein the integral image includes a vertical integral image and a 45-degree integral image;

[0017] Step S2.4: Use the Adaboost algorithm to select images containing rectangular features of crop growth status based on the rectangular feature values, and form a corresponding weak classifier. Each rectangular feature containing crops forms a fixed region, and these fixed regions are superimposed by weighted averaging.

[0018] Since each of the fixed regions is a weak classifier, the weak classifiers are stacked to form a strong classifier by weighted voting; and the strong classifiers are connected in series to form a cascaded classifier.

[0019] Step S2.5: The cascaded classifier re-extracts features from the original image based on the location of the crops in the image to obtain a feature map.

[0020] Furthermore, the 3D convolutional neural network recognition model described in step S3 starts from the input and sequentially includes a first hard-connected layer, a second convolutional layer, a third downsampling layer, a fourth convolutional layer, a fifth downsampling layer, a sixth convolutional layer, a seventh attention layer, an eighth fully connected layer, and a ninth normalized exponential function layer.

[0021] The first hard-connected layer is used to classify the image frames of the feature map to obtain feature channel information, and pass the feature channel information to the second convolutional layer. The feature channel information includes gray level, horizontal gradient, vertical gradient, horizontal optical flow field and vertical optical flow field.

[0022] The second convolutional layer performs a convolution operation on the feature channel information output by the first hard-connected layer and obtains an optimized feature map that is then passed to the third downsampling layer.

[0023] The third downsampling layer receives the optimized feature map and performs a downsampling operation to obtain the first intermediate feature map. The third downsampling layer is a pooling layer.

[0024] The fourth convolutional layer performs a convolution operation on the first intermediate feature map output by the third downsampling layer to obtain the second intermediate feature map.

[0025] The fifth downsampling layer receives the second intermediate feature map and performs a downsampling operation to obtain the third intermediate feature map;

[0026] The sixth convolutional layer performs a convolution operation on the third intermediate feature map to obtain the fourth intermediate feature map, which is then transmitted to the seventh attention layer.

[0027] The seventh attention layer filters the fourth intermediate feature map according to different weight values ​​to obtain the optimal feature map;

[0028] The optimal feature map is sequentially passed through the eighth fully connected layer and the ninth normalized exponential function layer for classification and recognition, thereby obtaining complete data on the crop growth status.

[0029] Furthermore, in step S4, the relevance feature list L is obtained. rank Specifically

[0030] The Person analysis method was used to perform correlation analysis between the different real-time data and the complete crop growth cycle data. The calculation method is shown in formula (1):

[0031]

[0032] Wherein, variables x and y represent the sensor features and the growth state recognition results at the same time, respectively. These are the means of variables x and y, respectively. i y i , i, and y are the values ​​of the i-th observation point of variables x and y, respectively. The correlation coefficient r ranges from [-1, 1] and represents the degree of linear correlation between variables x and y. The larger the absolute value of r, the stronger the correlation between variables x and y. n is the feature dimension.

[0033] After performing correlation analysis on the different real-time data and complete crop growth cycle data, the data are sorted in descending order to obtain a correlation feature list L based on the correlation analysis. rank .

[0034] Furthermore, the crop growth prediction model described in step S5 is specifically as follows:

[0035] Step S5.1: Establish a bidirectional three-layer LSTM model. The structure of the LSTM model is shown in formula (2):

[0036]

[0037] Among them, W f W represents the weight of the forgotten door. i The weights represent the input gates; The weight of the unit; W o b represents the weight of the output gate; f Represents W f The corresponding bias; b i Represents W i The corresponding bias; represent Corresponding bias; b o Represents W o Corresponding bias; x t The input sequence; h t This represents the current hidden layer state; h t-1 This is the state of the previous hidden layer; f t Forgot the door; i t For input gate; o t For output gate; C t-1 For the previous unit; C t This is the current unit.

[0038] Step S5.2: The bidirectional three-layer LSTM model receives real-time data of the crop's surrounding growth environment periodically collected by sensors, and uses the mean squared error as an evaluation index to obtain the list of correlation features with the lowest mean squared error, L. rank subsequence;

[0039] Step S5.3: Using the aforementioned list of relevance features L rank The subsequence is used as the input of a bidirectional three-layer LSTM model. By training the real-time data in chronological order / reverse order, the corresponding crop growth status data is captured and transmitted to the fully connected layer for classification to obtain the real-time data corresponding to the crop growth status data.

[0040] Step S5.4: The crop growth prediction model uses real-time data corresponding to the crop growth status data and the 3D convolutional neural network recognition model to predict the future crop growth status.

[0041] Beneficial Effects: This invention provides a two-stage crop growth prediction method based on multimodal information. Addressing the issue of insufficient crop image data, it uses a camera to collect real-time video images of crop growth data throughout the entire crop growth cycle, extracts key regions from the video images to obtain feature maps, and uses a 3D convolutional neural network to identify the crop growth status. Simultaneously, it uses sensors to periodically collect real-time data of the crop's surrounding environment, obtaining the most contributing real-time data, and jointly predicts the identification results of the crop growth model. This improves the accuracy of crop growth prediction, enabling precise prediction of future plant growth status. It facilitates providing daily crop growth status information to farmers and businesses, allowing for timely reminders for agricultural activities such as crop protection and harvesting. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a two-stage crop growth prediction method based on multimodal information according to the present invention.

[0044] Figure 2 This is a flowchart of a two-stage crop growth prediction method based on multimodal information according to the present invention;

[0045] Figure 3 This is a schematic diagram of the structure of a 3D convolutional neural network recognition model for a two-stage crop growth prediction method based on multimodal information according to the present invention.

[0046] Figure 4 This is a schematic diagram of the crop growth prediction model structure of a two-stage crop growth prediction method based on multimodal information according to the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] This embodiment provides a two-stage crop growth prediction method based on multimodal information, such as... Figures 1 to 2 As shown, it includes:

[0049] This invention, combined with specific scenarios of crop growth, utilizes a two-stage crop growth prediction method based on deep learning and multimodal information to identify and predict crop growth. The method consists of two stages: a crop growth status identification stage based on a 3D convolutional neural network recognition model and a crop growth prediction stage based on a predictive crop growth model. The specific implementation method is as follows:

[0050] Step S1: Periodically collect video images of the growth status and planting area of ​​crops, and record the shooting time of the video images. The video images are a sequence of images with time characteristics. The video images are collected using cameras, including telephoto cameras and short-focus cameras.

[0051] Step S2: Extract key regions from the video image and obtain feature maps, wherein the feature maps are images containing crops;

[0052] Step S3: Based on the 3D convolutional neural network recognition model, the feature map is trained to obtain the spatial and temporal features of the crop, so as to obtain complete state data of the crop growth cycle; the spatial features include the number of leaves, leaf area and plant height of the crop, and the temporal features are the growth stages of the crop.

[0053] Step S4: Periodically collect real-time data of the crop's surrounding environment using sensors. Perform correlation analysis on different real-time data and the state data of the complete crop growth cycle, and sort them to obtain a correlation feature list L. rank ;

[0054] The real-time data includes air temperature, air humidity, light intensity, CO2 concentration, soil temperature, soil humidity, soil nitrogen content, soil phosphorus content, and soil potassium content.

[0055] Step S5: Establish a crop growth prediction model, which is based on the correlation feature list L rank Predicting crop growth.

[0056] The system uses cameras to collect real-time video images of crop growth data throughout the entire crop growth cycle, extracts key regions from the video images to obtain feature maps, and uses a 3D convolutional neural network to identify the crop growth status. It also uses sensors to periodically collect real-time data of the crop's surrounding growth environment, obtains the most contributing real-time data, and jointly predicts the identification results of the crop growth model, thereby improving the accuracy of crop growth prediction.

[0057] In a specific embodiment, obtaining the feature map in step S2 specifically involves:

[0058] Step S2.1: Extract video images according to a preset fixed time period, and cut several original images from the video images. The video images are obtained by programming to obtain coarse-grained area images containing crops.

[0059] Crop growth status recognition uses video modal sequence information to effectively identify crop status. This is a task that classifies crops based on the deformation or motion trajectory of image features. During crop growth, deformation occurs in parts such as roots, leaves, and stems. If these changes are extracted, the crop growth status can be identified. Since the video images are acquired by cameras installed outdoors, the outdoor environment is unlikely to change significantly in a short period of time. Therefore, we select 1 second of video every 2 hours as the basis, and then cut this 1 second of video into 100ms units. Finally, we can capture 10 original images from the video every 2 hours.

[0060] Step S2.2: The coarse-grained region image is processed by Haar-like feature extraction to obtain rectangular features of crop growth status. The Haar-like features include edge features, linear features, center features, and diagonal features.

[0061] Step S2.3: Apply the integral image calculation method to obtain the rectangular feature value of the rectangular feature, wherein the integral image includes a vertical integral image and a 45-degree integral image;

[0062] Step S2.4: Use the Adaboost algorithm to select images containing rectangular features of crop growth status based on the rectangular feature values, and form a corresponding weak classifier. Each rectangular feature containing crops forms a fixed region, and these fixed regions are superimposed by weighted averaging.

[0063] The weak classifiers are stacked together to form a strong classifier using a weighted voting method; and the strong classifiers are then connected in series to obtain a cascaded classifier.

[0064] Step S2.5: The cascaded classifier re-extracts features from the original image based on the location of the crops in the image to obtain a feature map.

[0065] Based on the location of crops in the image, feature maps are re-extracted from all original images according to the sampling frequency. The feature maps are then combined to generate new video images, where only images containing crops are retained, and the mapping between video segments and time nodes is realized.

[0066] In a specific embodiment, such as Figure 3 As shown, the 3D convolutional neural network recognition model described in step S3 starts from the input and includes, in sequence, a first hard-connected layer, a second convolutional layer, a third downsampling layer, a fourth convolutional layer, a fifth downsampling layer, a sixth convolutional layer, a seventh attention layer, an eighth fully connected layer, and a ninth normalized exponential function layer.

[0067] The first hard-connected layer is used to classify the image frames of the feature map to obtain feature channel information, and pass the feature channel information to the second convolutional layer. The feature channel information includes gray level, horizontal gradient, vertical gradient, horizontal optical flow field and vertical optical flow field.

[0068] The first hard-wired layer uses a fixed hardwired kernel to process the original image frames of the feature map and generate 5 feature channel information. The grayscale, horizontal gradient, and vertical gradient channels are calculated on a frame-by-frame basis. The horizontal optical flow field and the vertical optical flow are calculated once every frame. The feature channel information requires information from two frames to be extracted.

[0069] The second convolutional layer uses a 7*7*3 3D convolutional kernel to perform convolution operations on the feature channel information output by the first hard-connected layer, and obtain two sets of two-dimensional optimized feature maps with a resolution of 54*34.

[0070] The third downsampling layer receives the optimized feature map, performs a downsampling operation on the optimized feature map using a 2*2 sampling window, and outputs a first intermediate feature map with a resolution reduced to 27*17. The third downsampling layer is a pooling layer.

[0071] The fourth convolutional layer uses a 7*6*3 3D convolutional kernel to perform a convolution operation on the first intermediate feature map output by the third downsampling layer, to obtain a second intermediate feature map with a resolution of 21*12.

[0072] The fifth downsampling layer receives a second intermediate feature map with a resolution of 21*12, and performs a downsampling operation on the second intermediate feature map with a resolution of 21*12 using a 3*3 sampling window to obtain a third intermediate feature map with a resolution of 7*4.

[0073] The sixth convolutional layer uses a 2D convolutional kernel with a resolution of 7*4 to perform a convolution operation on the third intermediate feature map, obtains a fourth intermediate feature map with a resolution of 1*1, and transmits it to the seventh attention layer.

[0074] The seventh attention layer filters the fourth intermediate feature map with a resolution of 1*1 according to different weight values ​​to obtain the optimal feature map with a resolution of 128*1.

[0075] The optimal 128*1 feature map is sequentially passed through the eighth fully connected layer and the ninth normalized exponential function layer for classification and recognition. The feature size of all 128*1 two-dimensional feature maps is flattened into one dimension to obtain complete data on crop growth status.

[0076] A 3D Convolutional Neural Network (3D CNN) recognition model is designed to capture crop features along spatial and temporal dimensions and classify information such as crop growth stage, leaves, plant height, and leaf area. This 3D CNN recognition model can generate multiple information channels from adjacent frames, performing convolution and downsampling operations in each channel. Furthermore, the feature maps in the convolutional layers are connected to multiple adjacent frames in the previous layer. To extract different types of feature maps, two convolutional kernels of the same size but different kernel features are used at each location. This results in two sets of feature maps of the same size in the second convolutional layer. The third downsampling layer uses a max pooling layer to downsample the feature map of the second convolutional layer, thus obtaining a feature map with reduced spatial resolution. The fourth convolutional layer uses 9*9*3 3D convolutional kernels in each of the five channels. The fifth downsampling layer performs a downsampling operation. The sixth convolutional layer only performs convolutions in the spatial dimension. The seventh attention layer filters and selects feature maps with different weights, amplifying effective features and suppressing ineffective features. Finally, the data passes through a fully connected layer and a normalized exponential function (Softmax) layer to output information such as predicted crop growth stage, leaf size, plant height, and leaf area.

[0077] In a specific embodiment, the step S4 of obtaining the relevance feature list L rank Specifically

[0078] To preliminarily determine the reliability of different environmental characteristics on crop growth, a correlation analysis was first performed based on sensor information. Here, the Person method was selected for correlation analysis. The Person analysis method was used to perform correlation analysis between different real-time data and complete crop growth cycle data. The calculation method is shown in formula (1):

[0079]

[0080] Wherein, variables x and y represent the sensor features and the growth state recognition results at the same time, respectively. These are the means of variables x and y, respectively. i y i , i, and y are the values ​​of the i-th observation point of variables x and y, respectively. The correlation coefficient r ranges from [-1, 1] and represents the degree of linear correlation between variables x and y. The larger the absolute value of r, the stronger the correlation between variables x and y. n is the feature dimension.

[0081] After performing correlation analysis on the different real-time data and complete crop growth cycle data, the data are sorted in descending order to obtain a correlation feature list L based on the correlation analysis. rank ;

[0082] In a specific embodiment, such as Figure 4 As shown, the crop growth prediction model mentioned in step S5 is specifically as follows:

[0083] Step S5.1: Establish a bidirectional three-layer LSTM model. The structure of the LSTM model is shown in formula (2):

[0084]

[0085] Among them, W f W represents the weight of the forgotten door. i The weights represent the input gates; The weight of the unit; W o b represents the weight of the output gate; f Represents W f The corresponding bias; b i Represents W i The corresponding bias; represent Corresponding bias; b o Represents W o Corresponding bias; x t The input sequence; h t This represents the current hidden layer state; h t-1 This is the state of the previous hidden layer; f t Forgot the door; i t For input gate; o tFor output gate; C t-1 For the previous unit; C t For the current unit, To obtain C t Intermediate variables; forget gate f t Compared to the previous state C t-1 The decision on whether to retain the data is made jointly;

[0086] A recurrent neural network model is trained using a Long Short-Term Memory (LSTM) network to find the feature list L. rank The effective features are used to predict crop growth. Since the real-time information collected by the sensor is based on time series information, and the real-time data is correlated with the crop growth status data, a neural network model with short-term memory is considered. An LSTM model is introduced to control the accumulation rate of input information, selectively adding new real-time data and crop growth status data, and selectively forgetting previously accumulated input information. The forget gate is a non-linear function that can control the accumulation of memory information or selectively forget the memory information.

[0087] Step S5.2: The bidirectional three-layer LSTM model receives real-time data of the crop's surrounding growth environment periodically collected by sensors, and uses the mean squared error as an evaluation index to obtain the list of correlation features with the lowest mean squared error, L. rank subsequence;

[0088] Step S5.3: Using the aforementioned list of relevance features L rank The subsequence is used as the input of a bidirectional three-layer LSTM model. By training the real-time data in chronological order / reverse order, the corresponding crop growth status data is captured and transmitted to the fully connected layer for classification to obtain the real-time data corresponding to the crop growth status data.

[0089] Step S5.4: The crop growth prediction model uses real-time data corresponding to the crop growth status data and the 3D convolutional neural network recognition model to predict the future crop growth status.

[0090] A bidirectional, three-layer LSTM model was adopted. "Bidirectional" means there are two recurrent layers transmitting opposite information. The first recurrent layer transmits information sequentially in time, and the second recurrent layer transmits information in reverse chronological order. This implies that both past and future information can be successfully captured. This is because the temporal characteristics of real-time data acquired by the sensor can be determined by the input information from several frames of the previous and next time points. Therefore, a bidirectional, three-layer LSTM model (for predicting crop growth) was designed based on this idea, using the correlation feature list L... rankThe subsequences are used as input to a bidirectional three-layer LSTM model, with the crop growth status identification results at the same time point as labels. The mean squared error is used as the evaluation metric to find the feature list L with the lowest mean squared error. rank The method uses subsequences of crops as input to a filtered LSTM model to predict future crop growth status. This method can provide farmers and businesses with daily crop growth status information, offering timely reminders for important agricultural activities such as crop protection and harvesting. Furthermore, the two-stage crop growth prediction method based on multimodal information presented in this paper has strong versatility and can be applied to the identification and prediction of growth status for more crops. The predicted crop growth status obtained by this method has high accuracy, broad coverage of crop growth information obtained based on multimodal information, low error, good robustness, and strong fault tolerance. Simultaneously, this method is applicable to self-built crop image and text data sources. Users can independently train the model to adapt to environmental changes, and the feature extraction list of real-time sensor data can adapt to the characteristics of user-built data sources, thereby finding the real-time sensor data that best represents the current environment.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A two-stage crop growth prediction method based on multimodal information, characterized in that, include: Step S1: Periodically collect video images of the growth status and planting area of ​​crops, and record the shooting time of the video images. The video images are a sequence of images containing time characteristics. Step S2: Extract key regions from the video image and obtain feature maps, wherein the feature maps are images containing crops; Step S3: Based on the 3D convolutional neural network recognition model, the feature map is trained to obtain the spatial and temporal features of the crop, so as to obtain complete state data of the crop growth cycle; the spatial features include the number of leaves, leaf area and plant height of the crop, and the temporal features are the growth stages of the crop. Step S4: Periodically collect real-time data of the crop's surrounding environment using sensors. Perform correlation analysis on different real-time data and the status data of the complete crop growth cycle, and sort them to obtain a list of correlation features. ; The real-time data includes air temperature, air humidity, light intensity, CO2 concentration, soil temperature, soil humidity, soil nitrogen content, soil phosphorus content, and soil potassium content. Step S5: Establish a crop growth prediction model, which is based on a list of correlation features. Predicting crop growth; The specific model for predicting crop growth is as follows. Step S5.1: Establish a bidirectional three-layer LSTM model. The structure of the LSTM model is shown in formula (2): （2） in, This represents the weight of the forgotten door; The weights represent the input gates; The weight of the representative unit; Represents the weight of the output gate; represent The corresponding bias; represent The corresponding bias; represent Corresponding bias; represent Corresponding bias; The input sequence; This represents the current hidden layer state. This is the state of the previous hidden layer; Forgot the door; For input gates; For output gate; Ct represents the previous element; Ct represents the current element. Step S5.2: The bidirectional three-layer LSTM model receives real-time data of the crop's surrounding growth environment periodically collected by sensors, and uses the mean squared error as an evaluation index to obtain a list of correlation features with the lowest mean squared error. subsequence; Step S5.3: Using the aforementioned list of relevance features The subsequence is used as the input of a bidirectional three-layer LSTM model. By training the real-time data in chronological order / reverse order, the corresponding crop growth status data is captured and transmitted to the fully connected layer for classification to obtain the real-time data corresponding to the crop growth status data. Step S5.4: The crop growth prediction model uses real-time data corresponding to the crop growth status data and the 3D convolutional neural network recognition model to predict the future crop growth status.

2. The two-stage crop growth prediction method based on multimodal information according to claim 1, characterized in that, The specific steps for obtaining the feature map in step S2 are as follows: Step S2.1: Extract video images according to a preset fixed time period, and cut several original images from the video images. The video images are obtained by programming to obtain coarse-grained area images containing crops. Step S2.2: The coarse-grained region image is processed by Haar-like feature extraction to obtain rectangular features of crop growth status. The Haar-like features include edge features, linear features, center features, and diagonal features. Step S2.3: Apply the integral image calculation method to obtain the rectangular feature value of the rectangular feature, wherein the integral image includes a vertical integral image and a 45-degree integral image; Step S2.4: Use the Adaboost algorithm to select images containing rectangular features of crop growth status based on the rectangular feature values, and form a corresponding weak classifier. Each rectangular feature containing crops forms a fixed region, and these fixed regions are superimposed by weighted averaging. The weak classifiers are stacked together to form a strong classifier using a weighted voting method; and the strong classifiers are then connected in series to obtain a cascaded classifier. Step S2.5: The cascaded classifier re-extracts features from the original image based on the location of the crops in the image to obtain a feature map.

3. The two-stage crop growth prediction method based on multimodal information according to claim 1, characterized in that, The 3D convolutional neural network recognition model described in step S3 starts from the input and includes, in sequence, a first hard-connected layer, a second convolutional layer, a third downsampling layer, a fourth convolutional layer, a fifth downsampling layer, a sixth convolutional layer, a seventh attention layer, an eighth fully connected layer, and a ninth normalized exponential function layer. The first hard-connected layer is used to classify the image frames of the feature map to obtain feature channel information, and pass the feature channel information to the second convolutional layer. The feature channel information includes gray level, horizontal gradient, vertical gradient, horizontal optical flow field and vertical optical flow field. The second convolutional layer performs a convolution operation on the feature channel information output by the first hard-connected layer and obtains an optimized feature map that is then passed to the third downsampling layer. The third downsampling layer receives the optimized feature map and performs a downsampling operation to obtain the first intermediate feature map. The third downsampling layer is a pooling layer. The fourth convolutional layer performs a convolution operation on the first intermediate feature map output by the third downsampling layer to obtain the second intermediate feature map. The fifth downsampling layer receives the second intermediate feature map and performs a downsampling operation to obtain the third intermediate feature map; The sixth convolutional layer performs a convolution operation on the third intermediate feature map to obtain the fourth intermediate feature map, which is then transmitted to the seventh attention layer. The seventh attention layer filters the fourth intermediate feature map according to different weight values ​​to obtain the optimal feature map; The optimal feature map is sequentially passed through the eighth fully connected layer and the ninth normalized exponential function layer for classification and recognition, thereby obtaining complete data on the crop growth status.

4. The two-stage crop growth prediction method based on multimodal information according to claim 1, characterized in that, The process of obtaining the relevance feature list in step S4 Specifically The correlation between the different real-time data and the complete crop growth cycle data was analyzed using the Pearson analysis method. The calculation method is shown in formula (1): （1） Wherein, variables x and y are the sensor features and the growth state recognition results at the same time, respectively. , These are the means of variables x and y, respectively. , , i, and y are the values ​​of the i-th observation points of variables x and y, respectively. The correlation coefficient r ranges from [-1, 1] and represents the degree of linear correlation between variables x and y. The larger the absolute value of r, the stronger the correlation between variables x and y. n is the feature dimension. After performing correlation analysis on the different real-time data and complete crop growth cycle data, the data were sorted in descending order to obtain a list of correlation features based on the correlation analysis. .

Images