Crop identification and growth process prediction method and system based on multi-modal data

Abstract

The invention belongs to the technical field of intelligent agricultural data analysis, and relates to a crop identification and growth process prediction method and system based on multi-modal data. The method comprises the following steps: collecting growth process images and multi-modal environment data of crops in each growth stage; constructing a crop feature data set; constructing a sample data set; training to obtain a crop type identification model; obtaining a crop type identification result; generating a fusion feature vector; performing data enhancement processing; constructing a long-short-term memory network for training to obtain a growth stage prediction model; and predicting the growth stage of the crops. Crop type identification is performed by constructing a deep neural network model, feature fusion is performed on crop growth process image features and multi-modal environment data, crop growth stage prediction is performed by using a long-short-term memory network, fertilization, irrigation and pest control measures are optimized in combination with the multi-modal environment data, and the crop growth stage prediction accuracy is improved. The crop yield and quality are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart agricultural data analysis, and specifically, relates to a method and system for crop identification and growth process prediction based on multimodal data. Background Art

[0002] The development of modern agriculture faces many challenges, such as climate change, soil degradation, and limited resources. Accurate crop identification and growth stage prediction are not only crucial to improving crop yields and quality, but also effectively reduce agricultural production costs. Traditional crop identification and growth prediction methods are usually labor-intensive and time-consuming, requiring manual field surveys and manual recording of crop types, growth conditions, and other data. This method is time-consuming and its accuracy is affected by human factors.

[0003] Existing crop identification and growth stage prediction methods lack accuracy and reliability, and cannot provide efficient and accurate decision support for agricultural management. Therefore, how to efficiently collect, process and analyze multimodal crop data has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a multimodal crop identification and growth process prediction method and device.

[0005] In a first aspect, the present invention provides a method for crop identification and growth process prediction based on multimodal data, comprising: Use drones and tower site cameras to collect growth process images of various types of crops at various growth stages from medium and high altitude perspectives and perform standardized processing, and use sensors to collect multimodal environmental data; Extract the image features of crop growth process to obtain crop feature data and construct crop feature data set; Obtain the crop types and growth stages corresponding to the crop characteristic data and construct a sample data set; Construct a deep neural network model, use crop feature data as input of the deep neural network model, and use crop species as output of the deep neural network model; The deep neural network model is trained using the sample data set to obtain a crop species recognition model; The crop type recognition model is used to obtain the crop type recognition result; Preprocess the image features of the crop growth process and the multimodal environmental data, extract features from the multimodal environmental data to obtain environmental features, and perform weighted fusion on the reduced-dimensional image features of the crop growth process and the environmental features to generate a fused feature vector; Perform data enhancement on the preprocessed data; Based on the crop species recognition model, a long short-term memory network with a global channel spatiotemporal attention mechanism is constructed. The fused feature vector after data enhancement is input into the long short-term memory network for training to obtain a growth stage prediction model. The growth stage prediction model is used to predict the growth stage of crops.

[0006] In a second aspect, the present invention provides a crop recognition and growth process prediction system based on multimodal data, comprising an image acquisition and processing unit, a crop multimodal data acquisition unit, a data set construction unit, a sample data set construction unit, a first model construction and training unit, an identification unit, a data enhancement processing unit, a second model construction and training unit, and a prediction output unit; An image acquisition and processing unit is used to use drones and tower site cameras to collect growth process images of various types of crops at various growth stages from a medium and high altitude perspective and perform standardized processing; A crop multimodal data collection unit, used to collect multimodal environmental data using sensors; A data set construction unit is used to extract the image features of the crop growth process to obtain crop feature data and construct a crop feature data set; A sample data set construction unit is used to obtain the crop types and growth stages corresponding to the crop characteristic data and construct a sample data set; The first model building and training unit is used to build a deep neural network model, use the crop feature data as the input of the deep neural network model, and use the crop species as the output of the deep neural network model; use the sample data set to train the deep neural network model to obtain a crop species recognition model; An identification unit, used for obtaining a crop species identification result by using a crop species identification model; Preprocess the image features of the crop growth process and the multimodal environmental data, extract features from the multimodal environmental data to obtain environmental features, and perform weighted fusion on the reduced-dimensional image features of the crop growth process and the environmental features to generate a fused feature vector; A data enhancement processing unit, used for performing data enhancement processing on the preprocessed data; The second model building and training unit builds a long short-term memory network with a global channel spatiotemporal attention mechanism based on the crop species recognition model, inputs the fused feature vector after data enhancement into the long short-term memory network for training, and obtains a growth stage prediction model; The prediction output unit is used to predict the growth stage of crops using the growth stage prediction model.

[0007] Based on the above technical solution, the present invention can also be improved as follows.

[0008] Furthermore, the growth process images of crops at each growth stage are annotated, and the multimodal environmental data of each type of crops at each growth stage are stored according to timestamps; the multimodal environmental data includes geographic location data, temperature and humidity data, air carbon dioxide concentration data and light intensity data.

[0009] Furthermore, the growth process images of the crops at each growth stage are standardized, including adjusting the brightness, contrast and color balance of the growth process images; the original data is converted into a format, the contrast of the growth process images is adjusted using a histogram equalization method, and the crop area in the growth process images is extracted using an image segmentation algorithm to generate a binary segmented image; the features of the crop growth process images are extracted to obtain crop feature data, including extracting the color histogram and texture features of the crop images.

[0010] Furthermore, the sample data set is divided into a training set, a test set, and a validation set; the pre-trained convolutional neural network is used to extract the image features of the crop growth process, the features are extracted through convolution operations, and the fully connected layer is used to fuse the features to generate feature vectors; the feature vectors are input into the deep neural network model for training to obtain a crop type recognition model; The deep neural network model is a YOLOv8 model, which includes a feature extraction unit, a multi-scale fusion unit, and a target detection and classification unit; The feature extraction unit uses convolutional and deconvolutional layers, and uses residual connections and bottleneck structures; The multi-scale fusion unit includes an SPPF module, a PAA module, and two PAN modules; the SPPF module is used for pooling operations at different scales to stitch together feature maps of different scales; the PAA module is used to intelligently allocate anchor boxes to optimize the selection of positive and negative samples; the two PAN modules are used for path aggregation of features at different levels; The object detection and classification unit consists of a detection head and a classification head; the detection head predicts the bounding box regression value of each anchor box and the confidence of the object existence, and the classification head uses global average pooling to classify each feature map.

[0011] Furthermore, the crop type recognition model is used to obtain crop type recognition results, including: inputting the segmented crop image area into the crop type recognition model, performing feature extraction and classification through the convolution layer and the fully connected layer, and the model outputs the category label of the crop; marking the boundary lines of different crops on the image, and combining the classification results to display the distribution of crops, and using the U-Net model for image segmentation; inputting the segmentation result into the edge detection algorithm to extract the boundary lines of crops; using the OPENCV image processing tool to draw the boundary lines and category labels on the image, generating a visualization result, and obtaining the crop area and crop type in the current crop scene image.

[0012] Furthermore, the preprocessing of image features of the crop growth process includes geometric operations, color transformation, random erasing and noise addition on the image information; the preprocessing of multimodal environmental data includes noise elimination, missing value filling and data standardization; the discrete multimodal environmental data is converted into continuous time series data, and the multimodal environmental data is divided into segments with fixed time intervals using a sliding window.

[0013] Furthermore, the image features of the crop growth process and the text features of the factors affecting the growth process are aligned, and the image features of the crop growth process and the environmental features after dimensionality reduction are concatenated and weighted fused to generate a fused feature vector.

[0014] Furthermore, the preprocessed data is enhanced by randomly selecting two samples from the sample data set and combining them in proportion to form new samples and weighted labels. The final loss value is the cross entropy loss function calculated on each label, and the weighted sum is taken to control the strength of the interpolation of the two samples. The generator and discriminator networks are constructed, and the generative adversarial network GAN is used to generate synthetic images to obtain an enhanced data set.

[0015] Furthermore, the long short-term memory network includes a forget gate, an input layer gate and an output layer gate; the forget gate is composed of a sigmoid activation function, and the image features and environmental features of the crop growth process at the previous time node are screened and removed according to the weights of each fused feature vector after data enhancement; the input layer gate screens the crop growth process images and multimodal environmental data obtained at the next time node; the output layer gate is composed of a sigmoid activation function and a tanh activation function, and the probability generated by the sigmoid activation function of the output layer gate determines the output state; the output of the input layer gate is obtained by multiplying the tanh activation function and the output state.

[0016] The beneficial effects of the present invention are as follows: the present invention recognizes crop species by constructing a deep neural network model, fuses the image features of the crop growth process with the multimodal environmental data, and uses the long short-term memory network of the global channel spatiotemporal attention mechanism to predict the crop growth stage, and inputs the fused feature vector after data enhancement into the long short-term memory network for training to capture the long-term and short-term dependencies and spatial features in the time series data. The long short-term memory network can effectively capture long-term dependencies in a long sequence through its unique gating mechanism, wherein the forgetting input gate and the output gate are respectively responsible for controlling the forgetting, updating and output of information, thereby ensuring that the network model can retain important information related to the growth stage and effectively handle the dependencies over a long time span, thereby constructing a long short-term memory network model that accurately predicts the growth stage of crops; the crop species recognition and the crop growth stage prediction results are applied to farmland management, and a reasonable management plan is formulated according to the predicted crop growth stage, and the multimodal environmental data is combined to optimize fertilization, irrigation and pest control measures to improve the yield and quality of crops. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a method for crop identification and growth process prediction based on multimodal data provided in Example 1 of the present invention; Figure 2 Schematic diagram of a crop identification and growth process prediction system based on multimodal data provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0019] Example 1 As an example, Figure 1 As shown, in order to solve the above technical problems, this embodiment provides a method for crop identification and growth process prediction based on multimodal data, including: Use drones and tower site cameras to collect growth process images of various types of crops at various growth stages from medium and high altitude perspectives and perform standardized processing, and use sensors to collect multimodal environmental data; Extract the image features of crop growth process to obtain crop feature data and construct crop feature data set; Obtain the crop types and growth stages corresponding to the crop characteristic data and construct a sample data set; Construct a deep neural network model, use crop feature data as input of the deep neural network model, and use crop species as output of the deep neural network model; The deep neural network model is trained using the sample data set to obtain a crop species recognition model; The crop type recognition model is used to obtain the crop type recognition result; Preprocess the image features of the crop growth process and the multimodal environmental data, extract features from the multimodal environmental data to obtain environmental features, and perform weighted fusion on the reduced-dimensional image features of the crop growth process and the environmental features to generate a fused feature vector; Perform data enhancement on the preprocessed data; Based on the crop species recognition model, a long short-term memory network with a global channel spatiotemporal attention mechanism is constructed. The fused feature vector after data enhancement is input into the long short-term memory network for training to obtain a growth stage prediction model. The growth stage prediction model is used to predict the growth stage of crops.

[0020] Optionally, the growth process images of each growth stage of the crops are annotated, and the multimodal environmental data of each type of crops at each growth stage are stored according to timestamps; the multimodal environmental data includes geographic location data, temperature and humidity data, air carbon dioxide concentration data and light intensity data.

[0021] Select a drone with a high-resolution camera. The drone is equipped with a GPS positioning device to obtain clear images of farmland. The drone flight path covers the entire farmland. The drone flight altitude is set to 50 meters to 300 meters. The altitude is adjusted as needed. When flying along the preset path, the camera shoots at a frequency of 2 frames per second to ensure that sufficiently dense image data is obtained. After each flight, the image is transmitted to the ground station via a wireless network for storage; the tower site camera acquisition method selects a high-resolution camera bound around the tower. Image acquisition provides clear farmland images at a frequency of 3 frames per day. Image files are classified and managed according to acquisition method, date and flight path.

[0022] The growth process of crops, such as seedlings, maturity, fruiting and harvesting, is annotated on the collected growth process images as training data samples. The types of crops include wheat, potatoes, rice and rapeseed. The prediction results are applied to farmland management. According to the predicted growth stage, a reasonable management plan is formulated. Combined with environmental data, fertilization, irrigation and pest control measures can be optimized to improve the yield and quality of crops.

[0023] By setting up temperature and humidity sensors, light sensors, carbon dioxide sensors, and soil moisture sensors, multimodal environmental data can be collected. For example, the temperature and humidity sensor uses the DHT11 temperature and humidity sensor, which is a temperature and humidity composite sensor that outputs digital signals. After the temperature and humidity sensor collects the temperature and humidity data, it is also necessary to verify the data. When the temperature and humidity are read and verified, the return value is 1, and the acquired data is correct; otherwise, the return value is 0 and the data is discarded. The sensor can directly transmit the processed digital signal to the microcontroller. The illumination module uses a photoresistor as a light intensity detection component. The photoresistor converts the light intensity into a resistance value signal, and then converts the resistance value into an electrical signal to achieve the purpose of collecting light signals. The carbon dioxide sensor uses MG-812. When the sensor is kept at a certain operating temperature and placed in a carbon dioxide atmosphere, the positive and negative electrodes of the battery react with electrodes, and an electromotive force is generated between the sensor's sensitive electrode and the reference electrode. The output signal voltage is inversely proportional to the logarithm of the carbon dioxide concentration. The change in carbon dioxide concentration can be detected by testing the change in the signal voltage. The data collection of carbon dioxide concentration needs to be realized by using the serial port of the microcontroller. The serial port sends a data reading instruction to the carbon dioxide sensor. The sensor will put the read data in the return value, and then transmit the return value to the microcontroller. The return value transmitted to the microcontroller will be processed and saved by the interrupt service program; if the received data is wrong, the microcontroller will receive the data again; if the received data is correct, the concentration of carbon dioxide is calculated. The soil moisture sensor uses the YL-69 sensor to collect soil moisture information in the greenhouse. The temperature and humidity sensor, light sensor, carbon dioxide sensor, and soil moisture sensor are connected to the data processing unit through the bus and uploaded to the data server.

[0024] Optionally, the growth process images of the crops at each growth stage are standardized, including adjusting the brightness, contrast and color balance of the growth process images; converting the format of the original data, adjusting the contrast of the growth process images using a histogram equalization method, extracting the crop area in the growth process images using an image segmentation algorithm, and generating a binary segmented image; extracting features of the crop growth process images to obtain crop feature data, including extracting the color histogram and texture features of the crop images.

[0025] Optionally, the sample data set is divided into a training set, a test set and a validation set; a pre-trained convolutional neural network is used to extract image features of the crop growth process, features are extracted through convolution operations, and feature fusion is performed through a fully connected layer to generate feature vectors; the feature vectors are input into a deep neural network model for training to obtain a crop species recognition model; The deep neural network model is a YOLOv8 model, which includes a feature extraction unit, a multi-scale fusion unit, and a target detection and classification unit; The feature extraction unit uses convolution and deconvolution layers, and uses residual connections and bottleneck structures; The multi-scale fusion unit includes the SPPF module (Spatial Pyramid Pooling Fast), the PAA module (Probabilistic Anchor Assignment) and two PAN modules (Path Aggregation Network). The SPPF module is used for pooling operations at different scales to stitch together feature maps of different scales. The PAA module is used to intelligently allocate anchor boxes to optimize the selection of positive and negative samples. The two PAN modules are used for path aggregation of features at different levels. The object detection and classification unit consists of a detection head and a classification head; the detection head predicts the bounding box regression value of each anchor box and the confidence of the object existence, and the classification head uses global average pooling to classify each feature map.

[0026] In actual application, the sample data set is divided into training set, test set and validation set according to 70%, 20% and 10% respectively. Optionally, the image features are first extracted using the pre-trained convolutional neural network ResNet-50. ResNet-50 has a depth of 50 layers and can capture subtle features in the image. Features are extracted through convolution operations, and then feature fusion is performed through the fully connected layer to generate feature vectors. The feature vectors are input into the deep neural network model for training to obtain the crop species recognition model. The crop species recognition results are compared with the actual collected data to verify the accuracy of the model.

[0027] Optionally, a crop type recognition model is used to obtain crop type recognition results, including: inputting the segmented crop image area into the crop type recognition model, performing feature extraction and classification through convolutional layers and fully connected layers, and the model outputs crop category labels; marking the boundary lines of different crops on the image, and displaying the distribution of crops in combination with the classification results, using a U-Net model for image segmentation; inputting the segmentation results into an edge detection algorithm to extract the boundary lines of crops; using OPENCV image processing tools to draw boundary lines and category labels on the image, generating visualization results, and obtaining crop areas and crop types in the current crop scene image.

[0028] Optionally, preprocessing of image features of the crop growth process includes geometric operations, color transformation, random erasing and noise addition on image information; preprocessing of multimodal environmental data includes noise elimination, missing value filling and data standardization; converting discrete multimodal environmental data into continuous time series data, and using a sliding window to divide the multimodal environmental data into segments with fixed time intervals.

[0029] Preprocessing of crop growth image features, including: geometric correction of drone images to correct geometric distortion caused by shooting angle and altitude changes; using geographic information system to georeference images to correspond to actual geographic locations; spectral correction of multispectral and thermal infrared images to eliminate spectral deviations between different bands; using standard targets with known reflectivity (such as whiteboards or blackboards) for correction to ensure comparability between different image bands. Regularly calibrate sensors to ensure measurement accuracy; use calibration boxes to calibrate temperature, humidity, light intensity, and carbon dioxide sensors.

[0030] Preprocess the multimodal environmental data, including: preliminary screening of the collected multimodal environmental data to remove blurred, out-of-focus or environmentally disturbed images (such as haze, strong light reflection). Use image denoising techniques such as median filtering and Gaussian filtering to remove noise in the image. Median filtering can effectively remove salt and pepper noise in crop images, while Gaussian filtering is used to smooth Gaussian noise in images. The 3σ rule in statistical methods is used to detect and remove outliers in environmental data that may be caused by sensor failure or environmental mutations. For missing data, linear interpolation and spline interpolation are used to fill in to ensure data continuity and integrity. Perform geometric correction on images taken by drones to correct geometric distortion caused by changes in shooting angle and altitude.

[0031] Optionally, the image features of the crop growth process and the text features of the factors affecting the growth process are aligned, and the image features of the crop growth process and the environmental features after dimensionality reduction are concatenated and weightedly fused to generate a fused feature vector.

[0032] Optionally, perform data augmentation on the preprocessed data. Two samples are randomly selected from the sample data set, and the two samples are combined in proportion to form new samples and weighted labels. The final loss value is the cross entropy loss function calculated on each label, and the weighted sum is taken to control the strength of the interpolation of the two samples. The generator and discriminator networks are constructed, and the generative adversarial network GAN is used to generate synthetic images to obtain an enhanced data set.

[0033] By processing the collected image data, the robustness and generalization ability of the system are improved. The image is rotated at random angles; the image is scaled at random ratios to simulate the change in the flight altitude of the drone, using a scaling ratio between 0.8 and 1.2; the image is translated in random directions to simulate slight changes in the position of the drone, using a translation range of less than 10% of the image size; the image brightness is adjusted to simulate images under different lighting conditions, and the brightness adjustment range is set to -30% to 30%; the image contrast is adjusted to simulate images under different weather conditions; the image hue is adjusted to simulate images under different ambient light conditions, and the hue adjustment range is set to -30% to 30%. Gaussian noise is added to the image to simulate sensor noise and environmental interference, and the standard deviation of Gaussian noise is set between 0 and 25; random white and black pixels are added to the image to simulate interference during image transmission, and the density of salt and pepper noise is set to between 0.01 and 0.05. Linear interpolation is used to linearly interpolate the missing environmental data to generate continuous time series data. Gaussian noise is added to the environmental data to simulate sensor measurement errors and environmental interference. Random noise is added to environmental data to simulate unforeseen environmental changes; exponential smoothing methods are used to smooth data; different types of data (such as crop growth process image data and multimodal environmental data) are fused and processed through multimodal data fusion enhancement to generate new data features, and the crop growth process image features extracted by convolutional neural networks are spliced ​​with environmental features such as temperature and humidity. Combined with time and space information, data at different times and locations are fused. The principal component analysis method is used to reduce the dimensionality of high-dimensional features, retain the main information, and reduce data redundancy. The principal component analysis method can improve data processing efficiency and reduce computational complexity. The multilayer perceptron is used to perform nonlinear transformation on the fused features to generate richer feature representations. The multilayer perceptron can capture the complex relationship between features and improve the accuracy of model prediction.

[0034] Optionally, the long short-term memory network includes a forget gate, an input layer gate and an output layer gate; the forget gate is composed of a sigmoid activation function, and the image features and environmental features of the crop growth process at the previous time node are screened and removed according to the weights of each fused feature vector after data enhancement; the input layer gate screens the crop growth process images and multimodal environmental data obtained at the next time node; the output layer gate is composed of a sigmoid activation function and a tanh activation function, and the probability generated by the sigmoid activation function of the output layer gate determines the output state; the output of the input layer gate is obtained by multiplying the tanh activation function and the output state.

[0035] The crop growth process involves multiple complex and dynamic factors, and the interactions between these factors are often temporal and spatially heterogeneous. The multimodal data of crop growth are processed through the global channel attention mechanism, which can adaptively allocate attention weights and highlight the most critical time points and spatial regions for crop growth stage prediction, so that the deep learning neural network model can accurately capture growth patterns and abnormal changes when processing data with spatiotemporal complexity. Based on the crop variety recognition model, a long short-term memory network with a global channel spatiotemporal attention mechanism is constructed, and the fused feature vector after data enhancement is input into the long short-term memory network for training to capture the long-term and short-term dependencies and spatial features in the time series data. The long short-term memory network can effectively capture long-term dependencies in long sequences through its unique gating mechanism, in which the forgetting input gate and the output gate are responsible for controlling the forgetting, updating and output of information, respectively, so as to ensure that the network model can retain important information related to the growth stage and effectively process long-time span dependencies, thus constructing a long short-term memory network model that accurately predicts the growth stage of crops.

[0036] Example 2 Based on the same principle as the method shown in Example 1 of the present invention, as shown in the attached Figure 2 As shown, an embodiment of the present invention also provides a crop recognition and growth process prediction system based on multimodal data, including an image acquisition and processing unit, a crop multimodal data acquisition unit, a data set construction unit, a sample data set construction unit, a first model construction and training unit, a recognition unit, a data enhancement processing unit, a second model construction and training unit, and a prediction output unit; An image acquisition and processing unit is used to use drones and tower site cameras to collect growth process images of various types of crops at various growth stages from a medium and high altitude perspective and perform standardized processing; A crop multimodal data collection unit, used to collect multimodal environmental data using sensors; A data set construction unit is used to extract the image features of the crop growth process to obtain crop feature data and construct a crop feature data set; A sample data set construction unit is used to obtain the crop types and growth stages corresponding to the crop characteristic data and construct a sample data set; The first model building and training unit is used to build a deep neural network model, use the crop feature data as the input of the deep neural network model, and use the crop species as the output of the deep neural network model; use the sample data set to train the deep neural network model to obtain a crop species recognition model; An identification unit, used for obtaining a crop species identification result by using a crop species identification model; Preprocess the image features of the crop growth process and the multimodal environmental data, extract features from the multimodal environmental data to obtain environmental features, and perform weighted fusion on the reduced-dimensional image features of the crop growth process and the environmental features to generate a fused feature vector; A data enhancement processing unit, used for performing data enhancement processing on the preprocessed data; The second model building and training unit builds a long short-term memory network with a global channel spatiotemporal attention mechanism based on the crop species recognition model, inputs the fused feature vector after data enhancement into the long short-term memory network for training, and obtains a growth stage prediction model; The prediction output unit is used to predict the growth stage of crops using the growth stage prediction model.

[0037] Optionally, the growth process images of each growth stage of the crops are annotated, and the multimodal environmental data of each type of crops at each growth stage are stored according to timestamps; the multimodal environmental data includes geographic location data, temperature and humidity data, air carbon dioxide concentration data and light intensity data.

[0038] Optionally, the growth process images of the crops at each growth stage are standardized, including adjusting the brightness, contrast and color balance of the growth process images; converting the format of the original data, adjusting the contrast of the growth process images using a histogram equalization method, extracting the crop area in the growth process images using an image segmentation algorithm, and generating a binary segmented image; extracting features of the crop growth process images to obtain crop feature data, including extracting the color histogram and texture features of the crop images.

[0039] Optionally, the sample data set is divided into a training set, a test set and a validation set; a pre-trained convolutional neural network is used to extract image features of the crop growth process, features are extracted through convolution operations, and feature fusion is performed through a fully connected layer to generate feature vectors; the feature vectors are input into a deep neural network model for training to obtain a crop species recognition model; The deep neural network model is a YOLOv8 model, which includes a feature extraction unit, a multi-scale fusion unit, and a target detection and classification unit; The feature extraction unit uses convolutional and deconvolutional layers, and uses residual connections and bottleneck structures; The multi-scale fusion unit includes an SPPF module, a PAA module, and two PAN modules; the SPPF module is used for pooling operations at different scales to stitch together feature maps of different scales; the PAA module is used to intelligently allocate anchor boxes to optimize the selection of positive and negative samples; the two PAN modules are used for path aggregation of features at different levels; The object detection and classification unit consists of a detection head and a classification head; the detection head predicts the bounding box regression value of each anchor box and the confidence of the object existence, and the classification head uses global average pooling to classify each feature map.

[0040] Optionally, a crop type recognition model is used to obtain crop type recognition results, including: inputting the segmented crop image area into the crop type recognition model, performing feature extraction and classification through convolutional layers and fully connected layers, and the model outputs crop category labels; marking the boundary lines of different crops on the image, and displaying the distribution of crops in combination with the classification results, using a U-Net model for image segmentation; inputting the segmentation results into an edge detection algorithm to extract the boundary lines of crops; using OPENCV image processing tools to draw boundary lines and category labels on the image, generating visualization results, and obtaining crop areas and crop types in the current crop scene image.

[0041] Optionally, preprocessing of image features of the crop growth process includes geometric operations, color transformation, random erasing and noise addition on image information; preprocessing of multimodal environmental data includes noise elimination, missing value filling and data standardization; converting discrete multimodal environmental data into continuous time series data, and using a sliding window to divide the multimodal environmental data into segments with fixed time intervals.

[0042] Optionally, the image features of the crop growth process and the text features of the factors affecting the growth process are aligned, and the image features of the crop growth process and the environmental features after dimensionality reduction are concatenated and weightedly fused to generate a fused feature vector.

[0043] Optionally, perform data augmentation on the preprocessed data. Two samples are randomly selected from the sample data set, and the two samples are combined in proportion to form new samples and weighted labels. The final loss value is the cross entropy loss function calculated on each label, and the weighted sum is taken to control the strength of the interpolation of the two samples. The generator and discriminator networks are constructed, and the generative adversarial network GAN is used to generate synthetic images to obtain an enhanced data set.

[0044] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for crop identification and growth process prediction based on multimodal data, characterized in that: include: Use drones and tower site cameras to collect growth process images of various types of crops at various growth stages from medium and high altitude perspectives and perform standardized processing, and use sensors to collect multimodal environmental data; Extract the image features of crop growth process to obtain crop feature data and construct crop feature data set; Obtain the crop types and growth stages corresponding to the crop characteristic data and construct a sample data set; Construct a deep neural network model, use crop feature data as input of the deep neural network model, and use crop species as output of the deep neural network model; The deep neural network model is trained using the sample data set to obtain a crop species recognition model; The crop type recognition model is used to obtain the crop type recognition result; Preprocess the image features of the crop growth process and the multimodal environmental data, extract features from the multimodal environmental data to obtain environmental features, and perform weighted fusion on the reduced-dimensional image features of the crop growth process and the environmental features to generate a fused feature vector; Perform data enhancement on the preprocessed data; Based on the crop species recognition model, a long short-term memory network with a global channel spatiotemporal attention mechanism is constructed. The fused feature vector after data enhancement is input into the long short-term memory network for training to obtain a growth stage prediction model. The growth stage prediction model is used to predict the growth stage of crops.

2. The method for crop identification and growth process prediction based on multimodal data according to claim 1, characterized in that: The growth process images of crops at each growth stage are annotated, and the multimodal environmental data of each type of crop at each growth stage are stored according to timestamps; the multimodal environmental data includes geographic location data, temperature and humidity data, air carbon dioxide concentration data and light intensity data.

3. The method for crop identification and growth process prediction based on multimodal data according to claim 1, characterized in that: The growth process images of crops at each growth stage are standardized, including adjusting the brightness, contrast and color balance of the growth process images; the original data is converted into a format, the contrast of the growth process images is adjusted using a histogram equalization method, and the crop area in the growth process images is extracted using an image segmentation algorithm to generate a binary segmented image; the features of the crop growth process images are extracted to obtain crop feature data, including extracting the color histogram and texture features of the crop images.

4. The method for crop identification and growth process prediction based on multimodal data according to claim 1, characterized in that: The sample data set is divided into a training set, a test set, and a validation set. The pre-trained convolutional neural network is used to extract the image features of the crop growth process. The features are extracted through convolution operations, and the fully connected layer is used to fuse the features to generate feature vectors. The feature vector is input into the deep neural network model for training to obtain a crop species recognition model; The deep neural network model is a YOLOv8 model, which includes a feature extraction unit, a multi-scale fusion unit, and a target detection and classification unit; The feature extraction unit uses convolution and deconvolution layers, and uses residual connections and bottleneck structures; The multi-scale fusion unit includes an SPPF module, a PAA module, and two PAN modules; the SPPF module is used for pooling operations at different scales to stitch together feature maps of different scales; the PAA module is used to intelligently allocate anchor boxes to optimize the selection of positive and negative samples; the two PAN modules are used for path aggregation of features at different levels; The object detection and classification unit consists of a detection head and a classification head; the detection head predicts the bounding box regression value of each anchor box and the confidence of the object existence, and the classification head uses global average pooling to classify each feature map.

5. The method for crop identification and growth process prediction based on multimodal data according to claim 1, characterized in that: The crop type recognition model is used to obtain the crop type recognition result, including: inputting the segmented crop image area into the crop type recognition model, performing feature extraction and classification through the convolution layer and the fully connected layer, and the model outputs the category label of the crop; marking the boundary lines of different crops on the image, and combining the classification results to display the distribution of crops, and using the U-Net model for image segmentation; inputting the segmentation result into the edge detection algorithm to extract the boundary lines of crops; using the OPENCV image processing tool to draw the boundary lines and category labels on the image, generating a visualization result, and obtaining the crop area and crop type in the current crop scene image.

6. The method for crop identification and growth process prediction based on multimodal data according to claim 1, characterized in that: Preprocessing of image features of crop growth process includes geometric operations, color transformation, random erasure and noise addition on image information; preprocessing of multimodal environmental data includes noise elimination, missing value filling and data standardization; discrete multimodal environmental data are converted into continuous time series data, and sliding windows are used to divide multimodal environmental data into segments with fixed time intervals.

7. The method for crop identification and growth process prediction based on multimodal data according to claim 1, characterized in that: The image features of the crop growth process and the text features of the factors affecting the growth process are aligned, and the image features of the crop growth process and the environmental features after dimensionality reduction are spliced ​​and weighted fused to generate a fused feature vector.

8. The method for crop identification and growth process prediction based on multimodal data according to claim 1, characterized in that: The preprocessed data is enhanced by randomly selecting two samples from the sample data set and combining them in proportion to form new samples and weighted labels. The final loss value is the cross entropy loss function calculated on each label, and the weighted sum is taken to control the strength of the interpolation of the two samples. The generator and discriminator networks are constructed, and the generative adversarial network GAN is used to generate synthetic images to obtain an enhanced data set.

9. The method for crop identification and growth process prediction based on multimodal data according to claim 1, characterized in that: The long short-term memory network includes a forget gate, an input layer gate and an output layer gate; the forget gate is composed of a sigmoid activation function, which filters and removes the image features and environmental features of the crop growth process at the previous time node according to the weights of each fused feature vector after data enhancement; the input layer gate filters the crop growth process images and multimodal environmental data obtained at the next time node; the output layer gate is composed of a sigmoid activation function and a tanh activation function, and the probability generated by the sigmoid activation function of the output layer gate determines the output state; the output of the input layer gate is obtained by multiplying the tanh activation function and the output state.

10. A crop identification and growth process prediction system based on multimodal data, characterized in that: It includes an image acquisition and processing unit, a crop multimodal data acquisition unit, a data set construction unit, a sample data set construction unit, a first model construction and training unit, a recognition unit, a data enhancement processing unit, a second model construction and training unit, and a prediction output unit; An image acquisition and processing unit is used to use drones and tower site cameras to collect growth process images of various types of crops at various growth stages from a medium and high altitude perspective and perform standardized processing; A crop multimodal data collection unit, used to collect multimodal environmental data using sensors; A data set construction unit is used to extract the image features of the crop growth process to obtain crop feature data and construct a crop feature data set; A sample data set construction unit is used to obtain the crop types and growth stages corresponding to the crop characteristic data and construct a sample data set; The first model building and training unit is used to build a deep neural network model, use the crop feature data as the input of the deep neural network model, and use the crop species as the output of the deep neural network model; use the sample data set to train the deep neural network model to obtain a crop species recognition model; An identification unit, used for obtaining a crop species identification result by using a crop species identification model; Preprocess the image features of the crop growth process and the multimodal environmental data, extract features from the multimodal environmental data to obtain environmental features, and perform weighted fusion on the reduced-dimensional image features of the crop growth process and the environmental features to generate a fused feature vector; A data enhancement processing unit, used for performing data enhancement processing on the preprocessed data; The second model building and training unit builds a long short-term memory network with a global channel spatiotemporal attention mechanism based on the crop species recognition model, inputs the fused feature vector after data enhancement into the long short-term memory network for training, and obtains a growth stage prediction model; The prediction output unit is used to predict the growth stage of crops using the growth stage prediction model.

Images