Plant growth key element multi-modal database creation method, growth state prediction method and monitoring system

Abstract

The application discloses a plant growth key element multi-modal database creation method, a growth state prediction method and a monitoring system, and relates to the field of agricultural information technology. The method comprises collecting multi-modal data in the plant growth process and performing pretreatment; a deep learning model is used to extract the features of the pretreated data, and then all the data are time-aligned; a database table structure is determined according to the multi-modal data collection requirements; the database table structure comprises a feature storage field, an index field and a hyperlink identifier, and the hyperlink identifier is used to establish a hyperlink relationship between each piece of data and the modal data other than the modal to which each piece of data belongs; according to the database table structure, a storage container is established for each kind of modal data to store the data, and a plant growth key element multi-modal database is obtained. The application can effectively extract and analyze the multi-dimensional features of the plant growth key elements, fuse the multi-modal data, and realize efficient storage and management of large-scale plant growth data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of agricultural information technology, in particular to a plant growth key element multi-modal database creation method, a growth state prediction method and a monitoring system. BACKGROUND

[0002] With the rapid development of modern agriculture towards intelligence and precision, plant growth dynamic monitoring has gradually become an important link to realize intelligent management of agricultural production. However, traditional plant growth monitoring methods are mainly based on single data source, which is difficult to integrate multi-modal data (such as text, image and video) for comprehensive analysis, and there are problems such as data island, limited analysis dimension, weak dynamic correlation, etc.

[0003] At the same time, the rapid growth of agricultural big data puts forward higher requirements for data storage, processing and analysis. The existing technology cannot fully utilize modern deep learning methods, and cannot effectively extract and analyze the multi-dimensional features of plant growth key elements, especially in time series dynamic monitoring and cross-modal correlation analysis. Therefore, a multi-modal database creation method combining deep learning technology is needed to solve the bottleneck problem of traditional agricultural data management, and to provide technical support for intelligent agricultural management and precise decision-making. SUMMARY

[0004] The purpose of the present application is to provide a plant growth key element multi-modal database creation method, a growth state prediction method and a monitoring system, which can effectively extract and analyze the multi-dimensional features of plant growth key elements, integrate multi-modal data, and realize efficient storage and management of large-scale plant growth data.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a plant growth key element multi-modal database creation method, comprising:

[0007] According to the target database creation instruction related to plant growth, determine the multi-modal data acquisition requirement related to plant growth;

[0008] According to the multi-modal data acquisition requirement, collect multi-modal data in the plant growth process; the multi-modal data includes text record information, image, video and sensing data;

[0009] Preprocess the multi-modal data;

[0010] Extract the preprocessed text record information, the preprocessed image and the preprocessed video by using a deep learning model respectively, to obtain text features, image features and video features;

[0011] The text feature vector, the image feature vector, the video feature vector and the sensor feature vector are obtained by time alignment on the text feature, the image feature, the video feature and the preprocessed sensor data.

[0012] According to the multi-modal data acquisition requirement, a database table structure is determined; the database table structure includes a feature storage field, an index field and a hyperlink identifier; the hyperlink identifier is used to establish a hyperlink relationship between each piece of data and the multi-modal data other than the modal to which each piece of data belongs;

[0013] According to the database table structure, a storage container is established for each kind of modal data;

[0014] The text feature vector, the image feature vector, the video feature vector and the sensor feature vector are respectively stored into the respective storage containers, and a plant growth key element multi-modal database is obtained.

[0015] In a second aspect, the application provides a plant growth state prediction method, which applies the plant growth key element multi-modal database creation method described above, and the plant growth state prediction method comprises the following steps:

[0016] Multi-modal feature vectors at multiple time points in the plant growth process are obtained from the plant growth key element multi-modal database; the multi-modal feature vectors include text feature vectors, image feature vectors, video feature vectors and sensor feature vectors;

[0017] After splicing or weighted summation of the multi-modal feature vectors at the multiple time points, feature fusion is performed by using a cross-modal attention mechanism to obtain fused features;

[0018] The fused features are input into an LSTM time sequence prediction model, and a plant growth state at a prediction time point is output.

[0019] In a third aspect, the application provides a plant growth dynamic monitoring system, which comprises a retrieval unit, a visualization interface, a plant growth key element multi-modal database and a prediction unit.

[0020] The retrieval unit is used to retrieve single-modal data or cross-modal data from the plant growth key element multi-modal database.

[0021] The prediction unit predicts the plant growth state by using the plant growth state prediction method described above.

[0022] The visualization interface is used to display the retrieved single-modal data or cross-modal data, and to draw a predicted trend curve of the plant growth according to the predicted plant growth state.

[0023] According to the specific embodiments provided in the application, the following technical effects are achieved:

[0024] The application provides a plant growth key element multi-modal database creation method, a growth state prediction method and a monitoring system. A deep learning model can extract text features, image features and video features, effectively extracts and analyzes multi-dimensional features of plant growth key elements, aligns the text features, the image features, the video features and the preprocessed sensing data in time, ensures the accuracy of the logical correlation between the data, establishes the correlation between the modal data through the hyperlink identifier in the database table structure, realizes the fusion of multi-modal data, and can efficiently store and manage the multi-modal data of large-scale plant growth. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1 A flowchart of a plant growth key element multi-modal database creation method provided by an embodiment of the present application is shown in the figure.

[0027] Figure 2 A brief flowchart of a plant growth key element multi-modal database creation method provided by an embodiment of the present application is shown in the figure.

[0028] Figure 3 A flowchart of a plant growth state prediction method provided by an embodiment of the present application is shown in the figure.

[0029] Figure 4 A flowchart of the overall technical solution of the present application is shown in the figure. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0032] In an exemplary embodiment, as Figure 1As shown, a plant growth key element multi-modal database creation method is provided, comprising the following steps 101 to 108. Among them:

[0033] Step 101: According to the target database creation instruction related to plant growth, determine the multi-modal data acquisition requirement related to plant growth.

[0034] Step 102: According to the multi-modal data acquisition requirement, collect multi-modal data in the process of plant growth; the multi-modal data includes text record information, image, video and sensor data.

[0035] Step 103: Preprocess the multi-modal data.

[0036] Step 104: Extract the preprocessed text record information, preprocessed image and preprocessed video respectively by using deep learning model, and obtain text feature, image feature and video feature.

[0037] Step 105: Time align the text feature, image feature, video feature and preprocessed sensor data, and obtain text feature vector, image feature vector, video feature vector and sensor feature vector.

[0038] Step 106: According to the multi-modal data acquisition requirement, determine the database table structure; the database table structure includes feature storage field, index field and hyperlink identifier; the hyperlink identifier is used to establish the hyperlink relationship between each data and the modal data other than the modal to which each data belongs.

[0039] Step 107: According to the database table structure, establish a storage container for each kind of modal data.

[0040] Step 108: Store the text feature vector, image feature vector, video feature vector and sensor feature vector into the respective storage container, and obtain the plant growth key element multi-modal database.

[0041] Implementing the above steps 101 to 108 solves the problems of single data source, difficult integration and complex correlation analysis in traditional plant growth monitoring.

[0042] In another exemplary embodiment of the present application, the multi-modal data acquisition requirement includes six elements and ten indexes. The six elements include soil, temperature, light, moisture, air and fertilizer. The ten indexes include air temperature, air humidity, air pressure, light, noise, carbon dioxide, soil temperature, soil humidity, soil EC value (representing soluble salt content) and soil PH value.

[0043] In another exemplary embodiment of the present application, a comprehensive multi-modal data acquisition scheme is developed based on the multi-modal data acquisition requirements, ensuring coverage of key growth stages from planting to maturity, while meeting the technical requirements of analysis and modeling. The data collected mainly includes the following three forms:

[0044] (1) Text information acquisition

[0045] Record the plant species, planting time, fertilization record, growth environment parameters (such as temperature, humidity, light conditions, etc.), pest information, etc.

[0046] To improve the standardization of data, a unified data field is designed, such as: temperature range (unit: °C); humidity range (unit: %); light intensity (unit: lux); fertilizer amount (unit: g).

[0047] When recording structured fields, JSON format can be used to facilitate subsequent data storage and retrieval.

[0048] (2) Monitoring photo acquisition

[0049] Use multi-spectral cameras and high-resolution industrial cameras to capture high-definition images of plant leaves and overall morphology.

[0050] Acquisition content:

[0051] Visible light image: reflects the overall growth condition and color distribution of the plant.

[0052] Infrared and near-infrared images: help analyze the water content of leaves, photosynthesis level, and other hidden characteristics.

[0053] Collect at regular intervals every day (e.g. 8:00, 14:00 and 20:00) to ensure comprehensive coverage of data at different times.

[0054] (3) Video data acquisition

[0055] Use high-frame-rate cameras to record the dynamic growth process of plants. Capture the dynamic changes of plants, such as leaf unfolding speed, bending angle, and time series changes of plant height. Timely sampling (such as collecting two videos per day, recording the plant state at sunrise and sunset). The length of each video is recommended to be 30 seconds to 1 minute, and the frame rate is set to 30 fps.

[0056] (4) Determination of acquisition methods for six elements and ten indicators

[0057] For the six key elements and ten growth indicators of plant growth, the acquisition equipment, acquisition method and standardized operation process are clearly defined. The following are some specific contents:

[0058] Soil moisture is measured using a TDR (soil moisture sensor), which accurately records changes in soil water content. Data is collected hourly to ensure continuity and completeness.

[0059] Soil pH levels are measured periodically using a pH monitor to determine the acidity or alkalinity of the soil. The monitoring equipment can record data automatically at preset time intervals, or it can be measured manually through sampling.

[0060] An integrated temperature and humidity recorder is used to monitor and record changes in ambient temperature in real time, generating detailed time-series data to help analyze the impact of temperature on plant growth.

[0061] Light intensity variations on the plant surface were recorded using a photoluminescence meter. Data was collected every minute, reflecting the changes in light intensity received by the plants at different times of the day.

[0062] Soil moisture dynamics are recorded using moisture sensors (such as capacitive moisture probes), and data is collected periodically to support the analysis of the impact of water on plant growth.

[0063] Air quality data is obtained through carbon dioxide concentration meters and PM2.5 / PM10 detectors, mainly monitoring the concentration of pollutants and CO2 levels in the air, providing data support for research on plant photosynthesis and gas exchange.

[0064] Therefore, step 102 above can be replaced by the following steps 201 to 205:

[0065] Step 201: Record the plant species, planting time, fertilization records, growth environment parameters, and pest and disease information during the plant growth process, and record this information in written form.

[0066] Step 202: Collect visible light and infrared images of the plant at regular intervals.

[0067] Step 203: Use a camera to record video of the plant's dynamic growth process.

[0068] Step 204: Determine the sensing and acquisition equipment based on the multimodal data acquisition requirements.

[0069] Step 205: Collect sensor data during the plant growth process using sensor acquisition equipment.

[0070] In another exemplary embodiment of this application, the collected raw data often contains noise, outliers, and inconsistencies. Therefore, data normalization and preprocessing are required to ensure high-quality input for model training. Preprocessing of the collected data specifically includes: image data processing, video data processing, text information processing, and multimodal time synchronization.

[0071] (1) Text information preprocessing

[0072] Data cleaning: Check and handle missing values, outliers (e.g. incorrect fertilizer amount). For missing values, you can use mean filling or interpolation method.

[0073] Qualitative data encoding: Encode non-numeric fields. For example, for fertilizer type:

[0074]

[0075] (2) Image data preprocessing

[0076] Adjust image brightness using gamma correction algorithm to ensure image color consistent with actual scene:

[0077]

[0078] where I in (x,y) is the input image pixel value, γ is the correction parameter, commonly γ = 2.2.

[0079] Adjust all images to a uniform resolution (e.g. 224x224) to adapt to the input requirements of deep learning models:

[0080] I resized (x,y) = I input Resize(224,224).

[0081] The gray value of the image is usually affected by the acquisition conditions (such as light changes), so normalizing the image helps to eliminate this influence and improve the stability of subsequent processing. Normalization processing enhances the contrast of the image, making plant features (such as leaves, stems, etc.) more prominent, thereby improving the training effect of the deep learning model. Standardize the image gray value to enhance contrast:

[0082]

[0083] Extract the region of interest (Region of Interest, ROI) according to the target area, reduce background noise interference. ROI extraction can be achieved through image segmentation algorithm. Specifically, use the deep learning U-Net model to extract the plant region of interest (such as leaves or stems), and the U-Net model can effectively separate the main structure of the plant from the complex background, such as leaves, stems, flowers, etc. U-Net model gradually extracts the features of the image through downsampling and upsampling process, finally generates accurate segmentation results:

[0084] Output = U-Net(I).

[0085] The segmented plant region can be used for subsequent morphological analysis, such as calculating leaf area, leaf number, leaf shape, and other features.

[0086] (3) Video data preprocessing

[0087] Video data contains continuous frame images, usually with a large amount of redundant frames. In order to reduce the computational burden and retain key information, inter-frame difference method is used to extract key frames of the video. By extracting key frames through inter-frame difference method, redundant data is reduced:

[0088] ΔF = ||I t+1 -I t ||.

[0089] When ΔF > threshold, the current frame is saved and extracted as a key frame.

[0090] The extracted key frames are arranged in chronological order to form time series image data, providing a basis for dynamic change analysis.

[0091] Time series is generated by sliding window method:

[0092] S = {I t ,I t+1 ,...,I t+k}.

[0093] Optical flow method is used to estimate the motion vector of each pixel point in the video. Optical flow method analyzes the changes of pixel points between consecutive frames to calculate the motion direction and speed of the pixels. Through these dynamic characteristics, the motion trajectory, speed, direction and other characteristics of plant leaves can be further analyzed. Optical flow method is used to analyze the motion direction and speed of leaves:

[0094]

[0095] Where u, v represent the motion vector of the pixel point.

[0096] (4) Sensor data preprocessing

[0097] The common method is smoothing filter, using algorithms such as moving average filter, etc. Through sliding window processing of data, high frequency noise is reduced. The time series data collected by the sensor is smoothed.

[0098] Moving average filter formula:

[0099] Time alignment: synchronize and integrate multi-source sensor data according to timestamp.

[0100] Therefore, the above step 103 can be replaced by the following steps 301-305:

[0101] Step 301: data cleaning is performed on the text record information, and non-numeric fields are encoded to obtain preprocessed text record information.

[0102] Step 302: the image brightness is adjusted using a gamma correction algorithm, and the resolution is unified to obtain a preliminary adjusted image.

[0103] Step 303: the preliminary adjusted image is normalized, and a U-Net model is used to extract a plant region of interest from the normalized image to obtain a plant region of interest image as a preprocessed image.

[0104] Step 304: the key frames of the video are extracted using an inter-frame difference method, the extracted key frames are arranged in chronological order, and a key frame time sequence is generated by a sliding window method as a preprocessed video.

[0105] Step 305: each type of sensing data is smoothed, and the smoothed multi-source sensing data is synchronized and aligned according to the time stamp to obtain preprocessed sensing data.

[0106] In another exemplary embodiment of the present application, feature extraction and classification of multi-modal data are performed by a deep learning model. Convolutional neural networks (CNN) are used to extract image and video feature vectors, and natural language processing models are used to encode the semantic information of text data.

[0107] The above step 104 can be replaced by the following steps 401-406:

[0108] Step 401: the preprocessed text record information is processed by word segmentation, and stop words are deleted to obtain text words.

[0109] Step 402: the text words are converted into vector form using a pre-trained word vector model to obtain text word vectors.

[0110] Step 403: text features are extracted using a natural language processing model based on the text word vectors.

[0111] Step 404: image features are extracted using a pre-trained convolutional neural network from the preprocessed image to obtain image features.

[0112] Step 405: static image features of the preprocessed video are extracted using a convolutional neural network.

[0113] Step 406: spatiotemporal dynamic features of the preprocessed video are extracted using a time series model, and together with the static image features, form video features.

[0114] The more detailed implementation processes of steps 401 to 406 are as follows:

[0115] (1) Text information processing

[0116] Word segmentation processing:

[0117] Use the Jieba word segmentation tool to perform word segmentation on Chinese text. Jieba uses a dictionary-based word segmentation method and a hidden Markov model to accurately segment Chinese sentences into words, providing a basis for subsequent text analysis.

[0118] For English text, use the word_tokenize function of the NLTK library to perform word segmentation, separating each word to ensure that the structure of the text can be effectively understood by deep learning models.

[0119] Chinese word segmentation example: Input text: "Plants grow under sunlight", the output after Jieba word segmentation is: "Plants", "grow", "under", "sunlight".

[0120] English word segmentation example: Input text: "Plants grow under sunlight", the output after NLTK word segmentation is: ['Plants', 'grow', 'under','sunlight'].

[0121] Stop word removal and word vector representation:

[0122] Stop words are words that appear frequently in text but are not actually helpful for semantic analysis (such as "of", "is", "in"). In the text preprocessing process, use a stop word list to remove these irrelevant words, thereby reducing data noise and improving the accuracy of feature extraction.

[0123] By deleting stop words from the text, a cleaner set of words is obtained for use by subsequent deep learning models. After removing stop words, the words in the text are then converted into vector form. This process is usually performed using a pre-trained word vector model (such as Word2Vec or GloVe).

[0124] Text feature extraction and deep learning application:

[0125] Text feature extraction based on a convolutional neural network, and its characteristics of local connection and weight sharing also make it suitable for text feature extraction. By applying a convolutional layer, local semantic features (such as phrases, keywords, etc.) are extracted from the word vectors.

[0126] Convolutional layer operation: Assume the word vector of the input text is X = [x1, x2,..., x TConvolution operation is performed by convolution kernel w, and convolution output y is obtained j :

[0127]

[0128] where w i is the convolution kernel, b is the bias, k is the convolution window size, ReLU is the activation function, and y j is the output after convolution operation.

[0129] A pre-trained BERT (Bidirectional Encoder Representations from Transformers) model is used to further understand the deep semantics of the text. BERT uses a bidirectional Transformer structure, which can capture the polysemy and complex syntax of words in the context.

[0130] BERT model formula: BERT learns the relationship between words in the text through self-attention mechanism (Self-Attention). After tokenization, the input text is represented as a matrix Q = XW Q , K = XW K , V = XW V . Where Q, K, and V are query, key, and value vectors, respectively, and W Q , W K , W V are learned weight matrices. By calculating the similarity between the query and the key, the BERT model outputs the context vector representation of each word.

[0131] (2) Image feature extraction

[0132] Use a pre-trained convolutional neural network to extract high-dimensional feature vectors of images.

[0133] Use a classic CNN model (such as ResNet-50 or EfficientNet) as a feature extractor, and input the image into the convolutional layer of the model: F = CNN(I) where F is the extracted feature vector, containing high-dimensional semantic information. The dimension of the feature vector is usually (N, d), where N is the number of images and d is the feature dimension (such as 2048).

[0134] Before extracting features, perform data augmentation on the image, such as rotation, flipping, random cropping, etc., to improve the robustness of the model.

[0135] (3) Video feature extraction

[0136] Use CNN to extract static image features of each key frame (similar to image feature extraction).

[0137] Temporal-spatial dynamic features of the video are extracted based on a time series model (such as 3D-CNN or LSTM).

[0138] 3D-CNN model inputs time series frames (T, H, W, C) and outputs video features F video :

[0139] F video = 3D-CNN (S);

[0140] wherein T is the number of time frames, H, W, and C are the height, width, and channel number of the frame.

[0141] CNN in the deep learning model is used to extract spatial features of images and videos, and NLP model is used to extract semantic features of text information, and the two work together to realize efficient classification, fusion, and storage of data.

[0142] In another exemplary embodiment of the present application, to ensure the consistency of different modal data in the time dimension, a unified timestamp needs to be added to all data and aligned.

[0143] Generate timestamp, record the collection time t for each data: T = {t1, t2,..., t n}. Where t i is the time point of a single record.

[0144] Align the multi-modal data according to the timestamp, and use the linear interpolation method to process the time difference: D aligned (t) = Interpolate (D modality , T).

[0145] Wherein D aligned (t) refers to the data obtained after aligning all modal data at time t. That is, it is hoped that the data of different modalities is synchronized at the same time (on the same unified timestamp set). Purpose: to eliminate the inconsistency caused by different collection times of different modal data, so that they can be compared and analyzed in the same time dimension.

[0146] Interpolate (D modality , T) means that a certain modal data (such as video frame or sensor data) is interpolated so that it can obtain corresponding data values on the unified timestamp set T.

[0147] Interpolation: refers to the process of calculating unknown data points from known data points. Here, the known modal data points (such as video frame timestamp and corresponding data) and the target timestamp T are interpolated to obtain the data value at the missing time point.

[0148] Specific operation: assuming that the data D of a certain modality modality has a timestamp set T modality ={t1,t2,…}.

[0149] But these timestamps do not completely align with the expected uniform timestamp set T={t'1,t'2,…}. At this time, the corresponding modality data value on T can be estimated by interpolation.

[0150] Linear interpolation: linear interpolation is a commonly used method, which calculates the value of any time point between two known time points by proportion. Assuming that there are two known data points D(t1) and D(t2) between t1 and t2, the value at time t can be calculated by the following formula: This formula uses the data of two known time points t1 and t2 to calculate the estimated value at time t by linear relationship.

[0151] D modality represents a specific modality data set, such as video frame data, sensor data, audio data, etc. These data are usually collected according to different timestamps T modality ={t1,t2,…}. For example: video data may be collected every 0.5 seconds, with timestamps T video ={t1,t2,…}. Because the time intervals of different modalities of data are different, their timestamps T modality are inconsistent. For example, if the time intervals of video frames and sensor data are inconsistent, the missing data at the corresponding time can be generated by interpolation.

[0152] Splicing text, image, video and sensor features to generate a comprehensive feature matrix:

[0153] X fused =[X text ,X image ,X video ,X sensor ].

[0154] In another exemplary embodiment of the present application, in order to realize the unified storage and dynamic association of multi-modal information, a database table structure containing multiple fields needs to be designed. A multi-modal database based on deep learning embedding vector is designed, and the database table structure contains feature storage field, index field and hyperlink identifier, which are used to realize the unified storage and dynamic association of multi-modal information. The index field includes timestamp, plant ID, geographical location and sensor ID, and the index field is used to retrieve single modality data. The hyperlink identifier is used to retrieve cross-modal data, and the hyperlink identifier realizes the accurate association and retrieval of text information and monitoring image, video data, improves the interactivity and usability of the database, and facilitates cross-modal query for users.

[0155] Feature Storage Field: Stores feature vectors for each modality, image features, video features, text features, etc. Features for each modality can be stored in vectorized form, e.g., using binary format to store feature vectors.

[0156] Index Field: Index fields for accelerating data retrieval, such as timestamp, data source, modality type, etc. Each record will have a unique identifier (ID), and indexes can be created based on time, modality, etc.

[0157] Hyperlink Identification: Fields for establishing relationships between different data records. For example, images, videos, and text data collected at different time points for the same plant can be linked through hyperlinks. Each data item will contain hyperlinks to related data, facilitating queries and analysis.

[0158] The database table structure can be designed as shown in Table 1.

[0159] Table 1 Database Table Structure

[0160]

[0161]

[0162] Database Relationship Design:

[0163] To support dynamic association and query of multi-modal data, a reasonable relationship structure needs to be designed. For example, for different modal data (images, videos, texts, etc.) of the same plant, relationships can be established through shared timestamps or species IDs.

[0164] Suppose there is image data image, video data video, and text description data text for a plant, relationships can be established in the database according to timestamps and plant IDs as shown in Tables 2-4.

[0165] Table 2 Image Data Table

[0166] ID Timestamp Plant_ID Modality Feature_Vector Related_Link img_001 2024 / 1 / 112:00 Plant_001 Image BLOB video_001

[0167] Table 3 Video Data Table

[0168] ID Timestamp Plant_ID Modality Feature_Vector Related_Link video_001 2024 / 1 / 112:00 Plant_001 Video BLOB img_001

[0169] Table 4 Text Data Table

[0170] ID Timestamp Plant_ID Modality Feature_Vector Related_Link text_001 2024 / 1 / 112:00 Plant_001 Text BLOB img_001

[0171] Through this design, all relevant data (images, videos, text, etc.) of a certain plant can be flexibly queried, and dynamic association can be made according to the timestamp or plant ID at the time of query. By establishing a hyperlink field, dynamic association between multi-modal data can be achieved. For example, a certain image feature vector in the image data table may be associated with video features and text descriptions of the same plant collected at different times, supporting cross-modal query and data visualization.

[0172] In another exemplary embodiment of the present application, according to the designed database structure, a suitable storage container (Hadoop cluster, cloud storage platform or distributed file system) is used for data storage and management. The present embodiment involves the construction of a storage container using distributed storage technology and the efficient storage and management of multi-modal data to support the plant growth dynamic monitoring and analysis system. By using suitable storage technology (such as Hadoop Distributed File System (HDFS), cloud storage platform or other distributed file system), the high availability, reliability and scalability of system data are ensured. The following are the specific implementation steps:

[0173] (1) Construction of data storage container

[0174] In order to support the efficient storage and management of multi-modal data (such as text, images, videos, sensor data), a storage container needs to be established through distributed storage technology.

[0175] Storage technology:

[0176] Hadoop Distributed File System (HDFS): Suitable for large-scale data storage, with good scalability, fault tolerance and high throughput, capable of supporting large amounts of image, video, sensor and other data storage. HDFS is suitable for high-concurrency data storage requirements and can meet the requirements of the plant growth monitoring system for large data storage.

[0177] Cloud storage platform (such as AWS S3, Aliyun OSS, etc.): For scenarios requiring high availability and global distributed access, cloud storage platforms provide flexible storage expansion and efficient storage management functions. Cloud storage has the feature of on-demand expansion, suitable for large-scale, multi-modal data storage requirements.

[0178] Other distributed file systems: such as Ceph, GlusterFS, etc. These file systems can provide high availability and high fault tolerance, and support efficient storage of multiple types of data.

[0179] (2) Establishing a storage container

[0180] To improve storage efficiency and access speed, data sharding technology is adopted. By splitting large data sets into multiple small data blocks and distributing them across different nodes, the balance and efficiency of data storage are ensured. For example, HDFS splits data files into fixed-size data blocks (usually 128MB) and stores multiple copies on different nodes to ensure high availability of data in the event of node failure.

[0181] According to the database structure design, storage containers related to the plant growth monitoring system are created. Each container is responsible for storing different types of multi-modal data (such as text data containers, image data containers, video data containers, sensor data containers, etc.). When configuring the container, you need to set the storage format and fields for each data type to ensure the standardization and consistency of data storage.

[0182] (3) Data redundancy and fault tolerance mechanism

[0183] Data replication: To ensure high reliability and fault tolerance of data, data replication is set up to ensure that when some nodes fail, data can still be recovered from other replica nodes.

[0184] Fault tolerance mechanism: Automatic fault tolerance mechanism is adopted to ensure that in the event of node failure, the system can automatically recover or migrate data to ensure data integrity and availability.

[0185] (4) Data storage and management

[0186] After establishing the storage container, data storage and management need to be performed according to the designed database structure.

[0187] Multi-modal data storage:

[0188] Text data storage: Text data (such as weather reports, soil analysis reports, plant growth status descriptions, etc.) will be stored in the database through the feature vectors extracted by the natural language processing (NLP) model, with storage formats such as vector format or serialized format (such as JSON, CSV). Each data item includes metadata of the text content (such as data source, timestamp, location, etc.) and the processed feature vector.

[0189] Image data storage: Image data will be stored in binary format in the image data container, with storage paths including plant ID, image type (such as leaf image, root image, etc.), shooting time, etc. Each image will be processed and features extracted (such as morphological features extracted by convolutional neural network) for subsequent cross-modal analysis.

[0190] Video data storage: Video data will be stored in video file formats (such as MP4, AVI, etc.), along with the metadata of the video (such as video ID, shooting time, camera location, etc.). Each video will undergo frame extraction and key frame identification before storage, and will be time-synchronized with image data and sensor data to ensure consistency between video and other data modalities.

[0191] Sensor data storage: Sensor data (such as soil humidity, temperature, light intensity, etc.) will be stored in time series data format, containing timestamp, sensor ID, measurement value, etc. The collected data of each sensor will be synchronized regularly and stored in the designated database table, supporting efficient time series query.

[0192] (5) Data indexing and retrieval

[0193] In order to support efficient data retrieval and query, index fields related to data types are designed (such as timestamp, plant ID, geographic location, sensor ID, etc.). These index fields will enable the system to quickly locate data at a specific time, place or specific plant.

[0194] Each piece of data will establish a hyperlink relationship with other modal data (such as image, video, text, sensor data). For example, image data can be linked with related text data and sensor data through plant ID, realizing cross-modal query. The association between data items is marked by a unique identifier (such as UUID), ensuring the integrity and association of data.

[0195] In another exemplary embodiment of the present application, the database creation parameters can also be determined from the creation instructions, including the application of deep learning technology and the setting of database structure. Based on the creation parameters, the storage container is established in the adaptive cluster through the management service to complete data storage and management.

[0196] Figure 2 A brief flowchart of the plant growth key element multi-modal database creation method of the present application.

[0197] The embodiment of the present application also provides a plant growth state prediction method applying the plant growth key element multi-modal database creation method described above, as shown in Figure 3 The plant growth state prediction method includes the following steps 501-503.

[0198] Step 501: Obtain multi-modal feature vectors at multiple time points during plant growth from the plant growth key element multi-modal database; the multi-modal feature vectors include text feature vectors, image feature vectors, video feature vectors and sensor feature vectors.

[0199] Step 502: After concatenating or weighting the multimodal feature vectors at multiple time points, a cross-modal attention mechanism is used to fuse the features and obtain the fused features.

[0200] Step 503: Input the fused features into the LSTM time-series prediction model and output the plant growth status at the predicted time.

[0201] A time-series prediction model for plant growth dynamics is constructed using Long Short-Term Memory (LSTM) networks. A cross-modal attention mechanism (Transformer) is employed to integrate multimodal features, enabling deep correlation analysis of the data.

[0202] LSTM-based plant growth time series prediction model:

[0203] LSTM is a deep learning model particularly well-suited for processing time-series data. During plant growth, factors such as soil moisture, temperature, and light intensity change over time, and LSTM can effectively capture the long-term dependencies in these time-series data.

[0204] The core of the LSTM model is its unique gating mechanism, including the input gate, forget gate, and output gate. LSTM can be used to predict the growth status of plants (e.g., height, leaf area, etc.). Suppose we have a time-series dataset of plant growth, denoted as X = {x1, x2, ..., x...}. t}, where x t The input features (such as temperature and humidity) are at time t, and the target output is the plant's growth state y at that time. t (e.g., growth height, leaf area, etc.). Multimodal data classification enables continuous monitoring and analysis of plant growth dynamics through time-series data grouping, facilitating the identification of trends and anomalies in plant growth.

[0205] By training an LSTM model, it is possible to predict the growth status of plants at future points in time, such as the plant's height, leaf area, and number of fruits at a certain point in time.

[0206] The training process of an LSTM model uses gradient descent methods (such as the Adam optimizer) to minimize the loss function. Assuming the prediction target is the height of the plant, the output of the LSTM is... The real label is y t Then the loss function is the mean squared error (MSE):

[0207]

[0208] By minimizing the loss function, the parameters W of the LSTM model can be optimized. f W iW C W o and bias term b f ,b i ,b C ,b o .

[0209] Cross-modal feature integration and deep correlation analysis based on Transformer:

[0210] The Transformer model uses self-attention to efficiently process long sequence data and can fuse feature information from different modalities. Through the Transformer, data from different modalities such as images, videos, and text can be integrated to obtain a multimodal feature representation for deep correlation analysis.

[0211] The Transformer consists of an encoder and a decoder, but in cross-modal feature fusion tasks, usually only the encoder is used. The core of the Transformer is the self-attention mechanism.

[0212] Self-Attention mechanism: For each input feature vector X = [x1, x2, ..., x...], ... n First, three vectors are calculated: Query Q, Key K, and Value V. The self-attention mechanism calculates a weighted average for each position through the following steps:

[0213]

[0214] Where d k is the dimension of the query vector, and A is the attention matrix, representing the correlation between each position.

[0215] Multi-Head Attention: To capture different relationships, Transformer typically employs a multi-head attention mechanism.

[0216] MultiHead(Q,K,V)=Concat(head1,head2,...,head h W O ;

[0217] in, W O It is the output linear transformation matrix.

[0218] Cross-modal fusion:

[0219] Through the self-attention mechanism of the Transformer, features from different modalities (such as image features, video features, text features, etc.) can be fused to enable deep correlation analysis of plant growth.

[0220] Assuming there are three modalities of feature vectors After concatenating or weighted summing them, input them into the Transformer model for feature fusion to obtain a comprehensive representation F cross cross across modalities image video text .

[0221] The Transformer model effectively captures the correlation between different modalities through the self-attention mechanism, generating a unified feature representation F cross , which provides in-depth understanding for plant growth analysis.

[0222] The present application realizes the whole process design from multi-modal data acquisition, preprocessing, feature extraction and fusion to data storage, dynamic monitoring and intelligent analysis, providing technical support for precise monitoring and prediction of plant growth, and has good scalability and practicality.

[0223] The present application has the following beneficial effects:

[0224] 1. A plant growth dynamic monitoring and analysis method based on multi-modal data fusion is proposed, which can capture key features of plant growth more comprehensively compared to single modal data, and improve prediction accuracy.

[0225] 2. Deep learning technology (including CNN and Transformer) is innovatively introduced for feature extraction and intelligent analysis of multi-modal data, realizing deep integration of text, image and video information.

[0226] 3. Through multi-modal time synchronization and fusion technology, the accuracy of logical correlation between data is ensured, effectively overcoming the problem of inconsistent time sequence between different modalities of data.

[0227] 4. A multi-modal database with dynamic correlation function is constructed, which can support complex queries and flexible expansion, providing an efficient solution for storage and management of large-scale plant growth data.

[0228] 5. Combined with the time series prediction model and the cross-modal attention mechanism, the dynamic analysis and prediction ability of plant growth state is significantly improved, providing a scientific basis for precision agriculture.

[0229] ​​​Based on the same inventive concept, the embodiments of the present application also provide a plant growth dynamic monitoring system, comprising: a retrieval unit, a visualization interface, a plant growth key element multi-modal database, and a prediction unit.

[0230] The retrieval unit is used to retrieve single-modal data or cross-modal data from the plant growth key element multi-modal database. The prediction unit predicts the plant growth state by using the plant growth state prediction method described above. The visualization interface is used to display the retrieved single-modal data or cross-modal data, and to draw a predicted trend curve of plant growth according to the predicted plant growth state.

[0231] The plant growth key element multi-modal database can dynamically store and manage multi-modal data in the plant growth process, support intelligent retrieval and analysis, and provide a powerful data platform to support agricultural intelligent management.

[0232] The plant growth key element multi-modal database is scalable, and can dynamically add new data fields, retrieval functions and analysis modules according to user needs, continuously supporting the needs of agricultural intelligent management.

[0233] The plant growth dynamic monitoring system supports multi-dimensional retrieval functions, including queries by time, geographic location, plant species or growth stage, enhancing the flexibility of data queries and meeting the diversified needs in agricultural production.

[0234] As an optional implementation, in order to enable users to intuitively understand the dynamic process of plant growth, a visualization interface needs to be built, and multi-dimensional dynamic monitoring and prediction functions need to be integrated.

[0235] The visualization interface should include the following functions:

[0236] ① Plant growth curve: display a curve graph of the change of plant growth state (such as height, leaf area) over time.

[0237] ② Prediction result: based on the LSTM time series prediction model, show the predicted trend of plant growth in the next few days.

[0238] ③ Multi-modal data display: show the correlation analysis results of different modal data (such as images, videos and texts) through charts or pictures.

[0239] ④ Trend chart: draw a plant growth trend chart for the next few days or weeks, including the predicted changes of various indicators.

[0240] ⑤ Heat map: show the influence degree of different growth factors on plant growth, and identify key factors.

[0241] Growth factors include hormone-like growth factors, photosynthetic-like growth factors, enzyme-like growth factors, and vitamin-like growth factors. Hormone-like growth factors: including auxins, gibberellins, cytokinins, abscisic acid, etc. These growth factors can affect plant growth and development, as well as regulate plant adaptation to adversity. Photosynthetic-like growth factors: including photosynthetic pigments, carotenoids, etc. These growth factors can participate in plant photosynthesis and promote plant growth and development. Enzyme-like growth factors: including amylase, protease, oxidase, etc. These growth factors can participate in plant metabolism and material transport, promoting plant growth and development. Vitamin-like growth factors: including vitamin A, vitamin B, vitamin C, etc. These growth factors can participate in plant metabolism and regulate plant adaptation to adversity.

[0242] To determine the degree of influence of different growth factors on plant growth, experimental design, quantitative analysis, and multidimensional evaluation methods are needed.

[0243] I. Experimental design and control variable method

[0244] 1. Single-factor control experiment

[0245] By isolating other variables, only changing the concentration or activity of the target growth factor, observe the changes in plant phenotype (such as plant height, biomass, flowering time, etc.). For example: hormone class: apply exogenous auxin (IAA) or gibberellin (GA) inhibitor, compare with the control group. Vitamins: remove specific vitamins (such as vitamin B) from the culture medium and observe the metabolic blockage.

[0246] 2. Multi-factor interaction experiment

[0247] Use orthogonal experimental design to analyze the synergistic or antagonistic effects of multiple growth factors.

[0248] II. Molecular biology and gene expression analysis

[0249] 1. Gene knockout / overexpression technology

[0250] Silence or enhance specific growth factor-related genes through gene editing technology (such as CRISPR), and observe the changes in phenotype.

[0251] 2. Transcriptome and proteomics

[0252] Analyze the gene expression profile and protein interaction network regulated by growth factors.

[0253] III. Quantitative analysis and statistical model

[0254] 1. Biomass measurement and growth rate calculation

[0255] Determine plant dry weight, leaf area, and other indicators regularly, and fit the growth curve.

[0256] 2. Principal Component Analysis (PCA) with Regression Models

[0257] Statistical analysis of the contribution of each factor to the growth indicators. For example, establish a mathematical model of environmental factors and leafy vegetable growth cycle, quantify the weight of light, temperature.

[0258] Four, Physiological and Biochemical Index Detection

[0259] 1. Enzyme Activity Determination

[0260] Detect the activity changes of key enzymes such as amylase and oxidase. For example: reflect the carbon metabolism state through amylase activity; analyze the regulation of vitamin C on oxidase activity.

[0261] 2. Metabolite Accumulation Analysis

[0262] Use HPLC or mass spectrometry technology to determine the content of hormones, vitamins and other metabolites.

[0263] Five, Environmental Simulation and Long-term Observation

[0264] 1. Controllable Environment Experiment

[0265] Accurately control factor parameters (such as light period, CO2 concentration) in a plant factory or artificial climate chamber.

[0266] 2. Long-term Monitoring in the Field

[0267] Combine data collection of dynamic changes of multiple factors in natural environment.

[0268] Six, Cross-validation and Model Optimization

[0269] Integrate experimental data and theoretical models to predict factor influence weights through machine learning algorithms (such as random forest, neural network).

[0270] The process of generating a heat map using the Seaborn library can be divided into the following steps:

[0271] 1. Data Preparation

[0272] Data Collection: First, collect multi-modal data related to plant growth, including environmental factors (such as light intensity, temperature, humidity) and plant growth indicators (such as height, leaf area).

[0273] Data Organization: Organize these data into a structured DataFrame, where each column represents a variable and each row represents an observation.

[0274] Data Cleaning: Remove outliers, fill in missing values, and ensure data accuracy and consistency.

[0275] 2. Correlation Calculation

[0276] Correlation Matrix: Use the corr() method of pandas to calculate the correlation between each column in the dataframe. This matrix is a two-dimensional array where each element represents the correlation coefficient (ranging from -1 to 1) between two variables.

[0277] Correlation Coefficient: The correlation coefficient measures the strength and direction of the linear relationship between two variables. A positive value indicates a positive correlation, a negative value indicates a negative correlation, and a larger absolute value indicates a stronger correlation.

[0278] 3. Heatmap Generation

[0279] Select Color Map: Choose an appropriate color map to represent the size of the correlation coefficient. Commonly used maps include coolwarm (cool and warm tones), viridis (continuous tone), etc.

[0280] Set Plot Parameters: Determine the size of the plot, line width, whether to display the numerical value of the correlation coefficient, and other parameters.

[0281] Draw Heatmap: Use the heatmap function of Seaborn to generate a heatmap by inputting the correlation matrix.

[0282] 4. Graph Display and Interpretation

[0283] Graph Display: Display the generated heatmap through matplotlib, and add a title and labels to make the graph clearer and easier to understand.

[0284] Result Interpretation: Analyze the degree of influence of different growth factors on plant growth through the color depth and numerical value of the correlation coefficient in the heatmap, and identify key factors.

[0285] Distribution and Confidence Interval: Draw a distribution graph of the prediction results, and combine the confidence interval to provide uncertainty measurement for the prediction results.

[0286] The user interface can be designed as an interactive dashboard to display dynamic data in multiple dimensions. For example, users can select different time periods, different plant species, or different environmental conditions to view growth trend predictions, historical data comparisons, and the impact of different modal features.

[0287] Using time series prediction models to intelligently analyze and accurately predict plant growth processes, according to the internal logic and association rules of the fused data, output visual analysis results, and provide scientific guidance for agricultural production.

[0288] By integrating LSTM and Transformer models, the system can provide the following dynamic monitoring functions:

[0289] Real-time monitoring: Updates plant growth status in real time based on sensor data (such as temperature, humidity, and light) and predicts growth trends for the next few days.

[0290] Accurate prediction: The LSTM time series model is used to output predictions of future plant growth, which are then optimized by combining historical data to provide scientific guidance for agricultural production.

[0291] As an optional implementation, the plant growth dynamic monitoring system also includes a decision support unit. The decision support unit is used to provide irrigation plans based on soil moisture and weather forecasts; recommend fertilization strategies based on changes in soil nutrients and plant requirements; and provide early warnings of potential pest and disease risks.

[0292] Decision support systems combine the results of time-series forecasts to provide scientific guidance for agricultural production.

[0293] Irrigation: The system automatically provides irrigation plans based on soil moisture and weather forecasts.

[0294] Fertilization: Based on changes in soil nutrients and plant needs, fertilization strategies are recommended.

[0295] Pest and disease early warning: By analyzing the correlation between climate, vegetation growth and historical pest and disease data, intelligent early warning of potential pest and disease risks can be provided.

[0296] Real-time feedback and optimization: The system can adjust and optimize based on the deviation between real-time data and prediction results. For example, when plant growth rate deviates from expectations, the system can adjust relevant agricultural operations (such as temperature and humidity control in greenhouses).

[0297] The process of providing an irrigation solution is as follows:

[0298] 1. Data Collection

[0299] Soil moisture monitoring: Soil moisture is monitored in real time using moisture sensors buried in the soil.

[0300] Weather forecast data acquisition: Obtain information such as the probability of precipitation and temperature changes for the next few days from meteorological forecasts.

[0301] 2. Irrigation Decision Making

[0302] Multi-source data fusion: Combining soil moisture data with weather forecast data for comprehensive analysis.

[0303] Soil moisture-based decision-making: Based on real-time soil moisture data and preset moisture thresholds, the system determines whether irrigation is needed. If the soil moisture is below the set lower limit, irrigation will be initiated; if it is above the set upper limit, irrigation will be stopped.

[0304] Optimization in conjunction with weather forecasts: Based on the precipitation probability of weather forecasts, the system will adjust the irrigation strategy. If the precipitation probability is high in the next few days, the system will reduce the irrigation amount or delay the irrigation time; if the precipitation probability is low, the system will increase the irrigation amount or advance the irrigation time.

[0305] 4. Irrigation scheme execution

[0306] Control of irrigation equipment: The system sends signals to the irrigation equipment (such as sprinklers or drip irrigation systems) through the controller to start or stop irrigation, and accurately control the amount and time of irrigation.

[0307] 5. Real-time monitoring and feedback

[0308] Irrigation process monitoring: The system continuously monitors key parameters during the irrigation process, such as soil moisture, air temperature, etc., to ensure the optimization of irrigation effect.

[0309] Soil parameters: Volumetric water content; Electrical conductivity (0-5 dS / m); Temperature (5 cm depth); Field capacity threshold (FC = θ@-33 kPa).

[0310] Meteorological parameters: Precipitation probability in the next 72 hours; Evapotranspiration (ET0); Wind speed (affecting sprinkler efficiency); Dew point temperature (dew compensation).

[0311] The recommendation process of fertilization strategy is as follows:

[0312] I. Soil nutrient testing

[0313] 1. Soil sampling: Collect representative soil samples in the field to ensure that the samples accurately reflect the nutrient status of the soil.

[0314] 2. Laboratory analysis: Send the collected soil samples to a professional laboratory to determine the content of main nutrients such as nitrogen, phosphorus, and potassium in the soil, as well as the pH value, organic matter content, and other physical and chemical properties of the soil.

[0315] II. Analysis of plant nutrient requirement characteristics

[0316] 1. Fertilizer requirement rules at different growth stages: Understand the nutrient demand characteristics of plants at different growth stages (such as seedling stage, flowering stage, fruiting stage, etc.) for nitrogen, phosphorus, potassium, etc. For example, vegetables need more nitrogen fertilizer in the early growth stage to promote stem and leaf growth, and more phosphorus and potassium fertilizer in the flowering and fruiting stage to promote fruit development.

[0317] 2. Differences between crop types: Different types of plants have different nutrient requirements. For example, leguminous plants can fix nitrogen through rhizobium, and have relatively low nitrogen fertilizer requirements; while grass crops such as wheat and rice have high nitrogen fertilizer requirements.

[0318] III. Fertilization Strategy

[0319] 1. Distribution of Base Fertilizer and Topdressing:

[0320] Base Fertilizer: According to soil nutrient test results and plant nutrient characteristics, determine the type and amount of base fertilizer. Generally, base fertilizer mainly uses organic fertilizer and part of chemical fertilizer, such as more than 60% of phosphorus fertilizer, 20%-30% of nitrogen and potassium fertilizer as base fertilizer.

[0321] Topdressing: During plant growth, according to its growth stage and nutrient demand law, timely and appropriate topdressing is carried out. For example, during the flowering period to the fruiting period of vegetables, low-phosphorus high-potassium formula fertilizer can be applied.

[0322] 2. Fertilizer Selection:

[0323] Nitrogen Fertilizer: Common nitrogen fertilizers such as urea and ammonium sulfate can be selected for nitrogen fertilizer application. For areas with abundant rainfall, nitrogen fertilizers containing efficiency-enhancing agents (such as urease inhibitors and nitrification inhibitors) can be recommended to improve nitrogen utilization efficiency.

[0324] Phosphorus Fertilizer: Phosphorus fertilizer is mainly used as base fertilizer, and can choose superphosphate, calcium-magnesium phosphate, etc. In acid soil, the application amount of phosphorus fertilizer should be appropriately increased to improve the effectiveness of phosphorus.

[0325] Potassium Fertilizer: Potassium fertilizer is in great demand during the middle and late stages of plant growth, especially during the fruit enlargement period. Potassium fertilizers such as potassium chloride and potassium sulfate can be selected.

[0326] 3. Fertilization Time and Method:

[0327] Timely Fertilization: According to the growth stage of plants and the nutrient demand law, the best fertilization time is selected. For example, during the rosette stage to the post-heading stage of vegetables, calcium chloride or calcium nitrate solution can be sprayed on the leaves to supplement calcium elements.

[0328] Fertilization Method: Reasonable fertilization methods such as strip application, hole application, and foliar spraying can be adopted. For protected vegetables, drip irrigation fertilization technology can be used, following the principle of "small amount and multiple times" to improve water and fertilizer utilization efficiency.

[0329] IV. Special Situation Handling

[0330] 1. Soil pH Adjustment: For acid soil, lime can be applied to adjust the soil pH value and improve the effectiveness of phosphorus in the soil.

[0331] 2. Medium and Trace Element Supplementation: According to soil test results and plant needs, calcium, magnesium, boron, and other medium and trace elements can be supplemented in a timely manner. For example, in boron-deficient plots, borax can be applied as base fertilizer or borax solution can be sprayed on the leaves.

[0332] 3. Extreme weather response: In areas with frequent extreme weather, such as low-temperature and low-light environments, functional substances (such as amino acids, fulvic acid, and alginic acid) can be added to promote root growth and enhance the stress resistance of crops.

[0333] V. Fertilization effect monitoring and adjustment

[0334] 1. Regular monitoring: After fertilization, regularly monitor changes in soil nutrients and plant growth conditions to evaluate the effect of fertilization.

[0335] 2. Timely adjustment: Based on the monitoring results, timely adjust the fertilization strategy. If it is found that a certain nutrient is insufficient or excessive, the fertilization amount or time should be adjusted in time.

[0336] The process of intelligent early warning of potential pest risk is as follows:

[0337] The system is based on a multi-source sensor network, with environmental monitoring equipment, image acquisition devices, and biological information sensors deployed in the field and at the field head, forming a three-dimensional perception system covering the entire growth cycle of crops.

[0338] Environmental perception layer: Real-time collection of microclimate data by temperature and humidity sensors and light intensity monitors provides environmental risk early warning for disease and pest occurrence.

[0339] Crop physiology monitoring: Analyze vegetation index (such as NDVI) through spectral sensors, combined with stem micro-change sensors to dynamically assess crop health status.

[0340] Targeted identification of diseases and pests: Intelligent traps equipped with image recognition modules automatically count pest species and quantity and monitor pathogens in real time.

[0341] The core of the system is to build a multi-dimensional correlation model of "environment-crop-disease and pest", and to mine potential laws in massive data through machine learning.

[0342] Spacetime prediction algorithm: Based on LSTM neural network analysis of historical disease and pest occurrence data and meteorological satellite data, establish regional occurrence probability atlas;

[0343] Dynamic threshold determination: Continuously optimize early warning indicators through reinforcement learning, such as automatically triggering prevention and control instructions when aphid density exceeds 2 times the historical average for 3 consecutive days;

[0344] Multi-disaster coupling analysis: Integrate drought, high temperature and other stress data to evaluate their additive effects on disease and pest outbreaks and generate a comprehensive risk index.

[0345] The system breaks through the limitations of traditional monitoring tools and builds a complete management closed loop of "monitoring-early warning-decision-execution".

[0346] Intelligent decision support: Based on crop growth stage, economic value and pesticide residue standards, automatically generate differentiated prevention and control plans such as biological control, physical and chemical control, etc.

[0347] Precise pesticide application control: Interface with intelligent spraying robots, combined with spatial distribution thermodynamic maps of pests and diseases, achieve variable rate application and targeted prevention and control.

[0348] Blockchain traceability management: Store monitoring data and control operations on the chain to provide full-cycle traceable certificates for agricultural product quality and safety.

[0349] As an optional implementation, during the development of the system, the prediction accuracy is further improved by continuously optimizing the hyperparameters of the model, increasing the training data, and using regularization techniques. Through continuous feedback and adjustment, the intelligent analysis system will be able to provide more accurate predictions and scientific guidance based on continuously updated data, ultimately achieving the goal of intelligent agriculture.

[0350] Here are some common regularization methods and their applications:

[0351] L1 regularization (Lasso): L1 regularization adds an L1 norm term to the loss function to constrain the weights of the model, which makes the weight vector of the model sparse, i.e. some weights become zero. This can reduce the complexity of the model and improve the generalization ability of the model, thereby improving the prediction accuracy. For example, in a linear regression model, Lasso regression can be used to implement L1 regularization.

[0352] L2 regularization (Ridge): L2 regularization adds an L2 norm term to the loss function to constrain the weights of the model, which makes the weight vector of the model smooth, i.e. the weight values are uniformly distributed and small. This helps to prevent the model from being too complex, reduces overfitting, and improves the prediction accuracy of the model. In practical applications, L2 regularization is commonly used in linear regression, logistic regression and other models.

[0353] Dropout is a method of randomly dropping some neurons during the training process of neural networks. In this way, the complex co-adaptation between neurons can be reduced, and the generalization ability of the model can be improved, thereby improving the prediction accuracy. In deep learning, dropout is widely used in various neural network models such as convolutional neural networks (CNN) and recurrent neural networks (RNN).

[0354] The overall technical solution of the present application is as follows: Figure 4As shown. Through deep learning technology, the method constructs a multi-dimensional data structure covering six elements and ten growth indicators, and realizes the unified storage, dynamic association and deep analysis of multi-modal information such as text information, monitoring photos and video data. Specifically, it includes determining the data collection scheme and designing the preprocessing process; using data fusion technology to integrate multi-modal data, constructing a multi-modal database based on deep learning embedding vectors, and supporting cross-modal retrieval; according to the designed database structure, constructing a storage container for data storage and management; developing an intelligent retrieval and analysis system, realizing intelligent analysis and prediction of plant growth dynamics through LSTM and Transformer models, and constructing an intelligent system integrating retrieval, analysis and visualization, supporting intelligent management and accurate decision-making in agriculture. The present application improves the efficiency of data integration and analysis, enhances the ability of plant growth monitoring and accurate agricultural decision-making, and is suitable for large-scale agricultural data management and application.

[0355] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0356] The principles and implementation modes of the present application are described by applying specific examples herein, and the above embodiments are only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A plant growth state prediction method characterized by comprising: The plant growth state prediction method comprises: According to the target database related to plant growth, determine the multi-modal data acquisition requirement related to plant growth; According to the multi-modal data acquisition requirement, collect multi-modal data in the plant growth process; the multi-modal data includes text record information, image, video and sensing data; Preprocessing multi-modal data; Using a deep learning model to extract the preprocessed text record information, the preprocessed image and the preprocessed video respectively, to obtain text features, image features and video features; Time alignment is performed on the text features, image features, video features and preprocessed sensing data to obtain text feature vectors, image feature vectors, video feature vectors and sensing feature vectors; According to the multi-modal data acquisition requirement, determine the database table structure; the database table structure includes feature storage field, index field and hyperlink identifier; the hyperlink identifier is used to establish the hyperlink relationship between each data and the modal data other than the modal to which each data belongs; According to the database table structure, establish a storage container for each kind of modal data; The text feature vectors, image feature vectors, video feature vectors and sensing feature vectors are respectively stored in the respective storage containers to obtain a plant growth key element multi-modal database; Obtain multi-modal feature vectors at multiple times in the plant growth process from the plant growth key element multi-modal database; the multi-modal feature vectors include text feature vectors, image feature vectors, video feature vectors and sensing feature vectors; After splicing or weighted summing the multi-modal feature vectors at multiple times, feature fusion is performed by using a cross-modal attention mechanism to obtain fusion features; The fusion features are input into an LSTM time series prediction model to output the plant growth state at the prediction time.

2. The plant growth state prediction method according to claim 1, characterized by, The multi-modal data acquisition requirement includes six elements and ten indexes; The six elements include soil, temperature, light, moisture, air and fertilizer; The ten indexes include air temperature, air humidity, air pressure, light, noise, carbon dioxide, soil temperature, soil humidity, soil EC value and soil PH value.

3. The plant growth state prediction method according to claim 1, characterized by, According to the multi-modal data acquisition requirement, collect multi-modal data in the plant growth process, which specifically includes: Record the plant species, planting time, fertilization record, growth environment parameters and pest information in the plant growth process as text record information; Collect visible light images and infrared images of the plant at regular intervals; Use a video camera to record the video of the dynamic growth process of the plant; According to the multi-modal data acquisition requirement, determine the sensing acquisition equipment; Collect sensing data in the plant growth process by using the sensing acquisition equipment.

4. The plant growth state prediction method according to claim 1, characterized by, The preprocessing of multi-modal data specifically includes: Data cleaning is performed on the text record information, and non-numeric fields are encoded to obtain preprocessed text record information; The image brightness is adjusted using a gamma correction algorithm, and the resolution is unified to obtain the preliminarily adjusted image; The preliminarily adjusted image is normalized, and a U-Net model is used to extract the plant region of interest from the normalized image to obtain the plant region of interest image as the preprocessed image; The key frames of the video are extracted using an inter-frame difference method, the extracted key frames are arranged in chronological order, and a key frame time sequence is generated by a sliding window method as the preprocessed video; Each type of sensor data is smoothed and filtered, and the smoothed and filtered multi-source sensor data is synchronized and aligned by timestamp to obtain preprocessed sensor data.

5. The plant growth state prediction method according to claim 1, characterized by, A deep learning model is used to extract preprocessed text record information, preprocessed images and preprocessed videos to obtain text features, image features and video features, specifically including: The preprocessed text record information is processed by word segmentation and stop words are deleted to obtain text words; A pre-trained word vector model is used to convert the text words into vector form to obtain the text word vector; A natural language processing model is used to extract text features based on the text word vector; A pre-trained convolutional neural network is used to extract features of the preprocessed images to obtain image features; A convolutional neural network is used to extract static image features of the preprocessed video; A time series model is used to extract spatiotemporal dynamic features of the preprocessed video, which together with the static image features form the video features.

6. The plant growth state prediction method according to claim 1, characterized by, The index field includes timestamp, plant ID, geographic location and sensor ID; the index field is used to retrieve single-modal data; The hyperlink identifier is used to retrieve cross-modal data.

7. The plant growth state prediction method according to claim 1, characterized by, The storage container is a distributed file system or a cloud storage platform.

8. A plant growth dynamic monitoring system, characterized by, It includes: a retrieval unit, a visualization interface, a plant growth key element multi-modal database and a prediction unit; The retrieval unit is used to retrieve single-modal data or cross-modal data from the plant growth key element multi-modal database; The prediction unit uses the plant growth state prediction method of claim 1 to predict the plant growth state; The visualization interface is used to display the retrieved single-modal data or cross-modal data, and to draw a predicted trend curve of plant growth according to the predicted plant growth state.

9. The plant growth dynamic monitoring system of claim 8, wherein, It also includes: a decision support unit; The decision support unit provides irrigation schemes according to soil moisture and weather forecasts; Based on the changes in soil nutrients and plant needs, it recommends fertilization strategies; And warn of potential disease and pest risks.

Images