Intelligent plant phenotype analysis platform based on computer vision and internet of things technology

Abstract

The present application relates to a computer vision and internet of things technology-based intelligent plant phenotype analysis platform, comprising a data acquisition device, a cloud management platform and a phenotype prediction model; the data acquisition device is used to acquire multi-view two-dimensional images, left and right views and environmental time series data of plants; the cloud management platform is used to store and display the collected data and realize the regulation and control of the operation parameters of the data acquisition device; the phenotype prediction model is deployed on the cloud management platform and is used to receive the left and right views, multi-view two-dimensional images and environmental time series data acquired by the data acquisition device, predict and output the physiological indexes of plants; the present application has the beneficial effects of reducing the system deployment cost and maintenance difficulty, and realizing the full-link automatic closed loop from multi-view image acquisition, environmental sensing monitoring to cloud data storage and remote regulation and control.

Description

Technical Field

[0001] This invention relates to the field of plant phenotypic monitoring technology, and more specifically, to an intelligent plant phenotypic analysis platform based on computer vision and Internet of Things (IoT) technologies. Background Technology

[0002] Agricultural informatization and intelligentization, as important development directions of modern agricultural technology, are rapidly permeating all aspects of crop cultivation and monitoring, driving agricultural production towards precision. With the continuous improvement of global requirements for agricultural production efficiency and crop quality, plant phenotyping analysis, as a key link in assessing plant growth status and guiding resource allocation, is becoming increasingly important. Traditional plant phenotyping data acquisition mainly relies on manual contact measurement or large-scale fixed scanning and analysis equipment, which is limited in deployment and prone to causing irreversible physical damage to plants. Furthermore, existing systems generally store and analyze monitoring data from different sources independently, lacking effective data fusion mechanisms, making it difficult to dynamically and accurately predict the future growth status and physiological indicators of plants.

[0003] Furthermore, the 3D point clouds of plants reconstructed by binocular vision typically exhibit disorder and irregular adjacency relationships. Directly processing these points using a 2D convolutional network makes it difficult to represent the spatial connections between leaf points, local geometry, and canopy structure distribution. Therefore, how to convert the 3D point cloud of plants into a structured input suitable for neural network processing is the key technical problem addressed in this invention. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent plant phenotyping platform based on computer vision and Internet of Things technology.

[0005] Firstly, it provides an intelligent plant phenotyping platform based on computer vision and IoT technologies, including:

[0006] Data acquisition devices, cloud management platform, and phenotypic prediction models:

[0007] The data acquisition device is used to acquire multi-view two-dimensional images of plants, left and right views, and environmental time-series data.

[0008] The cloud management platform is communicatively connected to the data acquisition device, and is used to store and display the acquired data, as well as to control the operating parameters of the data acquisition device.

[0009] The phenotypic prediction model is deployed on a cloud management platform to receive left and right views, multi-view two-dimensional images, and environmental time-series data acquired by the data acquisition device. Based on the left and right views, a three-dimensional point cloud of the plant is generated. The three-dimensional point cloud is constructed into a three-dimensional phenotypic map of the plant containing node features and edge connections. The three-dimensional structural features of the plant are extracted using a graph neural network. At the same time, the time-series features of the environmental time-series data are extracted. After fusing the three-dimensional structural features and the environmental time-series features, the physiological indicators of the plant are predicted and output.

[0010] Preferably, the data acquisition device includes:

[0011] The visual acquisition module is used to simultaneously acquire left and right views and multi-view two-dimensional images of plants from different angles using a binocular camera.

[0012] The environmental sensing module is used to acquire ambient temperature, humidity, and light intensity data through temperature and humidity sensors and light intensity sensors, and encapsulate them into environmental time-series data.

[0013] The motion control module is used to control the rotating base that supports the plant to rotate at a set angle and frequency to facilitate multi-view visual acquisition.

[0014] Preferably, the cloud management platform includes:

[0015] An interface communication module is used to receive data and issue commands to the data acquisition device.

[0016] The cloud storage module is used to uniformly store the collected image data, environmental time-series data, and device operation logs;

[0017] The visualization and interaction module is used to display sensor data and the output results of the phenotypic prediction model in real time, and supports remote configuration of the acquisition parameters.

[0018] Preferably, the phenotypic prediction model includes:

[0019] The data processing module is used to perform stereo correction, parallax calculation and three-dimensional back projection on the left and right views acquired by the binocular camera to generate a three-dimensional point cloud of plants. The module also performs noise reduction, background removal and downsampling on the three-dimensional point cloud of plants. At the same time, it performs missing value filling and standardization on the acquired sensor data to construct environmental time series data.

[0020] Graph construction and feature encoding module: used to convert the plant 3D point cloud into a plant 3D phenotypic map containing node features and edge connections, extract plant 3D structural features through graph neural network, and extract temporal features of environmental temporal data through temporal coding network;

[0021] The modality fusion module is used to align and fuse the three-dimensional structural features with the environmental temporal features to generate a unified joint feature vector;

[0022] The task prediction module is used to output the prediction results of the plant's future physiological indicators, including the relative chlorophyll content and leaf area, based on the unified joint feature vector and the physiological indicator prediction head.

[0023] Preferably, in the data processing module of the phenotypic prediction model:

[0024] A background filtering algorithm based on color space is used to remove pixels that match the background features in a two-dimensional image, thereby extracting the main image of the plant.

[0025] Stereo correction is performed on the left and right views captured by the binocular camera, and a 3D point cloud of plants is generated based on parallax calculation and 3D back projection; depth range filtering, outlier removal and downsampling processing are performed on the plant 3D point cloud to obtain a set of point cloud sampling points for graph structure construction;

[0026] When missing sensor data is detected, an interpolation algorithm is used to fill in the missing data by combining valid values ​​from adjacent historical moments.

[0027] A normalization algorithm is used to perform uniform numerical scaling on the imputed data, transforming it into standardized environmental time series data.

[0028] Preferably, in the graph construction and feature encoding module of the phenotypic prediction model:

[0029] The point cloud graph construction unit uses point cloud sampling points as graph nodes, and uses at least one of the spatial coordinates, color components, local neighborhood density, normal vector, or curvature of the sampling points as node features, and establishes edge connections between nodes based on proximity relationships; the edge connections have edge features, and the edge features include the relative spatial relationship between adjacent nodes;

[0030] The graph neural coding unit aggregates neighborhood features based on the node features and edge connections to obtain three-dimensional structural features representing the unfolded morphology of plant leaves, canopy structure, and spatial density distribution.

[0031] By using a time-series coding network to read standardized environmental time-series data in chronological order, the fluctuation and change patterns of environmental parameters over time are captured, and environmental time-series features are generated.

[0032] Preferably, in the modal fusion module of the phenotypic prediction model:

[0033] Based on the timestamp of data acquisition, the three-dimensional structural features corresponding to the same growth moment are aligned with the environmental temporal features, and the two are combined into a unified joint feature vector using feature splicing operation.

[0034] Multilayer perceptron is used to perform feature weight allocation and nonlinear mapping on the unified joint feature vector to establish an intrinsic correspondence between environmental changes and the three-dimensional spatial morphology of plants.

[0035] Preferably, in the task prediction module of the phenotypic prediction model:

[0036] The physiological index prediction head constructs mutually independent leaf area regression prediction branches and chlorophyll relative content regression prediction branches, and receives the unified joint feature vector output by the modality fusion module as input;

[0037] The two regression prediction branches use a multilayer perceptron to perform nonlinear mapping and numerical regression calculation on the input joint feature vector, and are optimized and trained through a joint loss function. Finally, they independently map and output the predicted values ​​of leaf area and relative chlorophyll content.

[0038] Secondly, a smart plant phenotyping method based on computer vision and Internet of Things (IoT) technologies is provided, which is applied to the analysis platform described in any one of the first aspects, and includes the following steps:

[0039] S1. Data acquisition steps: Acquire multi-view two-dimensional images, left and right views, and environmental time-series data of the plant through the data acquisition device, and upload them to the cloud management platform;

[0040] S2. Point cloud generation and preprocessing steps: Perform stereo correction, parallax calculation and three-dimensional back projection on the left and right views to generate a three-dimensional point cloud of plants, and perform noise reduction, background removal and downsampling on the point cloud; at the same time, perform missing value filling and standardization on the sensor data to obtain standardized environmental time series data.

[0041] S3. Graph Construction and Feature Encoding Steps: The preprocessed 3D point cloud of plants is converted into a 3D phenotypic map of plants containing node features and edge connections. Graph neural networks are used to extract 3D structural features. At the same time, temporal coding networks are used to extract temporal features of environmental temporal data.

[0042] S4. Modal fusion step: Align and fuse the three-dimensional structural features with the environmental temporal features to generate a unified joint feature vector;

[0043] S5. Task prediction step: Based on the unified joint feature vector, the physiological index prediction head outputs the prediction results of the plant's physiological index, including the relative chlorophyll content and leaf area.

[0044] As a preferred option, in step S3:

[0045] Point cloud sampling points are used as graph nodes, and at least one of the spatial coordinates, color components, local neighborhood density, normal vector or curvature of the sampling points is used as node features. Edge connections between nodes are established based on proximity relationships, and the edge connections have edge features that include the relative spatial relationship between adjacent nodes.

[0046] The graph neural coding unit aggregates neighborhood features based on the node features and edge connections to obtain three-dimensional structural features;

[0047] The temporal coding network reads the standardized environmental temporal data in chronological order and generates environmental temporal features.

[0048] The beneficial effects of this invention are:

[0049] 1. The data acquisition device and cloud management platform of the hardware layer of this invention are built using existing mature commercial components and conventional Internet of Things architecture. This not only reduces the system deployment cost and maintenance difficulty, but also realizes the full-link automated closed loop from multi-view image acquisition and environmental sensing monitoring to cloud data storage and remote control. It can complete real-time, non-contact monitoring of the entire growth cycle of plants while maintaining the natural growth state of plants, providing stable and reliable data support for subsequent intelligent phenotypic prediction.

[0050] 2. This invention addresses the disordered and complex spatial structure of plant 3D point clouds by designing a proprietary strategy for constructing plant 3D phenotypic maps and a graph neural network encoding mechanism. Unlike general point cloud networks that only extract global features, this invention uses point cloud sampling points as nodes, constructs edges based on spatial proximity, and introduces edge features that include the relative spatial relationships between points. This allows the graph attention aggregation process to simultaneously utilize node attributes and edge geometric information, thereby accurately preserving the local surface orientation of leaves, edge curvature, and canopy layering structure, significantly enhancing the model's ability to represent the 3D spatial morphology of plants.

[0051] 3. This invention designs a nonlinear modal fusion module to deeply couple and align the 3D structural features extracted by the graph neural network with the environmental temporal features extracted by the temporal coding network, effectively modeling the intrinsic relationship between environmental temperature and humidity, light changes, and the dynamic evolution of plant 3D morphology. Simultaneously, it employs a bi-branch joint loss function to collaboratively optimize the regression prediction of two physiological indicators: leaf area and relative chlorophyll content. This overcomes the problem of unbalanced prediction accuracy among indicators during independent training in a single task, significantly improving the overall robustness of multi-task prediction and providing reliable technical support for precise phenotypic monitoring in smart agriculture. Attached Figure Description

[0052] Figure 1 This is a diagram showing the overall architecture of the intelligent plant phenotyping platform of the present invention.

[0053] Figure 2 This is a technical roadmap of the data acquisition device for the intelligent plant phenotyping platform of the present invention;

[0054] Figure 3 This is a roadmap of the cloud management platform technology for the intelligent plant phenotyping platform of this invention;

[0055] Figure 4 This is a technical roadmap for the phenotypic prediction model of the intelligent plant phenotypic analysis platform of this invention. Detailed Implementation

[0056] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0057] Example 1:

[0058] To address the problems of existing technologies, Embodiment 1 of this invention provides an intelligent plant phenotypic analysis platform based on computer vision and Internet of Things (IoT) technologies, such as... Figure 1 As shown, it includes:

[0059] Data acquisition devices, cloud management platform, and phenotypic prediction models:

[0060] The data acquisition device is used to acquire multi-view two-dimensional images of plants, left and right views, and environmental time-series data.

[0061] The cloud management platform is communicatively connected to the data acquisition device, and is used to store and display the acquired data, as well as to control the operating parameters of the data acquisition device.

[0062] The phenotypic prediction model is deployed on a cloud management platform to receive left and right views, multi-view two-dimensional images, and environmental time-series data acquired by the data acquisition device. Based on the left and right views, a three-dimensional point cloud of the plant is generated. The three-dimensional point cloud is constructed into a three-dimensional phenotypic map of the plant containing node features and edge connections. The three-dimensional structural features of the plant are extracted using a graph neural network. At the same time, the time-series features of the environmental time-series data are extracted. After fusing the three-dimensional structural features and the environmental time-series features, the physiological indicators of the plant are predicted and output.

[0063] like Figure 2 As shown, the data acquisition device includes:

[0064] The visual acquisition module is used to simultaneously acquire left and right views and multi-view two-dimensional images of plants from different angles using a binocular camera.

[0065] The environmental sensing module is used to acquire ambient temperature, humidity, and light intensity data through temperature and humidity sensors and light intensity sensors, and encapsulate them into environmental time-series data.

[0066] The motion control module is used to control the rotating base that supports the plant to rotate at a set angle and frequency to facilitate multi-view visual acquisition.

[0067] like Figure 3 As shown, the cloud management platform includes:

[0068] An interface communication module is used to receive data and issue commands to the data acquisition device.

[0069] The cloud storage module is used to uniformly store the collected image data, environmental time-series data, and device operation logs;

[0070] The visualization and interaction module is used to display sensor data and the output results of the phenotypic prediction model in real time, and supports remote configuration of the acquisition parameters.

[0071] like Figure 4 As shown, the phenotypic prediction model includes:

[0072] The data processing module is used to perform stereo correction, parallax calculation and three-dimensional back projection on the left and right views acquired by the binocular camera to generate a three-dimensional point cloud of plants. The module also performs noise reduction, background removal and downsampling on the three-dimensional point cloud of plants. At the same time, it performs missing value filling and standardization on the acquired sensor data to construct environmental time series data.

[0073] Graph construction and feature encoding module: used to convert the plant 3D point cloud into a plant 3D phenotypic map containing node features and edge connections, extract plant 3D structural features through graph neural network, and extract temporal features of environmental temporal data through temporal coding network;

[0074] The modality fusion module is used to align and fuse the three-dimensional structural features with the environmental temporal features to generate a unified joint feature vector;

[0075] The task prediction module is used to output the prediction results of the plant's future physiological indicators, including the relative chlorophyll content and leaf area, based on the unified joint feature vector and the physiological indicator prediction head.

[0076] In the data processing module of the phenotypic prediction model:

[0077] A background filtering algorithm based on color space is used to remove pixels that match the background features in a two-dimensional image, thereby extracting the main image of the plant.

[0078] Stereo correction is performed on the left and right views captured by the binocular camera, and a 3D point cloud of plants is generated based on parallax calculation and 3D back projection; depth range filtering, outlier removal and downsampling processing are performed on the plant 3D point cloud to obtain a set of point cloud sampling points for graph structure construction;

[0079] When missing sensor data is detected, an interpolation algorithm is used to fill in the missing data by combining valid values ​​from adjacent historical moments.

[0080] A normalization algorithm is used to perform uniform numerical scaling on the imputed data, transforming it into standardized environmental time series data.

[0081] In the graph construction and feature encoding module of the phenotypic prediction model:

[0082] The point cloud graph construction unit uses point cloud sampling points as graph nodes, and uses at least one of the spatial coordinates, color components, local neighborhood density, normal vector, or curvature of the sampling points as node features, and establishes edge connections between nodes based on proximity relationships; the edge connections have edge features, and the edge features include the relative spatial relationship between adjacent nodes;

[0083] The graph neural coding unit aggregates neighborhood features based on the node features and edge connections to obtain three-dimensional structural features representing the unfolded morphology of plant leaves, canopy structure, and spatial density distribution.

[0084] By using a time-series coding network to read standardized environmental time-series data in chronological order, the fluctuation and change patterns of environmental parameters over time are captured, and environmental time-series features are generated.

[0085] In the modal fusion module of the phenotypic prediction model:

[0086] Based on the timestamp of data acquisition, the three-dimensional structural features corresponding to the same growth moment are aligned with the environmental temporal features, and the two are combined into a unified joint feature vector using feature splicing operation.

[0087] Multilayer perceptron is used to perform feature weight allocation and nonlinear mapping on the unified joint feature vector to establish an intrinsic correspondence between environmental changes and the three-dimensional spatial morphology of plants.

[0088] In the task prediction module of the phenotypic prediction model:

[0089] The physiological index prediction head constructs mutually independent leaf area regression prediction branches and chlorophyll relative content regression prediction branches, and receives the unified joint feature vector output by the modality fusion module as input;

[0090] The two regression prediction branches use a multilayer perceptron to perform nonlinear mapping and numerical regression calculation on the input joint feature vector, and are optimized and trained through a joint loss function. Finally, they independently map and output the predicted values ​​of leaf area and relative chlorophyll content.

[0091] Example 2:

[0092] Based on Example 1, Example 2 of the present invention provides a more specific intelligent plant phenotypic analysis platform based on computer vision and Internet of Things technology, including: a data acquisition device, a cloud management platform, and a phenotypic prediction model.

[0093] The data acquisition device uses a Raspberry Pi as the main control chip, a binocular USB camera for acquiring multi-angle left and right views, a GY-30 light sensor and a DHT11 temperature and humidity sensor for acquiring environmental time-series data, and a stepper motor to drive a rotating base to achieve multi-angle image acquisition. The cloud management platform adopts a conventional IoT cloud architecture, stores data through a MySQL database, and provides a real-time visualization panel and acquisition parameter configuration interface. The above hardware selection and cloud architecture are common knowledge in the field, and the specific implementation methods will not be elaborated further.

[0094] The phenotypic prediction model consists of a data processing module, a graph construction and feature encoding module, a modality fusion module, and a task prediction module. Its core processing flow is as follows:

[0095] S1. Data Processing: The data processing module performs stereo correction, parallax calculation, and three-dimensional back projection based on the left and right views captured by the binocular camera to generate a three-dimensional point cloud of plants; and performs white background removal, depth range filtering, outlier removal, and voxel downsampling on the three-dimensional point cloud of plants, while simultaneously performing missing value imputation and normalization on the environmental time series data.

[0096] Specifically, for the left and right views acquired by the binocular cameras at the same acquisition time, epipolar correction is first performed based on the camera calibration parameters. Then, the disparity value of each effective pixel is calculated using a stereo matching algorithm. Finally, the two-dimensional pixel coordinates are back-projected into three-dimensional spatial coordinates by combining the focal length of the binocular cameras and the baseline distance. The three-dimensional back-projection relationship is expressed by the following formula:

[0097]

[0098] in, These are the coordinates of the valid pixels in the left view. This represents the disparity value corresponding to that pixel. For the focal length of a binocular camera, The baseline distance of the binocular cameras. , Let these be the coordinates of the camera's principal point. , , The coordinates of the reconstructed three-dimensional points.

[0099] After completing the 3D backprojection, the data processing module removes background points, outliers far from the plant body, and reconstructs noise points based on a preset background color threshold, effective depth range, and local neighborhood statistical characteristics. Subsequently, voxel downsampling is used to sparsify the point cloud. For environmental sensor data, when missing temperature, humidity, or illumination data is detected, linear interpolation is used to fill in the gaps, and max-min normalization is used to map environmental data of different dimensions to a unified numerical range. The interpolation of data at missing moments is calculated using the following formula:

[0100]

[0101] in, When data is missing Estimated sensor values; and For a moment The two most recent valid timestamps of successful data collection; and For the corresponding timestamp and The sensors collect data including temperature, humidity, and light intensity.

[0102] The normalization process is expressed by the following formula:

[0103]

[0104] in, These are the normalized sensor values. This is the original sensor value after interpolation at the current moment. and These are the minimum and maximum values ​​recorded by the corresponding sensor during the sampling period, respectively.

[0105] S2. Graph Construction and Feature Encoding: The graph construction and feature encoding module uses the processed point cloud sampling points as graph nodes. It extracts the 3D coordinates, color components, local neighborhood density, normal vector, or curvature of each node as node features, and establishes edge connections between nodes based on K-nearest neighbors to construct a 3D phenotypic map of the plant. Unlike existing general point cloud graph networks that only use node features for feature aggregation, this invention additionally constructs edge features that include the relative spatial relationships between points. These edge features are simultaneously introduced into the graph attention weight calculation and node feature update process, which can completely preserve geometric details such as the local relative positions of leaves, surface orientation, and canopy layering structure, thus solving the defect of existing models that lose plant spatial adjacency information.

[0106] For the Point cloud sampling points Its node feature vector Composed of spatial location, color, and local geometric descriptors, the node feature vector is represented by the following formula:

[0107]

[0108] in, , , Let i be the three-dimensional coordinates of the i-th point. , , For the corresponding color components, For local neighborhood density, For local normal vectors, This is a quantity describing local curvature.

[0109] To describe the spatial adjacency relationships between points on the surface of plant leaves, the graph construction unit establishes an edge set based on the Euclidean distance between point cloud sampling points. Using the K-nearest neighbor approach, the edge set is represented by the following formula:

[0110]

[0111] in, The set of edges in a three-dimensional phenotypic graph of a plant. Point In three-dimensional space The set of nearest neighbors.

[0112] For any node Pointing to node Edges, edge features Including the relative coordinates, Euclidean distance, and directional relationship between adjacent points, the edge features are represented by the following formula:

[0113]

[0114] Thus, the plant 3D point cloud was converted into a plant 3D phenotypic map. ,in For a set of nodes, Let be the set of edges. The node feature matrix, is the graph connectivity matrix determined by spatial adjacency relationships.

[0115] S3. The graph neural coding unit performs neighborhood aggregation on the plant's 3D phenotypic map, enabling each node to fuse the spatial location, color, and local geometric relationships of its neighboring nodes during feature update, thus obtaining node-level 3D structural features. Subsequently, graph-level pooling is used to obtain the global 3D phenotypic features of the entire plant. The graph neural coding unit uses graph attention aggregation to calculate the attention weight between node i and its neighboring node j, and the attention weight is expressed by the following formula:

[0116]

[0117] in, For nodes For nodes Attention weights For nodes Hidden features in the current graph neural network layer For neighboring nodes Hidden features, As edge features, , , For learnable linear transformation matrices, This is a learnable attention vector.

[0118] node In the The feature update of a layer is represented by the following neighborhood aggregation formula:

[0119]

[0120] in, For nodes In the Layer update characteristics For nodes Features in layer l For nodes The set of neighboring nodes, and For learnable weight matrix, It is a non-linear activation function. After multi-layer aggregation, the node features can simultaneously express the local leaf surface morphology and the large-scale canopy spatial structure.

[0121] The graph neural coding unit further performs graph-level pooling on all node features to obtain the global three-dimensional phenotypic feature vector of the entire plant. The graph-level pooling is expressed by the following formula:

[0122]

[0123] in, This represents the global three-dimensional phenotypic feature vector. For average pooling operation, This represents the number of nodes in the plant's three-dimensional phenotypic map. For the first The node features output by the layered graph neural network.

[0124] Meanwhile, the environmental temporal coding unit reads the standardized temperature, humidity, and light intensity temporal data in chronological order, extracts environmental change trend features, and obtains the environmental temporal feature vector. The environmental temporal coding unit can be implemented using any of the following: a long short-term memory network, a gated recurrent unit, or a temporal convolutional network.

[0125] S4. Modality Fusion: The modality fusion module combines the global three-dimensional phenotypic feature vectors. With environmental time series feature vector By concatenating and performing nonlinear mapping, a joint phenotypic feature vector is obtained. The fusion process is represented by the following formula:

[0126]

[0127] in, and To fuse learnable weights and biases in the mapping, It is a non-linear activation function, and the symbol || represents the feature concatenation operation.

[0128] S5. Task Prediction: The physiological indicator prediction head of the task prediction module receives the joint phenotypic feature vector. As input, leaf area regression prediction branch and relative chlorophyll content (SPAD) regression prediction branch are constructed. The outputs of the two branches are expressed by the following formula:

[0129]

[0130] in, This is the predicted value for leaf area. This is a predicted value for the relative chlorophyll content. and These are the regression mapping functions corresponding to the leaf area prediction branch and the relative chlorophyll content prediction branch, respectively.

[0131] To improve the robustness of regression prediction, the task prediction module uses a smoothed L1 loss function to jointly optimize and train the two prediction branches. The overall loss function is expressed by the following formula:

[0132]

[0133] in, As a result of overall training losses, and Let these be the loss weights for the two prediction tasks. To smooth the L1 loss function, and For predicted values, and For the corresponding real label values, where Determined manually, Measurements were taken using a handheld SPAD device.

[0134] Phenotypic prediction models often use single-task independent losses to train leaf area and SPAD prediction networks separately, which cannot balance the prediction errors of the two types of indicators and is prone to the problem of high accuracy for one indicator and large error for the other. This invention adopts a weighted joint loss to simultaneously constrain the dual regression task, and collaboratively optimizes the two types of physiological indicators to improve the overall robustness of multi-task prediction.

[0135] To quantitatively verify the accuracy advantage of the phenotypic prediction model of this invention compared with existing general point cloud and graph network models, and to verify the synergistic effect of the three core innovative features of this invention, a standardized plant phenotypic dataset was built, and a horizontal comparison experiment and an ablation control experiment were designed. All experimental hardware, data preprocessing, and training hyperparameters were kept completely uniform.

[0136] The experimental dataset was constructed as follows: a three-dimensional phenotypic dataset was built with tulips as the target plant, containing samples of the complete growth cycle of tulips; the dataset contains a total of 28,640 sets of stereo image samples, each set of samples is accompanied by synchronously collected time-series data of temperature, humidity and light environment; the dataset is divided into training set, validation set and test set in a 7:2:1 ratio.

[0137] The samples covered the seedling, growth, and flowering stages, and the environmental parameters covered the entire range of light intensity (0-8000 lux), temperature (10-20℃), and relative humidity (40%-75%). For each sample, the actual chlorophyll SPAD value and total leaf area of ​​the plant were collected simultaneously using a handheld SPAD instrument and manual image segmentation and calibration, which served as the true labels for model regression training and testing.

[0138] The control group was set up as follows: Three mainstream neural networks in the agricultural phenotyping field were selected as control models. Except for the backbone network structure, all configurations, including data acquisition, preprocessing, training epochs, optimizers, and loss weights, were kept consistent.

[0139] Comparative Example 1: PointNet++ general point cloud network, without graph structure or attention weighting mechanism, directly splices environmental temporal features after extracting global point cloud features for prediction;

[0140] Comparative Example 2: DGCNN basic dynamic graph convolutional network, which only relies on node features to complete neighborhood aggregation, without constructing or introducing edge geometric features to participate in attention calculation;

[0141] Comparative Example 3: Standard GAT basic graph attention network, which includes basic node attention, but lacks plant-specific nine-dimensional composite node features. The environment and three-dimensional features are simply stitched together, and there is no non-linear deep fusion layer.

[0142] The model of this invention is a complete model consisting of edge feature fusion graph attention encoding + multi-dimensional geometric node features + temporal-spatial nonlinear deep modality fusion + bi-branch smooth L1 joint loss.

[0143] The evaluation metrics are as follows: This task involves continuous value regression prediction, and general quantitative evaluation metrics commonly used in the regression field are employed. Lower metric values ​​indicate higher model prediction accuracy.

[0144] Leaf area: unit cm², evaluation indicators MAE (mean absolute error) and RMSE (root mean square error).

[0145] Chlorophyll relative content SPAD: dimensionless, evaluation indicators MAE (mean absolute error) and RMSE (root mean square error).

[0146] The average prediction performance of the test set is shown in Table 1.

[0147] Table 1

[0148]

[0149] As can be seen, compared with the closest existing technology, the GAT network, the model of this invention reduces the leaf area MAE by 43.6% and the SPAD index MAE by 49.0%; compared with the general point cloud networks PointNet++ and DGCNN, the error reduction of both indicators exceeds 50%, and the prediction accuracy is significantly improved.

[0150] To verify the necessity of each innovative module, three ablation experiment groups were set up. In each group, only one core innovative structure of this invention was removed, while the remaining network components and training parameters remained completely unchanged. The decline in prediction accuracy was observed to verify that the three innovative features have a synergistic effect and are indispensable.

[0151] Ablation 1: Remove the edge feature fusion attention mechanism and calculate attention weights based solely on node features;

[0152] Ablation 2: The nonlinear deep modal fusion layer is removed, and the 3D structural features and environmental temporal features are directly spliced ​​and fed into the prediction branch;

[0153] Ablation 3: Simplify node features, retain only the 6-dimensional basic features of XYZ 3D coordinates + RGB 3-channel color, and remove local density, normal vector, and curvature geometry.

[0154] The average index results of the ablation experiment test set are shown in Table 2.

[0155] Table 2

[0156]

[0157] It is evident that, regardless of whether MAE or RMSE is used, removing any one of the core innovative structures of this invention leads to a significant and simultaneous increase in the errors of the two prediction indicators: tulip leaf area and SPAD chlorophyll. The model accuracy decreases the most after removing the edge feature attention module, proving that the spatial adjacency geometry of the leaves plays a decisive role in phenotypic prediction. The MAE and RMSE decrease trends are completely consistent, indicating that the three innovative features of this invention not only reduce the conventional average prediction bias but also significantly suppress extreme large errors, thus simultaneously improving the overall prediction stability of the model.

[0158] Through the above-mentioned point cloud map construction, graph neural network encoding, environmental temporal encoding and task regression prediction, this invention can make full use of the plant's three-dimensional spatial structure information to predict phenotypic indicators such as leaf area and relative chlorophyll content without relying on complex three-dimensional growth visualization or three-dimensional structure generation tasks.

[0159] This invention, through the close cooperation of data acquisition device, cloud management platform and phenotypic prediction model, completes a complete technical loop from binocular image and environmental data acquisition, 3D point cloud generation, 3D plant phenotypic map construction to intelligent phenotypic prediction, realizing efficient, non-contact automated monitoring and prediction of plant growth status.

[0160] It should be noted that the parts in this embodiment that are the same as or similar to those in Embodiment 1 can be referred to each other, and will not be repeated in this application.

[0161] Example 3:

[0162] Based on Example 1, Example 3 of the present invention provides a smart plant phenotypic analysis method based on computer vision and Internet of Things technology, including the following steps:

[0163] S1. Data acquisition steps: Acquire multi-view two-dimensional images, left and right views, and environmental time-series data of the plant through the data acquisition device, and upload them to the cloud management platform;

[0164] S2. Point cloud generation and preprocessing steps: Perform stereo correction, parallax calculation and three-dimensional back projection on the left and right views to generate a three-dimensional point cloud of plants, and perform noise reduction, background removal and downsampling on the point cloud; at the same time, perform missing value filling and standardization on the sensor data to obtain standardized environmental time series data.

[0165] S3. Graph Construction and Feature Encoding Steps: The preprocessed 3D point cloud of plants is converted into a 3D phenotypic map of plants containing node features and edge connections. Graph neural networks are used to extract 3D structural features. At the same time, temporal coding networks are used to extract temporal features of environmental temporal data.

[0166] S4. Modal fusion step: Align and fuse the three-dimensional structural features with the environmental temporal features to generate a unified joint feature vector;

[0167] S5. Task prediction step: Based on the unified joint feature vector, the physiological index prediction head outputs the prediction results of the plant's physiological index, including the relative chlorophyll content and leaf area.

[0168] In step S3:

[0169] Point cloud sampling points are used as graph nodes, and at least one of the spatial coordinates, color components, local neighborhood density, normal vector or curvature of the sampling points is used as node features. Edge connections between nodes are established based on proximity relationships, and the edge connections have edge features that include the relative spatial relationship between adjacent nodes.

[0170] The graph neural coding unit aggregates neighborhood features based on the node features and edge connections to obtain three-dimensional structural features;

[0171] The temporal coding network reads the standardized environmental temporal data in chronological order and generates environmental temporal features.

[0172] It should be noted that the method provided in this embodiment is the same as the method of the system provided in embodiment 2. Therefore, the parts that are the same as or similar to those in embodiment 1 can be referred to each other and will not be repeated in this application.

[0173] In summary, this invention utilizes existing mature solutions for both hardware and cloud components, reducing equipment deployment costs. The core phenotypic prediction model employs a custom graph structure designed for plant 3D point clouds, incorporating multi-dimensional geometric features of nodes and edge spatial relationship features. Edge features are used to assist in graph attention weighted aggregation, preserving local leaf morphology and canopy layering. A nonlinear modal fusion layer couples and aligns 3D structural features with environmental temporal features, modeling the correlation between environmental factors and plant phenotypic characteristics. A bi-branch joint loss method is used to collaboratively optimize the regression prediction of leaf area and SPAD phenotypic indicators. Experimental data show that compared to existing basic graph attention networks, the model of this invention reduces the average absolute error of leaf area by 43.6% and the average absolute error of SPAD by 49.0%. This invention enables real-time monitoring and physiological indicator prediction throughout the entire plant growth cycle while maintaining the plant's natural growth state, providing technical support for smart agriculture.

Claims

1. An intelligent plant phenotypic analysis platform based on computer vision and Internet of Things technologies, characterized in that, include: Data acquisition devices, cloud management platform, and phenotypic prediction models: The data acquisition device is used to acquire multi-view two-dimensional images of plants, left and right views, and environmental time-series data. The cloud management platform is communicatively connected to the data acquisition device, and is used to store and display the acquired data, as well as to control the operating parameters of the data acquisition device. The phenotypic prediction model is deployed on a cloud management platform to receive left and right views, multi-view two-dimensional images, and environmental time-series data acquired by the data acquisition device. Based on the left and right views, a three-dimensional point cloud of the plant is generated. The three-dimensional point cloud is constructed into a three-dimensional phenotypic map of the plant containing node features and edge connections. The three-dimensional structural features of the plant are extracted using a graph neural network. At the same time, the time-series features of the environmental time-series data are extracted. After fusing the three-dimensional structural features and the environmental time-series features, the physiological indicators of the plant are predicted and output.

2. The intelligent plant phenotyping platform based on computer vision and Internet of Things technology according to claim 1, characterized in that, The data acquisition device includes: The visual acquisition module is used to simultaneously acquire left and right views and multi-view two-dimensional images of plants from different angles using a binocular camera. The environmental sensing module is used to acquire ambient temperature, humidity, and light intensity data through temperature and humidity sensors and light intensity sensors, and encapsulate them into environmental time-series data. The motion control module is used to control the rotating base that supports the plant to rotate at a set angle and frequency to facilitate multi-view visual acquisition.

3. The intelligent plant phenotyping platform based on computer vision and Internet of Things technology according to claim 2, characterized in that, The cloud management platform includes: An interface communication module is used to receive data and issue commands to the data acquisition device. The cloud storage module is used to uniformly store the collected image data, environmental time-series data, and device operation logs; The visualization and interaction module is used to display sensor data and the output results of the phenotypic prediction model in real time, and supports remote configuration of the acquisition parameters.

4. The intelligent plant phenotyping platform based on computer vision and Internet of Things technology according to claim 3, characterized in that, The phenotypic prediction model includes: The data processing module is used to perform stereo correction, parallax calculation and three-dimensional back projection on the left and right views acquired by the binocular camera to generate a three-dimensional point cloud of plants. The module also performs noise reduction, background removal and downsampling on the three-dimensional point cloud of plants. At the same time, it performs missing value filling and standardization on the acquired sensor data to construct environmental time series data. Graph construction and feature encoding module: used to convert the plant 3D point cloud into a plant 3D phenotypic map containing node features and edge connections, extract plant 3D structural features through graph neural network, and extract temporal features of environmental temporal data through temporal coding network; The modality fusion module is used to align and fuse the three-dimensional structural features with the environmental temporal features to generate a unified joint feature vector; The task prediction module is used to output the prediction results of the plant's future physiological indicators, including the relative chlorophyll content and leaf area, based on the unified joint feature vector and the physiological indicator prediction head.

5. The intelligent plant phenotyping platform based on computer vision and Internet of Things technology according to claim 4, characterized in that, In the data processing module of the phenotypic prediction model: A background filtering algorithm based on color space is used to remove pixels that match the background features in a two-dimensional image, thereby extracting the main image of the plant. Stereo correction is performed on the left and right views captured by the binocular camera, and a 3D point cloud of plants is generated based on parallax calculation and 3D back projection. The plant 3D point cloud is subjected to depth range filtering, outlier removal and downsampling to obtain a set of point cloud sampling points for graph structure construction; When missing sensor data is detected, an interpolation algorithm is used to fill in the missing data by combining valid values ​​from adjacent historical moments. A normalization algorithm is used to perform uniform numerical scaling on the imputed data, transforming it into standardized environmental time series data.

6. The intelligent plant phenotyping platform based on computer vision and Internet of Things technology according to claim 5, characterized in that, In the graph construction and feature encoding module of the phenotypic prediction model: The point cloud graph construction unit uses point cloud sampling points as graph nodes, and uses at least one of the spatial coordinates, color components, local neighborhood density, normal vector, or curvature of the sampling points as node features, and establishes edge connections between nodes based on proximity relationships; the edge connections have edge features, and the edge features include the relative spatial relationship between adjacent nodes; The graph neural coding unit aggregates neighborhood features based on the node features and edge connections to obtain three-dimensional structural features representing the unfolded morphology of plant leaves, canopy structure, and spatial density distribution. By using a time-series coding network to read standardized environmental time-series data in chronological order, the fluctuation and change patterns of environmental parameters over time are captured, and environmental time-series features are generated.

7. The intelligent plant phenotyping platform based on computer vision and Internet of Things technology according to claim 6, characterized in that, In the modal fusion module of the phenotypic prediction model: Based on the timestamp of data acquisition, the three-dimensional structural features corresponding to the same growth moment are aligned with the environmental temporal features, and the two are combined into a unified joint feature vector using feature splicing operation. Multilayer perceptron is used to perform feature weight allocation and nonlinear mapping on the unified joint feature vector to establish an intrinsic correspondence between environmental changes and the three-dimensional spatial morphology of plants.

8. The intelligent plant phenotyping platform based on computer vision and Internet of Things technology according to claim 7, characterized in that, In the task prediction module of the phenotypic prediction model: The physiological index prediction head constructs mutually independent leaf area regression prediction branches and chlorophyll relative content regression prediction branches, and receives the unified joint feature vector output by the modality fusion module as input; The two regression prediction branches use a multilayer perceptron to perform nonlinear mapping and numerical regression calculation on the input joint feature vector, and are optimized and trained through a joint loss function. Finally, they independently map and output the predicted values ​​of leaf area and relative chlorophyll content.

9. A smart plant phenotypic analysis method based on computer vision and Internet of Things technology, applied to the analysis platform as described in any one of claims 1 to 8, characterized in that, Includes the following steps: S1. Data acquisition steps: Acquire multi-view two-dimensional images, left and right views, and environmental time-series data of the plant through the data acquisition device, and upload them to the cloud management platform; S2. Point cloud generation and preprocessing steps: Perform stereo correction, parallax calculation and three-dimensional back projection on the left and right views to generate a three-dimensional point cloud of plants, and perform noise reduction, background removal and downsampling on the point cloud; at the same time, perform missing value filling and standardization on the sensor data to obtain standardized environmental time series data. S3. Graph Construction and Feature Encoding Steps: The preprocessed 3D point cloud of plants is converted into a 3D phenotypic map of plants containing node features and edge connections. Graph neural networks are used to extract 3D structural features. At the same time, temporal coding networks are used to extract temporal features of environmental temporal data. S4. Modal fusion step: Align and fuse the three-dimensional structural features with the environmental temporal features to generate a unified joint feature vector; S5. Task prediction step: Based on the unified joint feature vector, the physiological index prediction head outputs the prediction results of the plant's physiological index, including the relative chlorophyll content and leaf area.

10. The intelligent plant phenotypic analysis method based on computer vision and Internet of Things technology according to claim 9, characterized in that, In step S3: Point cloud sampling points are used as graph nodes, and at least one of the spatial coordinates, color components, local neighborhood density, normal vector or curvature of the sampling points is used as node features. Edge connections between nodes are established based on proximity relationships, and the edge connections have edge features that include the relative spatial relationship between adjacent nodes. The graph neural coding unit aggregates neighborhood features based on the node features and edge connections to obtain three-dimensional structural features; The temporal coding network reads the standardized environmental temporal data in chronological order and generates environmental temporal features.

Images