Deep Learning-Based Intelligent Greenhouse Monitoring System

Abstract

The present invention belongs to the technical field of greenhouse monitoring, and discloses an intelligent greenhouse monitoring system based on deep learning. The intelligent greenhouse monitoring system based on deep learning provided by the present invention includes: a data acquisition module, a data preprocessing module, a main control module, a prediction module, a three-dimensional reconstruction module, a measurement module, and an analysis module. The present invention will adopt a method combining SFM and deep learning MVSNet, with the reconstruction of the three-dimensional model of plants or crops as the core. Through preprocessing such as downsampling, segmentation, and noise reduction of the three-dimensional point cloud of plants or crops, their phenotypic characteristics are extracted, so as to accurately obtain the phenotypic parameters of plants or crops. This method can more accurately and efficiently obtain the phenotypic parameters of plants or crops, provide a reliable reference for data processing methods in the fields of botany, agronomy, etc., help to accelerate the breeding process of plants or crops, optimize the growth environment, and promote the development of agricultural production and improve production efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of greenhouse monitoring, and particularly relates to an intelligent greenhouse monitoring system based on deep learning. Background Art

[0002] Research on the prediction of environmental factors in greenhouse at home and abroad shows that traditional prediction models perform well in short-term prediction, but have limitations in long-term prediction and non-linear sequence analysis. Methods such as artificial neural network (ANN), radial basis function neural network (RBF), and long short-term memory neural network (LSTM) have achieved certain results in improving prediction accuracy, but still have problems such as sensitivity to initial weights and high training complexity. In recent years, the combination of neural network and traditional methods has significantly improved the prediction accuracy.

[0003] In the aspect of crop phenotype measurement based on three-dimensional data, technologies such as binocular stereo vision, structured light, time-of-flight technology (ToF), lidar, and multi-view stereo vision are widely used. Binocular stereo vision depends on the light environment and the texture of the object surface, and it is difficult to accurately measure the plant phenotype parameters in complex environments. Structured light technology is easily affected by ambient light, and the imaging distance is limited. ToF technology has high cost and is easily affected by noise. Although lidar technology has accurate measurement and high resolution, the equipment price is expensive. The three-dimensional reconstruction technology of multi-view stereo vision has high accuracy and is non-destructive, but there are problems of long reconstruction time and difficult geometric calibration. Summary of the Invention

[0004] Aiming at the problems existing in the prior art, the present invention provides an intelligent greenhouse monitoring system based on deep learning.

[0005] The present invention is implemented as follows. An intelligent greenhouse monitoring system based on deep learning includes:

[0006] A data acquisition module, a data preprocessing module, a main control module, a prediction module, a three-dimensional reconstruction module, a measurement module, and an analysis module;

[0007] The data acquisition module, connected to the data preprocessing module, is used for acquiring the data of the crop growth environment inside the greenhouse;

[0008] The data preprocessing module, connected to the data acquisition module and the main control module, is used for preprocessing the acquired environmental data containing noise;

[0009] The main control module, connected to the data preprocessing module, the prediction module, the three-dimensional reconstruction module, the measurement module, and the analysis module, is used for controlling the normal operation of each module;

[0010] The prediction module, connected to the main control module, is used for performing short-term and medium-term prediction on the acquired environmental factor data through the WCACM combined neural network prediction model;

[0011] A 3D reconstruction module, connected to the main control module, for 3D reconstructing plants or crops in the greenhouse through the SfM-MVSNet neural network 3D reconstruction model;

[0012] A measurement module, connected to the main control module, for measuring and extracting plant phenotype parameter information;

[0013] An analysis module, connected to the main control module, for performing requirement analysis on the monitoring system and designing the system functions and architecture.

[0014] Furthermore, the method of the data preprocessing module:

[0015] Perform redundant value processing, outlier removal, missing value filling, and data standardization operations on the environmental data containing noise.

[0016] Furthermore, the method of the prediction module:

[0017] Utilize the WCACM combined neural network prediction model. First, process the data through wavelet packet denoising and complete ensemble empirical mode decomposition. Then, use the AutoFormer neural network model for short- and medium-term prediction. Finally, perform training and testing on the greenhouse environmental factor dataset.

[0018] Furthermore, the method of the 3D reconstruction module:

[0019] First, through multi-view image acquisition, use the Structure from Motion (SfM) method of incremental motion recovery to estimate the internal and external parameters of the camera. Then, use the MVSNet deep learning framework to construct the 3D point cloud of the plant. Finally, perform tests on the DTU public dataset and the self-made plant dataset.

[0020] Furthermore, the method of the measurement module:

[0021] First, preprocess the 3D point cloud data of the plant, including point cloud downsampling, coordinate correction, feature segmentation, and denoising operations. Then, use the axis-aligned bounding box method to calculate the plant height and crown width parameters, and use the triangular meshing method to calculate the plant surface area parameter. Finally, select 20 plants for an artificial measurement and algorithm measurement comparison experiment.

[0022] Furthermore, the method of the analysis module:

[0023] First, conduct a requirements analysis of the monitoring system and design the corresponding system functions and architecture; secondly, build the front and back ends of the system based on the Bootstrap and Flask frameworks; then, deploy the deep learning algorithm into the system to build a complete greenhouse monitoring system covering functions such as real-time data monitoring, environmental factor prediction, and measurement of plant three-dimensional model phenotypic parameters; finally, conduct functional tests and verification on the various functions of the system.

[0024] Another object of the present invention is to provide a smart greenhouse monitoring method based on deep learning, including:

[0025] Step 1, collect the crop growth environment data inside the greenhouse through the data collection module; preprocess the collected environmental data containing noise through the data preprocessing module.

[0026] Step 2, the main control module uses the WCACM combined neural network prediction model through the prediction module to perform short-term and medium-term predictions on the obtained environmental factor data.

[0027] Step 3, use the SfM-MVSNet neural network three-dimensional reconstruction model through the three-dimensional reconstruction module to perform three-dimensional reconstruction on the plants or crops in the greenhouse.

[0028] Step 4, measure and extract the plant phenotypic parameter information through the measurement module.

[0029] Step 5, conduct a requirements analysis on the monitoring system through the analysis module and design the system functions and architecture.

[0030] Another object of the present invention is to provide a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning.

[0031] Another object of the present invention is to provide a computer-readable storage medium, storing a computer program, and when the computer program is executed by a processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning.

[0032] Another object of the present invention is to provide an information data processing terminal, and the information data processing terminal is used to implement the smart greenhouse monitoring system based on deep learning.

[0033] Combined with the above technical solutions and the solved technical problems, the advantages and positive effects of the technical solution to be protected by the present invention are:

[0034] First, based on the existing 3D reconstruction methods, the present invention will adopt the method of combining SFM and deep learning MVSNet, with the core of reconstructing the 3D model of plants or crops. Through preprocessing such as downsampling, segmentation, and noise reduction of the 3D point cloud of plants or crops, and embedding it into the agricultural Internet of Things system, the phenotypic characteristics can be automatically extracted, so as to accurately obtain the phenotypic parameters of plants or crops, changing the traditional monitoring method of 3D reconstruction of plant or crop phenotypic parameters, and improving the traditional method of 3D reconstruction of plants or crops by using deep learning models. This method can obtain the phenotypic parameters of plants or crops more accurately, efficiently, and automatically, providing a reliable reference for data processing methods in the fields of botany, agronomy, etc., helping to accelerate the breeding process of plants or crops, optimize the growth environment, and promote the development of agricultural production and improve production efficiency.

[0035] First, the present invention uses a collection system to obtain the environmental data in the greenhouse and cleans and normalizes the data. Secondly, the present invention proposes a WCACM combined neural network prediction model. First, the data is processed by wavelet packet denoising and complete ensemble empirical mode decomposition, and then the AutoFormer neural network model is used for short- and medium-term prediction. Finally, it is trained and tested on the greenhouse environmental factor dataset. Qualitative and quantitative results show that this method outperforms other prediction models based on TransFormer in terms of performance.

[0036] The present invention proposes a 3D reconstruction method of plants based on SfM-MVSNet and multi-view images, that is, dense 3D reconstruction of plants is realized by combining the SfM algorithm and the deep learning algorithm MVSNet. First, through multi-view image acquisition, the incremental structure from motion method (SfM) is used to estimate the internal and external parameters of the camera. Then, the MVSNet deep learning framework is used to construct the 3D point cloud of plants. Finally, this method is tested on the DTU public dataset and the self-made plant dataset. Experimental results show that this method is superior to other advanced methods and traditional methods in terms of plant reconstruction quality and reconstruction speed.

[0037] The present invention proposes a 3D point cloud phenotypic measurement method for plants. First, preprocessing is performed on the 3D point cloud data of plants, and this method includes operations such as point cloud downsampling, coordinate correction, feature segmentation, and denoising. Then, the axis-aligned bounding box method is used to calculate the plant height and crown width parameters, and the triangular meshing method is used to calculate the plant surface area parameter. Finally, 20 plants are selected for a comparative experiment of manual measurement and algorithm measurement. Experimental results show that this method can quickly and accurately extract the phenotypic parameter information of plants.

[0038] First, the present invention conducts a requirements analysis on the monitoring system and designs corresponding system functions and architectures. Secondly, the present invention realizes the front-end and back-end construction of the system based on the Bootstrap and Flask frameworks. Then, combining the above research content, deep learning algorithms are deployed into the system to construct a complete greenhouse monitoring system covering functions such as real-time data monitoring, environmental factor prediction, and measurement of phenotypic parameters of plant three-dimensional models. Finally, functional tests and verifications are carried out on various functions of the system, and the results show that the system can be further deployed and applied to the agricultural Internet of Things intelligent monitoring system.

[0039] The present invention mainly studies the prediction model of environmental factors in the greenhouse based on the WCACM neural network, the method of plant three-dimensional reconstruction based on SfM-MVSNet and multi-view images, as well as the research on the method of measuring plant phenotypic parameters based on three-dimensional point clouds and the design and implementation of the greenhouse monitoring system. The WCACM neural network prediction model and the method of measuring plant three-dimensional point cloud phenotypes reconstructed based on the SfM-MVSNet method are integrated into the greenhouse monitoring system to realize a multi-functional greenhouse monitoring system to meet the needs of modern agricultural development. The main research work includes the following aspects:

[0040] (1) First, the acquisition system is used to collect the internal environmental data of the greenhouse. Then, data cleaning and preprocessing are carried out on the collected data to ensure the quality, understandability, and comparability of the data.

[0041] (2) A WCACM combined neural network prediction model is built by combining wavelet packet denoising, complete ensemble empirical mode decomposition, and AutoFormer, and this model is used to make short- and medium-term predictions on the collected data. The experimental results show that this model performs better than the standard TransFormer, LogTrans, InFormer, and the AutoFormer network model used alone in the predictions for the next 24, 96, 192, and 336 hours.

[0042] (3) High-precision and dense three-dimensional reconstruction of plants is realized. First, the present invention uses the SfM algorithm to recover the internal and external parameters of the camera and the sparse three-dimensional point cloud data of the plant. Next, combined with the deep learning algorithm MVSNet, by generating masks and depth maps and fusing them with paired information, the dense three-dimensional reconstruction of the plant is realized. The experimental results show that the framework proposed by the present invention can generate high-quality plant three-dimensional models quickly and efficiently compared with other methods.

[0043] (4) Design a method for extracting plant phenotypic parameters based on the three-dimensional model. First, preprocess the plant three-dimensional point cloud model, including operations such as point cloud downsampling, coordinate correction, feature segmentation, and denoising. Then, use the axis-aligned bounding box method to calculate the plant height and crown width, and use the triangular meshing method to calculate the plant surface area parameter.

[0044] (5) Conduct a requirements analysis on the monitoring system, design the system functions and architecture, and construct a complete monitoring system covering functions such as data monitoring, environmental factor prediction, three-dimensional reconstruction of plants, and measurement of phenotypic parameters, in combination with the above research content.

[0045] Second, the expected benefits and commercial value after the transformation of the technical solution of the present invention are as follows: For the whole country, the cost savings and output increase are at the level of tens of millions of yuan. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a structural block diagram of the intelligent greenhouse monitoring system based on deep learning provided by an embodiment of the present invention.

[0047] Figure 2 It is a flowchart of the intelligent greenhouse monitoring method based on deep learning provided by an embodiment of the present invention.

[0048] Figure 3 It is a technical roadmap provided by an embodiment of the present invention.

[0049] Figure 4 It is a diagram of the internal environment data transmission path of the greenhouse provided by an embodiment of the present invention.

[0050] Figure 5 It is a flowchart of data cleaning provided by an embodiment of the present invention.

[0051] Figure 6 It is a filling principle diagram provided by an embodiment of the present invention.

[0052] Figure 7 It is a basic structural diagram of the BP neural network provided by an embodiment of the present invention.

[0053] Figure 8 It is a basic structural diagram of the RNN neural network provided by an embodiment of the present invention.

[0054] Figure 9 It is a diagram of the LSTM neural network unit provided by an embodiment of the present invention.

[0055] Figure 10 It is a basic structural diagram of the TransFormer model provided by an embodiment of the present invention.

[0056] Figure 11 It is a basic end-to-end structural diagram provided by an embodiment of the present invention.

[0057] Figure 12 It is an implementation diagram of the Self-Attention mechanism provided by an embodiment of the present invention.

[0058] Figure 13It is the structure diagram of the AutoFormer model provided by the embodiments of the present invention.

[0059] Figure 14 It is the diagram of autocorrelation (left) and temporal delay aggregation (right) provided by the embodiments of the present invention.

[0060] Figure 15 It is the diagram of autocorrelation attention and self-attention series provided by the embodiments of the present invention.

[0061] Figure 16 It is the structure diagram of the WCACM prediction model provided by the embodiments of the present invention.

[0062] Figure 17 It is the diagram of the experimental results of the 24-hour prediction of air temperature by different models provided by the embodiments of the present invention.

[0063] Figure 18 It is the diagram of the experimental results of the 24-hour prediction of light intensity by different models provided by the embodiments of the present invention.

[0064] Figure 19 It is the diagram of the experimental results of the 24-hour prediction of soil humidity by different models provided by the embodiments of the present invention.

[0065] Figure 20 It is the diagram of the experimental results of the 96-hour prediction of air temperature by different models provided by the embodiments of the present invention.

[0066] Figure 21 It is the diagram of the experimental results of the 96-hour prediction of light intensity by different models provided by the embodiments of the present invention.

[0067] Figure 22 It is the diagram of the experimental results of the 96-hour prediction of soil humidity by different models provided by the embodiments of the present invention.

[0068] Figure 23 It is the diagram of the experimental results of the 192-hour prediction of air temperature by different models provided by the embodiments of the present invention.

[0069] Figure 24 It is the diagram of the experimental results of the 192-hour prediction of light intensity by different models provided by the embodiments of the present invention.

[0070] Figure 25 It is the diagram of the experimental results of the 192-hour prediction of soil humidity by different models provided by the embodiments of the present invention.

[0071] Figure 26 It is the diagram of the experimental results of the 336-hour prediction of air temperature by different models provided by the embodiments of the present invention.

[0072] Figure 27It is a graph showing the 336-hour prediction experimental results of different models for light intensity provided by an embodiment of the present invention.

[0073] Figure 28 It is a graph showing the 336-hour prediction experimental results of different models for soil humidity provided by an embodiment of the present invention.

[0074] Figure 29 It is an example graph of the DTU dataset provided by an embodiment of the present invention.

[0075] Figure 30 It is an example graph of the self-collected plant dataset provided by an embodiment of the present invention.

[0076] Figure 31 It is a flowchart of the SfM-MVSNet 3D reconstruction provided by an embodiment of the present invention.

[0077] Figure 32 It is a framework diagram of the incremental SfM algorithm provided by an embodiment of the present invention.

[0078] Figure 33 It is a network framework diagram of the MVSNet provided by an embodiment of the present invention.

[0079] Figure 34 It is a graph of depth map fusion provided by an embodiment of the present invention.

[0080] Figure 35 It is an example graph of the test results provided by an embodiment of the present invention. (a) Comparison of the 3D reconstruction quality of Scan1 (b) Comparison of the 3D reconstruction quality of Scan13 (c) Comparison of the 3D reconstruction quality of Scan32 (d) Comparison of the 3D reconstruction quality of Scan77.

[0081] Figure 36 It is a graph of the experimental results of typical image 3D reconstruction provided by an embodiment of the present invention. (a) Comparison of the 3D reconstruction quality of the plant Epipremnum aureum (b) Comparison of the 3D reconstruction quality of the plant Rohdea japonica.

[0082] Figure 37 It is a graph of the experimental results of self-collected image 3D reconstruction provided by an embodiment of the present invention.

[0083] Figure 38 It is a framework diagram of plant phenotype measurement provided by an embodiment of the present invention. (a) Original point cloud of the plant (b) Downsampled point cloud of the plant.

[0084] Figure 39 It is a graph of plant point cloud downsampling provided by an embodiment of the present invention. (a) Point cloud before coordinate transformation (b) Point cloud after coordinate transformation.

[0085] Figure 40 It is a graph of point cloud coordinate correction provided by an embodiment of the present invention. (a) Before point cloud segmentation (b) After point cloud segmentation.

[0086] Figure 41 It is the plant point cloud segmentation map provided by the embodiment of the present invention. (a) Point cloud before denoising (b) Point cloud after denoising.

[0087] Figure 42 It is the point cloud denoising map provided by the embodiment of the present invention.

[0088] Figure 43 It is the plant height and crown width extraction map provided by the embodiment of the present invention. (a) Plant point cloud (b) Point cloud triangular meshing.

[0089] Figure 44 It is the point cloud triangular meshing map provided by the embodiment of the present invention.

[0090] Figure 45 It is the plant grid detail map provided by the embodiment of the present invention.

[0091] Figure 46 It is the plant height measurement comparison scatter plot provided by the embodiment of the present invention.

[0092] Figure 47 It is the surface area measurement comparison scatter plot provided by the embodiment of the present invention.

[0093] Figure 48 It is the system function design framework diagram provided by the embodiment of the present invention.

[0094] Figure 49 It is the system architecture design diagram provided by the embodiment of the present invention.

[0095] Figure 50 It is the system technical route diagram provided by the embodiment of the present invention. (a) User login page (b) System registration page.

[0096] In the figure: 1. Data acquisition module; 2. Data preprocessing module; 3. Main control module; 4. Prediction module; 5. 3D reconstruction module; 6. Measurement module; 7. Analysis module. Detailed implementation manners

[0097] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0098] As Figure 1 shown, a smart greenhouse monitoring system based on deep learning provided by the embodiment of the present invention includes:

[0099] Data acquisition module 1, data preprocessing module 2, main control module 3, prediction module 4, 3D reconstruction module 5, measurement module 6, analysis module 7.

[0100] The data acquisition module 1, connected to the data preprocessing module 2, is used to acquire the data of the crop growth environment inside the greenhouse;

[0101] The data preprocessing module 2, connected to the data acquisition module 1 and the main control module 3, is used to preprocess the acquired environmental data containing noise;

[0102] The main control module 3, connected to the data preprocessing module 2, the prediction module 4, the 3D reconstruction module 5, the measurement module 6, and the analysis module 7, is used to control the normal operation of each module;

[0103] The prediction module 4, connected to the main control module 3, is used to perform short - and medium - term predictions on the acquired environmental factor data through the WCACM combined neural network prediction model;

[0104] The 3D reconstruction module 5, connected to the main control module 3, is used to perform 3D reconstruction on the plants or crops in the greenhouse through the SfM - MVSNet neural network 3D reconstruction model;

[0105] The measurement module 6, connected to the main control module 3, is used to measure and extract the phenotypic parameter information of plants;

[0106] The analysis module 7, connected to the main control module 3, is used to perform requirement analysis on the monitoring system and design the system functions and architecture.

[0107] The method of the data preprocessing module provided by the embodiment of the present invention:

[0108] Perform redundant value processing, outlier removal, missing value filling, and data standardization operations on the environmental data containing noise.

[0109] The method of the prediction module provided by the embodiment of the present invention:

[0110] Utilize the WCACM combined neural network prediction model. First, process the data through wavelet packet denoising and complete ensemble empirical mode decomposition. Then, use the AutoFormer neural network model for short - and medium - term predictions. Finally, perform training and testing on the greenhouse environmental factor dataset.

[0111] The method of the 3D reconstruction module provided by the embodiment of the present invention:

[0112] First, through multi - view image acquisition, use the structure - from - motion (SfM) method of incremental motion recovery to estimate the internal and external parameters of the camera. Then, use the MVSNet deep learning framework to construct the 3D point cloud of the plant. Finally, perform tests on the DTU public dataset and the self - made plant dataset.

[0113] The method of the measurement module provided by the embodiment of the present invention:

[0114] First, preprocess the three-dimensional point cloud data of plants, including point cloud downsampling, coordinate correction, feature segmentation, and denoising operations; then, use the axis-aligned bounding box method to calculate the plant height and crown width parameters, and use the triangular meshing method to calculate the plant surface area parameters; finally, select 20 plants for artificial measurement and algorithm measurement comparison experiments.

[0115] The analysis module method provided by the embodiments of the present invention:

[0116] First, perform a requirements analysis on the monitoring system and design the corresponding system functions and architectures; secondly, implement the front-end and back-end construction of the system based on the Bootstrap and Flask frameworks; then, deploy the deep learning algorithm into the system to build a complete greenhouse monitoring system covering functions such as real-time data monitoring, environmental factor prediction, and measurement of plant three-dimensional model phenotypic parameters; finally, conduct functional tests and verifications on the various functions of the system.

[0117] As Figure 2 shown, a smart greenhouse monitoring method based on deep learning provided by the embodiments of the present invention includes:

[0118] S101, collect the growth environment data of the crops inside the greenhouse through the data collection module; preprocess the collected environmental data containing noise through the data preprocessing module;

[0119] S102, the main control module uses the WCACM combined neural network prediction model through the prediction module to perform short-term and medium-term predictions on the obtained environmental factor data;

[0120] S103, use the SfM-MVSNet neural network three-dimensional reconstruction model through the three-dimensional reconstruction module to perform three-dimensional reconstruction on the plants or crops in the greenhouse;

[0121] S104, measure and extract the plant phenotypic parameter information through the measurement module;

[0122] S105, perform a requirements analysis on the monitoring system through the analysis module and design the system functions and architectures.

[0123] Another object of the present invention is to provide a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning.

[0124] Another object of the present invention is to provide a computer-readable storage medium, storing a computer program, and when the computer program is executed by a processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning.

[0125] Another object of the present invention is to provide an information data processing terminal for implementing the intelligent greenhouse monitoring system based on deep learning.

[0126] The main research contents of the present invention are divided into the following five parts:

[0127] (1) Data collection and preprocessing. An environmental factor data collection system was built using terminal devices such as sensors, monitoring stations, and gateways produced by Shenzhen Xinli Technology Co., Ltd. to collect the crop growth environment data inside the greenhouse. And for the environmental data with noise obtained by the collection system, preprocessing was performed on it, mainly including redundant value processing, outlier removal, missing value filling, and data standardization operations.

[0128] (2) Greenhouse environmental factor prediction. A WCACM combined neural network prediction model was proposed, and this model was used to perform short - and medium - term predictions on the obtained environmental factor data.

[0129] (3) Three - dimensional reconstruction of plants or crops. An SfM - MVSNet neural network three - dimensional reconstruction model was proposed, and this model was used to perform three - dimensional reconstruction on plants or crops in the greenhouse.

[0130] (4) Measurement of plant or crop phenotypic parameters. A method for extracting plant phenotypic parameters based on a three - dimensional model was designed. First, preprocessing was performed on the three - dimensional point cloud model of plants or crops, including operations such as point cloud downsampling, coordinate correction, feature segmentation, denoising, and triangulation. Then, according to the preprocessed model, parameters such as the plant height, crown width, and surface area of plants or crops were calculated.

[0131] (5) Design and implementation of the intelligent greenhouse monitoring system. Requirement analysis was carried out on the monitoring system, and the system functions and architecture were designed. Combining the foregoing research contents, a complete monitoring system covering functions such as data monitoring, environmental factor prediction, and measurement of phenotypic parameters of three - dimensional models of plants or crops was constructed. Figure 3 Technical roadmap

[0132] 2. Data collection and preprocessing

[0133] 2.1 Data collection

[0134] Data acquisition and preprocessing are indispensable steps in data science and machine learning. By ensuring data quality, eliminating biases, improving generalization ability, reducing computational costs, and improving features, reliable and efficient training data are provided for the model, thereby enhancing the model performance and interpretability.

[0135] 2.1.1 Principle of collecting internal environment data of the Internet of Things greenhouse

[0136] The data collection process is as Figure 4 shown.

[0137] 2.1.2 Introduction to the Lower Computer Data Acquisition System

[0138] (1) Gateway Device

[0139] The gateway used in this experimental acquisition system is XL90, and the XL90 intelligent gateway is the core of the intelligent sensing network.

[0140] (2) Monitoring Station Equipment

[0141] The XL68 environmental monitoring terminal also provides a variety of wireless radio frequency module options, including 433MHz, LoRa, 2.4G, etc.

[0142] (3) Wireless Communication Module

[0143] LoRa is selected as the method of wireless communication.

[0144] 2.2 Data Preprocessing

[0145] The main purpose of data preprocessing is to optimize the original data to meet the requirements of the prediction model and improve the performance of the model. It usually includes multiple aspects such as data cleaning, data integration, data transformation, and data reduction, aiming to improve the accuracy, stability, and generalization ability of the prediction model and ensure that the research results are more credible and reliable.

[0146] 2.2.1 Data Cleaning

[0147] The data of greenhouse environmental factors collected will be cleaned through three steps: redundant value processing, outlier removal, and missing value filling. The example process of data cleaning is as shown above Figure 5 as shown.

[0148] The introduction of redundant value processing, outlier removal, and missing value filling in data preprocessing is as follows:

[0149] (1) Redundant Value Processing: Redundant values refer to data that appears multiple times in the dataset and has the same or highly similar values in different entries.

[0150] (2) Outlier Removal: Outliers refer to values that deviate significantly from other data in the dataset.

[0151] The following are the detailed steps to identify outliers using MAD:

[0152] First, calculate the median of a dataset X of a certain environmental characteristic. Here, the Median function is used to calculate the median, which is expressed as Equation (2-1).

[0153] median = Median(X) (2-1)

[0154] Secondly, for each data point x in the dataseti Calculate its absolute deviation (absolutedeviationi) from the median, expressed as Equation (2-2).

[0155] absolute deviation i = |x i - median| (2-2)

[0156] Through the above calculations, the set of absolute deviations (Absolute Deviation) is obtained. Then, the median of all absolute deviation values is calculated to obtain the median absolute deviation MAD, expressed as Equation (2-3).

[0157] MAD = Median(Absolute Deviation) (2-3)

[0158] Finally, a threshold is set to judge outliers. In the experiment, if the absolute deviation of a certain data point exceeds three times the median MAD, it is regarded as an outlier.

[0159] (3) Missing value filling: Missing value filling is a method of estimating missing data points by using the relationships between known data points. Usually, linear interpolation is used to fill the missing values in the dataset. The only problem existing in the dataset after the two-step processing of (1) and (2) is data gaps. Therefore, the present invention uses linear interpolation to fill the gaps, and the algorithm principle adopted is as follows:

[0160] If there are two missing data to be filled between two known data points C and D, and the value of C is less than that of D, the original data is as Figure 6 shown.

[0161] If points C and D are marked on a one-dimensional number line and two new data points are inserted between them so that the interval [C, D] is divided into three segments, and the length of each segment is portion, expressed as Equation (2-4).

[0162]

[0163] Then, the value data1 is inserted at position 2, expressed as Equation (2-5).

[0164] data1 = C + portion * 1 (2-5)

[0165] The value data2 is inserted at position 3, expressed as Equation (2-6).

[0166] data2 = C + portion * 2 (2-6)

[0167] According to the previous setting, considering the case where the left endpoint value of the missing data is C, the right endpoint value is D, and the number of missing data in the middle is n. The following formula can be generalized: The interval formed by points C and D is equally divided into n + 1 segments, and the length of each segment is represented by portion, as shown in Equation (2-7).

[0168]

[0169] Then the value data k inserted at the k-th missing position is expressed as Equation (2-8).

[0170] data k = C + portion * k (2-8)

[0171] 2.2.2 Data normalization

[0172] Data normalization methods include: Min-Max normalization, linear ratio transformation method, and Z-Score standardization. All three methods can eliminate the influence of the dimension and order of magnitude of the data on subsequent analysis. The specific calculation methods are as follows:

[0173] Min-Max normalization: As shown in Equation (2-9).

[0174]

[0175] Linear ratio transformation method: Its core idea is to scale the data proportionally using a linear transformation. However, its disadvantage is that it is sensitive to extreme values and may cause the distribution of the data to shift, as shown in Equation (2-10).

[0176]

[0177] Z-Score standardization: The core of this method is to map the original data to a standard normal distribution with a mean of 0 and a standard deviation of 1 for data transformation, which helps to eliminate the scale differences between different features and makes the importance of each variable approach the same after transformation, as shown in Equation (2-11).

[0178]

[0179] In Equations (2-9), (2-10), and (2-11), x i represents the i-th data in the dataset, min(x) is the minimum value of this group of data, max(x) is the maximum value of this group of data, mean(x) is the average value of this group of data, δ is the standard deviation of this group of data, and yi is the result of this group of data after different normalizations.

[0180] The principle of wavelet packet denoising is as follows:

[0181] (1) Wavelet packet decomposition: Wavelet packet decomposition uses wavelet packet transform to decompose the original signal into sub-signals of different scales and frequency bands. Similar to continuous wavelet transform and discrete wavelet transform, wavelet packet transform is more flexible and can decompose the signal in more detail. Let \(x(t)\) be the original signal, \(\Psi j.k (t)\) be the wavelet packet basis function, and \(d j,k \) be the wavelet packet coefficients after decomposition, where \(j\) represents the scale (level) and \(k\) represents the frequency band at that scale. The decomposition is shown in Equation (3-1).

[0182] d j,k =\(\int x(t)\cdot\psi j,k (t)dt\) (3-1)

[0183] (2) Threshold processing: Perform threshold processing on a series of wavelet packet coefficients \(d j,k \) obtained from the decomposition to reduce or set to zero the noise in the wavelet packet coefficients. Threshold processing is usually divided into hard threshold processing and soft threshold processing. In hard threshold processing, smaller coefficients are set to zero while larger coefficients remain unchanged. In soft threshold processing, smaller coefficients are shrunk to zero while larger coefficients are reduced. The goal of this step is to reduce or set to zero the noise coefficients \(d\). Soft threshold processing is shown in Equation (3-2), and hard threshold processing is shown in Equation (3-3).

[0184]

[0185] D j,k =\(\text{sign}(d j,k )\cdot\max(|d j,k |-T,0)\) (3-3)

[0186] where \(D j,k \) are the wavelet packet coefficients after threshold processing, \(T\) is the threshold, \(\text{sign}(d j,k )\) is the sign function of \(d j,k \), that is, 1 for positive numbers, -1 for negative numbers, and 0 for zero.

[0187] (3) Signal reconstruction: Perform inverse wavelet packet transform on the sub-signals after threshold processing to obtain the denoised signal \(x denoised (t)\). The inverse wavelet packet transform can be obtained by iteratively combining the sub-signals, as shown in Equation (3-4).

[0188]

[0189] The steps of Complete Ensemble Empirical Mode Decomposition (CEEMD) are as follows:

[0190] The specific decomposition process is as follows:

[0191] ① In each iteration, a set of positive noise sequences with a normal distribution and negative noise sequences are added to the original signal X(t). and negative noise sequences The calculation formulas are shown in Equations (3-5) and (3-6).

[0192]

[0193] ② Perform empirical mode decomposition on and respectively to obtain the decomposition results of two sets of IMFs + and IMFs - The calculation formulas are shown in Equations (3-7) and (3-8).

[0194]

[0195] ③ Take the mean IMF of IMFs + and IMFs - as the decomposition result of the complete ensemble empirical mode decomposition. The calculation formula is shown in Equation (3-9).

[0196] IMF = AVG(IMFs + + IMFs - ) (3-9)

[0197] The basic structure of the BP neural network is as follows Figure 7 shown, where each node represents a neuron.

[0198] The structure of the LSTM neural network unit is as Figure 9 shown.

[0199] The LSTM neural network unit replaces the hidden unit in the general recurrent network to form the basic structure of the RNN neural network as shown in Section 3.2.2 Figure 8 shown, where the LSTM units are interconnected with each other through cyclic links.

[0200] 3.2.4 The neural network based on TransFormer is as Figure 10

[0201] To solve the inherent problems of traditional end-to-end models, the TransFormer model introduces a series of innovative structures and mechanisms. These include the following key innovation points. The traditional end-to-end model consists of two parts: an Encoder and a Decoder, such as the end-to-end LSTM model based on RNN. The Encoder, as the encoder, can encode the input into a semantic vector; the Decoder, as the decoder, decodes and outputs the input passed by the Encoder. The model structure is as Figure 11 shown.

[0202] (1) Self-Attention mechanism:

[0203] The Self-Attention mechanism allows the model to focus on different parts of the input sequence at each time step, solves the problem that a fixed-length semantic vector cannot fully carry information, and improves the long-term memory ability. The implementation method of the Self-Attention mechanism is as Figure 12 shown.

[0204] The self-attention mechanism calculates the attention scores through the weight matrices of Query (Q), Key (K), and Value (V). First, the query vector Q is dot-multiplied with the key vector K, and then divided by a constant term to stabilize the calculation. Then, the normalized attention weights are calculated through the Softmax function. Finally, the normalized weights are multiplied by the value matrix V to obtain the final self-attention output. This process allows the model to weight different parts of the input in order to capture important information. The specific formula is expressed as Equation (3-8).

[0205]

[0206] where Q, K, and V are matrices composed of query vectors, key vectors, and value vectors respectively, and d k is the dimensionality of the input vector.

[0207] (2) Multi-headed Attention:

[0208] In the Transformer model, usually 8 attention heads are adopted, and each head has its own set of weight matrices W i , which are multiplied by the input vector and then merged. Finally, the merged attention representation is sent to the feed-forward layer for the next calculation. The calculation formula of multi-headed attention is expressed as Equations (3-9) and (3-10).

[0209]

[0210] MultiHead(Q, K, V) = Concat(head1, …, head h )W o (3-10)

[0211] where, and are the parameter matrices of the Q, K, and V matrices, and W 0 is the additional matrix.

[0212] (3) Positional Encoding

[0213] Use the sine and cosine functions to generate positional encodings, and the calculation formulas are shown in Equations (3-11) and (3-12).

[0214]

[0215] where pos is the position in the sequence, i is the dimension index of the positional encoding, and d model is the dimension of the model embedding vector.

[0216] (4) Feed-Forward Neural Network and Layer Normalization

[0217] The Feed-Forward Neural Network (FFN) aims to provide non-linear transformations to enhance the model's expressive power, while layer normalization aims to normalize the input of each layer to accelerate training and improve the model's robustness, as shown in Equation (3-13).

[0218] FFN = max(0, xW1 + b1)W2 + b2 (3-13)

[0219] The Transformer model adopts the layer normalization method, and the calculation formula is shown in Equation (3-14).

[0220]

[0221] 3.3 AutoFormer Neural Network

[0222] 3.3.1 AutoFormer Model Structure

[0223] The AutoFormer model structure is as Figure 13 shown.

[0224] (1) Series Decomposition Block

[0225] The series decomposition block smooths the periodic fluctuations in the sequence by adjusting the moving average. In this method, the window size of the moving average is dynamically adjusted according to the characteristics and requirements of the sequence. The series decomposition block can better highlight the long-term trend of the sequence while reducing short-term fluctuations. For an input sequence X ∈ R L ×d of length L, the decomposition formula is shown in Equation (3-15).

[0226]

[0227] where, X s and X t respectively refer to the seasonal component and the periodic trend component in the time series, and both are the input sequence R L×dComponents of space. In addition, to maintain the consistency of the sequence length, the model performs moving average processing through AvgPool(·) and conducts corresponding padding operations. In the research of this invention, X s , X t = SeriesDecomp(X) is used to summarize the above equation, which constitutes a block within the model.

[0228] (2) Model inputs

[0229] In the encoder, the input includes data from the past I time steps, denoted as X en ∈R I ×d. The decomposition architecture is as Figure 13 shown. The input to the AutoFormer decoder includes the seasonal part X des and the trend cycle part X det , which belong to space respectively. Each initialization consists of two parts: the first part comes from the components decomposed from the second half of the encoder input X en , with a length of used to provide the most recent information; the second part comes from a scalar padding placeholder with a length of O. Such a setting enables the decoder to make full use of the past information provided by the encoder to better predict future sequences. It is shown in Equation

[59] (3-16).

[0230]

[0231] where X ens and X ent represent the seasonal part and the periodic part of X en respectively, and X0 and X Mean represent placeholders filled with zeros and the average value of X en respectively.

[0232] (3) Encoder

[0233] The overall calculation of the l-th encoder layer is shown in Equation (3-17).

[0234]

[0235] Specific details are shown in Equation

[59] (3-18).

[0236]

[0237] where, "_" represents the eliminated trend part, represents the output of the l-th layer encoder, represents the embedded X en , respectively represent the seasonal components after the decomposition of the $i$-th sequence block in the $l$-th layer.

[0238] (4) Decoder

[0239] As Figure 13 shown. Each layer of the decoder has two auto-correlation functions: inner auto-correlation and encoder-decoder auto-correlation. Using the latent variables from the encoder The equation for the first encoder layer can be summarized as The decoder can be expressed as Equation

[59] (3-19).

[0240]

[0241] where represents the output of the $l$-th decoder layer, embedded from $X$ des for depth transformation, represents the initial value for accumulation, and respectively represent the seasonal data component and the trend cycle component after the decomposition of the $i$-th sequence block in the first layer, $W$ l,i , $i \in \{1, 2, 3\}$ represents the projection of the $i$-th extracted trend . The final prediction result is the sum of the two refined decomposition components, where $W$ s represents projecting the depth-transformed seasonal data volume to the target dimension.

[0242] 3.3.2 Auto-Correlation Mechanism

[0243] The auto-correlation mechanism discovers the periodic dependencies of seasonal data by analyzing the sequence auto-correlation and merges similar subsequences through time-delay aggregation. Its structure is as Figure 14 shown.

[0244] (1) Periodicity-Based Dependencies

[0245] Inspired by the theory of stochastic processes, for a true discrete-time process $\{X$ t $\}$, the auto-correlation $R$ XX $(\tau)$ can be obtained through the following equation (3-20).

[0246]

[0247] $R$ XX $(\tau)$ reflects $\{X$ t}(τ) and its τ-lagged sequence {X t-τ}(τ). The autocorrelation function R(τ) is used as an unnormalized confidence measure for estimating the period length τ. Then, the k most likely period lengths τ1,…,τ k k are selected. Based on these estimated period lengths, the periodic dependencies in the time series are established and can be weighted by the corresponding autocorrelation function.

[0248] (2) Temporal Delay Aggregation

[0249] The confidence measures normalized by SoftMax are used to weight and aggregate subsequences. In the single-head case, the time series X of length L is passed through a projector to obtain the query Q, key K, and value V, enabling seamless replacement of the self-attention mechanism. The self-correlation mechanism process is expressed as Equation

[59] (3-21).

[0250]

[0251] In this process, argTopk(·) is a function used to obtain the Topk self-correlation parameters, where c is a hyperparameter. R Q,K represents the self-correlation relationship between the sequences Q and K. Roll(X,τ) represents an operation on the sequence X with a time delay τ, during which the elements beyond the first position of the sequence are re-introduced to the last position of the sequence. For the encoder-decoder self-correlation model (as Figure 14 shown), the key K and value V are from the output of the encoder and they will be adjusted to length O; while the query Q is from the previous block of the decoder.

[0252] (3) Efficient Computation

[0253] Since O(logL) sequences of length L are aggregated, the complexity of Equation (3-21) and its subsequent (3-22) is O(LlogL). For the autocorrelation calculation Equation (3-20), given the time series {X t}, R XX (τ) can be calculated by the fast Fourier transform (FFT), and the calculation formula is expressed as Equation

[59] (3-22).

[0254]

[0255] where F represents the FFT, F -1 represents the inverse operation of the FFT, * represents the conjugate operation, and S XX(f) Represent the autocorrelation in the frequency domain. Importantly, the FFT allows the calculation of the autocorrelation of the sequence for all time lags in {1, ···, L} at once. Therefore, the complexity of the autocorrelation calculation can be achieved at O(L log L).

[0256] (4) Attention mechanism VS self-attention mechanism series

[0257] The difference between the autocorrelation attention and the self-attention series lies in their different connection forms. The autocorrelation attention shows a sequential connection, such as Figure 15 as shown.

[0258] 3.4 Prediction model based on the WCACM neural network

[0259] 3.4.1 Model structure

[0260] The overall structure of the WCACM greenhouse internal environment data factor prediction model proposed by the present invention is as Figure 16 shown. First, the data of the previous four days (96H) are used as the input to predict the data of the next 24H, 96H, 192H, and 336H respectively. The algorithm flow of the model is as follows:

[0261] ① Perform wavelet packet denoising (WPD) on the environmental factor time series after data preprocessing.

[0262] ② Use the time series obtained after WPD denoising as the input of the complete ensemble empirical mode decomposition (CEEMD) for decomposition to obtain the IMF components.

[0263] ③ Input each IMF component after CEEMD decomposition into the AutoFormer neural network model for prediction.

[0264] ④ The AutoFormer neural network model uses the data of the previous 4 days (96H) as the input to predict the environmental factor data of the next 24H, 96H, 192H, and 336H.

[0265] ⑤ Finally, integrate and superimpose the predicted component signals to obtain the final prediction result.

[0266] 3.4.2 Experimental scheme

[0267] In the experiment, the dataset was divided into a training set, a validation set, and a test set in the ratio of 7:1:2 for model training, validation, and testing. Considering that the internal environmental data factors of each greenhouse are all time-series data, in order to complete the sequence prediction task, the experimental data was processed by sliding window sampling. In addition, neural network models such as LSTM, TransFormer, LogTrans, InFormer, and AutoFormer were also used to conduct experimental comparisons with the WCACM model; and the data of four days (96H) were used as input to predict the data of the next 24H, 96H, 192H, and 336H respectively to evaluate their performance in task implementation.

[0268] 3.5 Experimental Results and Analysis

[0269] In this section, the WCACM model was used to predict the collected internal environmental dataset of the greenhouse. Considering that the dataset contains multiple data types, the experiment selected air temperature, light intensity, and soil humidity data as examples and presented and analyzed the experimental results.

[0270] 3.5.1 Experimental Settings

[0271] The experimental environment configuration is as follows: The NVIDIA A100 GPU with 80GB of video memory was used, and the operating system was Ubuntu 20.04. In addition, PyTorch was selected as the algorithm framework for this experiment, and the Python language was used to write the algorithm code in the experiment.

[0272] After multiple experiments and comparisons, the following settings were made for the hyperparameters of the model in the present invention: The input sequence length was 96 steps, the prediction sequence lengths were 24, 96, 192, and 336 steps respectively, and the label length was 48 steps. The training batch size was 32 to ensure that the model processes an appropriate amount of data in each iteration, thereby maintaining the training speed and stability. The learning rate was 0.0001 to ensure the stability of the optimizer when adjusting the model weights. The model adopted the Adam optimizer, and its adaptive learning rate strategy adjusted the gradients between different layers, improving the training efficiency and stability. The dropout rates of the fully connected layer and other layers were both 0.05 to prevent the model from overfitting and enhance the generalization ability. The model dimension was 512, converting the input sequence data into a 512-dimensional internal representation to better capture the complex features in the data. The number of attention heads was 8, improving the performance of the model when calculating attention. The number of encoder layers was 2, and the number of decoder layers was 1, ensuring that the model maintains an appropriate depth when processing time-series data. The GELU activation function was selected for the model, improving the nonlinear performance ability of the model.

[0273] 3.5.2 Model Performance Evaluation Metrics

[0274] To more precisely quantify and evaluate the prediction accuracy of experimental results, the experiments of the present invention selected the Mean Squared Error (MSE) and the Mean Absolute Error (MAE) as the main evaluation indicators for verifying the model performance. The MAE is commonly used to measure the prediction accuracy of the model, and the smaller the value, the better, indicating that the overall prediction error of the model is smaller. In contrast, the MSE focuses more on evaluating the stability of the prediction error, is more sensitive to outliers, and the smaller the value, the better. The present invention focuses on these two indicators, MSE and MAE, because they can comprehensively evaluate the prediction performance of the model. Through the analysis of these indicators, a deeper understanding of the model's performance in time series prediction tasks can be obtained, and the model design can be further optimized and improved. Their formulas are respectively expressed as Equation

[60] (3-23) and Equation

[61] (3-24).

[0275]

[0276] Among them, n represents the number of samples, yi represents the true value, represents the predicted value.

[0277] In the next section of the experiment, the prediction results of the test set in different network models are shown. The orange line represents the predicted value of the model, while the blue line represents the true observed value.

[0278] 3.5.3 Comparative Experiment of Models for Predicting the Next 24 Hours

[0279] After data preprocessing, the datasets of air temperature, light intensity, and soil humidity all have 4,728 data, that is, 197 days. The hourly data in 197 days are divided according to the division method in Section 3.4.2. They are applied to the same prediction task scenario, that is, predicting the air temperature, light intensity, and soil humidity within the next 24 hours. In Figure 17 、 18 and 19, it is observed that the predicted data of the WCACM network model is basically consistent with the trend of the actual data, and the prediction result is the closest to the true value. It should be noted that after conducting the prediction experiment test for the LSTM model for the next 24 hours, it can be clearly observed that there are significant differences between the true value and the predicted value in the experimental results. Whether in terms of trend or extreme points, the deviation between the two is very obvious and the gap is extremely large. This indicates that the LSTM model fails to accurately fit the true value, that is, it is not suitable for short- and medium-term time series prediction tasks. Therefore, in the subsequent experimental demonstrations, the experimental results processed by the LSTM model will no longer be shown and analyzed.

[0280] Table 3.1 Comparison of evaluation indicators of models

[0281]

[0282]

[0283] Table 3.1 shows the mean square error (MSE) and mean absolute error (MAE) results of different network prediction models on different data test sets. Observing the table, it can be seen that the WCACM network model proposed by the present invention has lower MSE and MAE values compared with other network models, shows the best performance, and has a better prediction effect.

[0284] 3.5.4 Comparative experiment of models for predicting the next 96 hours

[0285] The data set is divided in the same way as in Section 3.4.2. They are applied to the same task scenario to conduct prediction experiments on the air temperature, light intensity, and soil humidity data for the next 96 hours. Figure 20 、 21 It is observed in 22 that the predicted data of the WCACM network model is basically consistent with the trend of the actual data, and the prediction result is closest to the true value.

[0286] Table 3.2 Comparison of evaluation indicators of models

[0287]

[0288] Table 3.2 shows the mean square error (MSE) and mean absolute error (MAE) results of different network models on the test set. Observing the table, it can be seen that the WCACM network model proposed by the present invention has lower MSE and MAE values compared with other network models, shows the best performance, and has a better prediction effect.

[0289] 3.5.5 Comparative experiment of models for predicting the next 192 hours

[0290] The data set is divided in the same way as in Section 3.4.2. They are applied to the same task scenario to conduct prediction experiments on the air temperature, light intensity, and soil humidity data for the next 192 hours. Figure 23 、 24 It is observed in 25 that the predicted data of the WCACM network model is basically consistent with the trend of the actual data, and the prediction result is closest to the true value.

[0291] Table 3.3 Comparison of evaluation indicators of models

[0292]

[0293] Table 3.3 shows the mean square error (MSE) and mean absolute error (MAE) results of different network models on the test set. Observing the table, it can be seen that the WCACM network model proposed by the present invention has lower MSE and MAE values compared with other network models, showing the best performance and better prediction effect.

[0294] 3.5.6 Comparative Experiment on Predicting the Next 336 Hours Model

[0295] The dataset division is the same as in Section 3.4.2. Apply them to the same task scenario to conduct prediction experiments on the air temperature, light intensity, and soil humidity for the next 336 hours. It is observed in Figure 26 、 27 and 28 that the predicted data of the WCACM network model is basically consistent with the trend of the actual data, and the prediction result is closest to the true value.

[0296] Table 3.4 Comparison of Evaluation Metrics of Each Model

[0297]

[0298] Table 3.4 shows the mean square error (MSE) and mean absolute error (MAE) results of different network models on the test set. Observing the table, it can be seen that the WCACM network model proposed by the present invention has lower MSE and MAE values compared with other network models, showing the best performance and better prediction effect.

[0299] Multi-view stereo matching algorithms can be divided into four types: based on point cloud, voxel, surface evolution, and depth map prediction. The essence of 3D reconstruction based on depth map prediction is to utilize the image information under multiple viewpoints, calculate the disparity between corresponding pixels in the images, and then infer the depth information of each point in the scene, and finally generate a depth map. Compared with other methods, this method has strong flexibility and scalability

[64] . Therefore, the present invention adopts a depth estimation method to perform multi-view Figure Three dimensional reconstruction.

[0300] The plane sweep method is a computer vision technique used to reconstruct the 3D scene structure from 2D images. Its essence is to divide the reference camera space into multiple parallel planes, scan each plane in turn from far to near, then project and match each point, and finally obtain the depth map of the reference view. During the scanning process, it is necessary to calculate the projection points of each point on different views and compare the similarity scores to determine the depth value of each point. Through this method, the depth estimation of each pixel point on the reference view can be achieved, and then the depth map can be obtained.

[0301] MVSNet (Multi-View Stereo Network) is a deep learning-based multi-view stereo matching algorithm. It directly learns the depth information of the scene from images of multiple views using a neural network model, achieving end-to-end multi-view 3D reconstruction

[45] . Its basic principle will be introduced in detail in Section 4.3.

[0302] Compared with traditional multi-view stereo matching algorithms, MVSNet has advantages such as end-to-end learning, accurate depth estimation, and strong generalization ability. By directly learning depth information from images of multiple views, the process can be simplified and the efficiency can be improved. The deep learning model can learn richer feature representations from a large amount of data, so MVSNet can produce more accurate depth estimation results. In addition, the features trained by MVSNet have good generalization and can adapt to various new scenarios, showing strong versatility and adaptability

[45] .

[0303] 4.2 Dataset Preparation

[0304] The present invention selects the large-scale and publicly available DTU

[65] multi-view 3D reconstruction dataset as the experimental training data. This dataset contains images of various objects taken under different lighting conditions and viewpoints, as well as the corresponding real 3D point cloud data. Researchers can evaluate the accuracy of the algorithm by comparing the algorithm output with the real point cloud data. The DTU dataset is widely used in the fields of computer vision and 3D reconstruction, providing a standardized evaluation platform for algorithm testing and optimization. Related examples are shown as Figure 29 shown.

[0305] For the experiment, the dataset is divided into a training set, a validation set, and a test set according to the ratio of 7:1:2. Among them, 87 scenes are used for training, 12 for validation, and 25 for testing. In addition to using the DTU dataset, real data of plants in the greenhouse are also collected to further verify and compare the performance of the method of the present invention with traditional multi-view 3D reconstruction methods in actual scenarios. Examples of the self-collected dataset are shown as Figure 30 shown.

[0306] 4.3 Introduction to the Plant 3D Reconstruction Method Based on SfM-MVSNet

[0307] The overall process of plant 3D reconstruction based on SfM-MVSNet is as shown in Figure 31 shown. It mainly uses the incremental SfM algorithm and the MVSNet deep learning algorithm to realize the 3D model reconstruction of different plants.

[0308] First, feature points are extracted from images taken from multiple perspectives and matched to establish the correspondence between images. Then, an incremental SfM algorithm is adopted to infer camera parameters based on the correspondence, determine the position and pose of the camera, and further obtain the sparse three-dimensional point cloud data of the plant. On this basis, the depth range can be determined according to the sparse point cloud model. Subsequently, using the MVSNet deep learning algorithm framework, combined with the internal and external camera parameters and depth range calculated previously, depth maps are generated for each image, and masks for the regions of interest are generated. Finally, by combining the previously obtained pairing information with the depth maps, a dense three-dimensional point cloud model of the plant is generated, which can then more accurately present the morphology and structure of the plant.

[0309] 4.3.1 Sparse Reconstruction Based on Multi-View Images

[0310] The MVSNet (Multi-View Stereo Network) algorithm is a deep learning-based multi-view stereo matching method used to reconstruct the depth information of a scene from images taken from multiple perspectives. The network framework is as Figure 32 shown.

[0311] MVSNet is a deep learning network that is essentially used for image depth estimation. It takes the images of the reference view and source views as inputs and outputs the depth map for each reference view. Its essence is to generate point clouds by fusing the depth maps with traditional MVS methods, rather than directly constructing a three-dimensional model. Therefore, MVSNet completes the end-to-end optimization of the point cloud reconstruction process on the depth map through end-to-end learning. Finally, standard metric methods can be used to compare the generated point cloud reconstructions.

[0312] First, a two-dimensional convolutional neural network (CNN) is used to extract features from the reference view and source views input to the network. As Figure 33 shown, this network uses a two-dimensional CNN structure with 8 layers to extract image features. To capture features at different scales, the stride of the 3rd and 6th layers is set to 2, and the stride of other layers is 1. At each scale, two convolutional layers are used to extract features from the output of the previous layer. Then, batch normalization (BN) is performed on the convolutional results, and non-linear activation is carried out through the rectified linear unit activation function (ReLU). Finally, the two-dimensional CNN network outputs a feature map with 32 channels. In addition, to improve the learning efficiency, the feature extraction networks for each image use the same weight parameters.

[0313] After completing the image feature extraction, the feature maps generated by the 2D CNN network are combined with the camera parameters obtained by the incremental SfM algorithm. With the help of the differentiable homography transformation technique, the feature maps of multiple source view images are mapped onto the feature map of the reference view to construct a 3D matching cost. Assume that I1 is the reference view for which the depth is to be predicted, represents the source views for multi-view stereo matching with I1, and each source view has a corresponding camera intrinsic matrix K i , rotation matrix R i and translation matrix t. The present invention uses homography transformation to transform the feature mapping map F i corresponding to each source view I i onto the plane of the feature mapping map F1 corresponding to the reference view I i . The homography transformation formula is shown in Equation

[45] (4-1).

[0314]

[0315] In the formula, d is a uniformly sampled value within the depth range determined during the incremental SfM algorithm process, and n1 is the direction of the principal axis of the reference camera. For the feature mapping map F1 corresponding to the reference view I1, a 3×3 identity matrix is used as the homography transformation matrix. This setting ensures the consistency and general applicability of the homography transformation process when processing all views.

[0316] After all source views are transformed by homography, they are integrated into the 3D space to form a feature mapping volume, which is represented as If the height and width of each source view are H and W respectively, the number of depth samplings is D, and the feature mapping map output by the 2D convolutional network contains F channels. Since two convolutional operations with a stride of 2 are performed, the final feature mapping volume V generated by homography transformation is represented as It should be noted that since the feature mapping map of the reference image is used as a benchmark, the feature mapping volume of the reference view is directly obtained by replicating its feature mapping map at each depth.

[0317] To enable the network to adapt to an arbitrary number of input views, the present invention adopts a variance-based mapping relationship to calculate the matching cost volume C, and its calculation formula is shown in Equation

[45] (4-2).

[0318]

[0319] Among them, N is the number of input views, is the mean of all feature maps. In the cost volume, the values of each point at different depths and channels represent the variances of the points in the corresponding feature maps of the depth and channels. The variance in the cost volume reflects the degree of consistency of the depth prediction values of the feature points at different depths on a certain channel of the feature map. In short, the smaller the variance, the higher the consistency of the points in the feature map in depth prediction at that depth, that is, the predicted depth value is more likely to be close to the true value. Therefore, by observing the variances of the points in the cost volume, the accuracy and reliability of the depth prediction of the feature points at different depths can be evaluated, which helps to select the depth value that best conforms to the actual situation.

[0320] Due to reasons such as image noise, incorrect matching, occlusion and reflection, and geometric distortion in the real scene, it will interfere with multi-view stereo matching. To reduce noise and unnecessary details, it is necessary to regularize the cost volume C to improve the accuracy and stability of depth estimation, and it can also effectively reduce memory consumption and computational cost. The present invention regularizes the cost volume through a multi-scale 3D CNN network and aggregates adjacent information using an encoder-decoder structure. The network reduces the number of channels from 32 to 8, and the number of convolutional layers at each scale is reduced from 3 to 2, and finally a single channel is output. A probability normalization operation is adopted in the depth direction to obtain the probability space P of depth estimation. For any point (x, y, d) on the probability space P, its corresponding value P (x,y,d) represents that the probability that the pixel point (x, y) corresponds to the depth value d is P (x,y,d) , which reflects the confidence of this depth estimation value.

[0321] Before constructing the depth map D, depth estimation is first performed by analyzing the probability space P pixel by pixel. However, the depth sampling value with the highest probability at each pixel point cannot be simply selected as the depth estimation value. This is because the depth sampling values are uniformly sampled, so the obtained depth estimation values are discrete. In addition, since the argmax operation cannot be differentiated, the model cannot use error backpropagation to learn network parameters.

[0322] In view of the foregoing problems, the softargmin operation is adopted to estimate the depth value. This operation essentially calculates the expected value in the depth direction to initialize the depth map. The softargmin operation is expressed as Equation

[45] (4-3).

[0323]

[0324] where d is the depth sampling value, P(d) is the probability estimation value of the pixel at depth d, d max is the maximum depth, d minis the minimum depth, and D is the expected value in the depth direction. Through the soft argmin operation, the model can achieve continuous depth estimation. And this operation is differentiable, so end-to-end network training can be carried out through error backpropagation to improve the model performance.

[0325] Next, depth map optimization is carried out. After obtaining the initial depth map, there are still two problems. First, for the pixel points in the background area of the reference view, their probability distributions at different depth sampling values are scattered and do not converge to an obvious peak. Therefore, the present invention uses the probability distribution of depth estimation to measure the confidence of the depth estimation result. By selecting the pixel points around the background area of the reference view and considering the sum of the depth estimation probabilities corresponding to the four nearest depth sampling values around these points, the confidence of the depth estimation value of each point is calculated. Then, the points with confidence lower than the preset threshold (set to 0.8 in this experiment) are regarded as invalid points and removed from the depth map. For the remaining valid points, their original depth estimation values are multiplied by the corresponding confidence to obtain a depth map with invalid points removed.

[0326] Second, to solve the problem that the initial depth map may be overly smoothed, the present invention uses a depth residual network to fine-tune the initially estimated depth map with the reference view as the standard. First, the initially estimated depth map is connected with the reference view to form a four-channel input. Then, it is processed through a depth residual network, which includes three 32-channel convolutional layers and one 1-channel convolutional layer to perform residual learning and improve the accuracy of the depth map. Such processing makes the boundaries of the depth map clearer.

[0327] After training the multi-view matching network, the depth map of the reference view can be predicted. However, directly using the fusion of these depth maps may lead to redundancy. To obtain a more accurate 3D reconstruction model, the present invention will combine the depth matching information generated in SfM, verify the depth value consistency through the projection and back-projection between the reference view and the 3D space, and fuse the depth maps of different views to generate a 3D point cloud. The fusion process is as follows Figure 34 shown. The calculation of converting the depth map to a 3D point cloud is shown in Equation (4-4).

[0328]

[0329] where X is the coordinate of the 3D point cloud in space, P is the homogeneous coordinate form of the pixel point, C i is the camera center coordinate of the current view, is the inverse matrix of the camera internal parameters, is the transpose operation of the rotation matrix, and d is the depth value.

[0330] First, use Equation (4-4) to calculate the C in the reference view iA certain pixel point captured from a perspective is projected into three-dimensional space, and a three-dimensional point X is obtained based on its depth estimation value. Then X is re-projected onto C i in the images of adjacent capture perspectives, and the corresponding depth value d is calculated. If C i the original depth estimation value d corresponding to the adjacent capture perspective image orig differs from the just calculated depth value d by within 1%, that is, it satisfies the constraint described by formula (4-5), then this point is retained, otherwise it is ignored. Finally, all depth maps are projected into three-dimensional space and merged into a three-dimensional point cloud.

[0331] ∣d orig -d∣ / d < 0.01 (4-5)

[0332] To evaluate the performance of the method of the present invention in three-dimensional reconstruction, first, the algorithm was tested on the DTU test set, and three quantitative indicators such as accuracy, completeness, and comprehensive performance were used to objectively evaluate the overall three-dimensional reconstruction performance. In addition, the method of the present invention was also compared with advanced methods based on deep learning (such as MVSFormer

[66] ) and traditional methods (such as VisualSFM

[67] , Bundler

[68] , OpenMVG

[69] , MVE

[70] , Colmap

[71] ), and the differences in visual effects, generated file size, and reconstruction time were analyzed.

[0333] The test results show that the deep learning method adopted by the present invention is superior to traditional algorithms in terms of the integrity, accuracy, and robustness of the reconstruction results. Although it is slightly higher than the current leading deep learning method in evaluation indicators, the method of the present invention has significant advantages in terms of reconstruction file size and reconstruction time. By visually comparing the reconstruction results intuitively, it is found that the model effect obtained by the method of the present invention is basically equivalent to that of the current leading method. However, in practical applications, due to the excessive file size and long reconstruction time, the plant characterization analysis system has too high memory occupancy, low efficiency, and low reconstruction efficiency. In contrast, the method of the present invention is more suitable for practical application in the plant characterization analysis system, further proving its superiority.

[0334] To evaluate the performance of the proposed method in a real environment, multi-angle real data of plants in a greenhouse were collected, and experimental tests and comparisons were carried out with advanced methods based on deep learning and various traditional methods. The test results show that the method performs equally well in a real environment.

[0335] The present invention uses the Matlab test script provided by the DTU dataset to evaluate the accuracy, integrity, and comprehensiveness indicators of 3D point clouds. The accuracy indicator determines the position accuracy by calculating the average distance from the reconstructed point cloud to the reference model. The integrity indicator evaluates the model integrity by measuring the average distance from the reference model to the reconstructed point cloud. The shorter these two distance indicators are, the higher the accuracy and integrity. The comprehensiveness indicator, which is the average of these two, is used to comprehensively measure the quality of the multi-view Figure Three dimensional reconstruction model.

[0336] The present invention tests the multi-view 3D reconstruction model based on deep learning on the DTU public dataset and self-made real datasets, and compares and analyzes it with other advanced deep learning methods and traditional methods. The following is the analysis of the test results.

[0337] (1) Test results of the DTU dataset

[0338] After training the network model, the present invention first tests the model on the DTU test set. The overall 3D reconstruction performance of the algorithm is objectively evaluated using three quantitative indicators: accuracy, integrity, and comprehensive performance. The reasons for choosing the DTU dataset for testing are as follows: Although the DTU dataset lacks plant scenes, it contains various types of scenes and objects, with broad representativeness and generality; it also provides rich real-world scene data and high-quality 3D reconstruction results, which can be used as the "ground truth" standard during algorithm testing; it helps to evaluate the generalization ability of the model on diverse scenes and unknown objects, thus verifying the generality and robustness of the model. Examples of test results are as Figure 35 shown. The experimental data shows that the reconstructed point cloud has an average accuracy of 0.396 mm and an average integrity of 0.527 mm, and its comprehensive performance reaches 0.462 mm. The results prove that the method proposed by the present invention has high integrity and accuracy in reconstructing the target object.

[0339] Subsequently, for scenes with different textures and non-diffuse reflections, the method of the present invention is compared with an advanced method (MVSFormer) based on deep learning and traditional methods (VisualSFM, MVE, Bundler, Colmap, OpenMVG) in terms of visual effect, storage space size of the generated file, and reconstruction time. Due to space limitations, the comparative experimental results show the comparison between the present invention and MVSFormer, VisualSFM, MVE, Bundler, and Colmap. The comparison of the reconstruction effects in different scenes is as Figure 36 shown, and the storage space size of the generated file and the reconstruction time are shown in Table 4.2.

[0340] According to Figure 36As shown, when the method of the present invention processes scenes with different textures and non-diffuse reflection characteristics, it shows obvious advantages compared with traditional methods. The reconstructed 3D model is superior to traditional methods in terms of density and details. Compared with advanced methods, the method of the present invention shows clearer details and less noise. Experimental results show that the method of the present invention can achieve complete and accurate 3D reconstruction when processing various objects.

[0341] The 3D reconstruction speed and the size of the storage space occupied by the generated file after reconstruction are both important indicators for evaluating the 3D reconstruction system. Under the experimental environment described in Section 4.4.3, the present invention uses the DTU test dataset to test the method of the present invention and advanced methods based on deep learning (MVSFormer) and traditional methods (VisualSFM, MVE, Bundler, Colmap, OpenMVG). And record and compare the average time consumed by each method for 3D reconstruction of a single group of data and the average storage space size of the generated file after reconstruction. The experimental results are shown in Table 4.2.

[0342] Table 4.2 Average reconstruction time and average file size for a single scene by different methods

[0343]

[0344]

[0345] As can be seen from Table 4.2, for the average 3D reconstruction time of a single scene, the method of the present invention is far superior to other methods. Among traditional 3D reconstruction methods, Colmap has the best reconstruction quality, but its reconstruction time is the longest; and compared with the method of the present invention, the reconstructed 3D point cloud is slightly sparse and lacks some details.

[0346] Among the average storage space sizes of the generated files after reconstructing a single scene, the file generated by the VisualSFM method occupies the smallest storage space. However, the 3D point cloud it reconstructs is relatively sparse, with poor quality and low integrity. On the contrary, the file generated by the MVSFormer method has the largest average storage space occupancy, about 18 times that of the method of the present invention. This will lead to too high memory occupancy and low efficiency in the subsequent plant phenotype analysis system. In addition, the 3D point cloud it reconstructs contains more miscellaneous points, which will affect the detail analysis of the point cloud.

[0347] According to the experimental results, the multi-view Figure Three 3D reconstruction method based on deep learning proposed by the present invention not only has a shorter reconstruction time in the public dataset test, but also can generate a complete, dense, less noisy and detail-rich 3D point cloud while keeping the generated reconstruction file occupying less storage space. Therefore, this method has great practical application potential.

[0348] (2) Test results of real scenarios (self-made dataset)

[0349] In the present invention, an ordinary monocular camera is used to take pictures of different plants in the greenhouse along a circle. The first group of data contains 150 images of Epipremnum aureum potted plants, while the second group of data contains 150 images of Rohdea japonica potted plants. Then, these image data are tested, compared and analyzed using the method of the present invention, an advanced method based on deep learning (MVSFormer), and traditional methods (VisualSFM, MVE, Bundler, Colmap, and OpenMVG) to evaluate their performance in terms of visual effect, storage space occupied by the generated files, and reconstruction time. Due to space limitations, the comparative experimental results only show the comparison between the present invention and MVSFormer, VisualSFM, MVE, Bundler, and Colmap. The comparison of the reconstruction effects under different plant scenarios is as Figure 37 shown, and the storage space occupied by the generated files and the reconstruction time are shown in Table 4.3

[0350] As Figure 37 can be seen, when the method of the present invention is used to process plants in real scenarios, it shows obvious advantages compared with traditional methods. The reconstructed 3D models of plants are superior to traditional methods in terms of density and details. Compared with the advanced method, the method of the present invention shows clearer details and significantly fewer noise points in the reconstructed 3D models of plants. It should be noted that there are also cases where the advanced method fails to reconstruct. The experimental results show that the method of the present invention can achieve complete and accurate 3D reconstruction in real plant scenarios, is not restricted by a strict reconstruction environment, and has high robustness

[0351] The present invention uses a self-made plant dataset to test the method of the present invention, an advanced method based on deep learning (MVSFormer), and traditional methods (VisualSFM, MVE, Bundler, Colmap, OpenMVG). And record and compare the average time consumed by each method for 3D reconstruction of plant data and the average storage space occupied by the generated files after reconstruction. The experimental results are shown in Table 4.3

[0352] Table 4.3 Average reconstruction time and average file size for plant scenarios of different methods

[0353]

[0354] As can be seen from Table 4.3, for the average 3D reconstruction time of a single plant scenario, the method of the present invention is far superior to other methods. The reconstruction quality of Colmap is the best among traditional 3D reconstruction methods, but the reconstruction time is the longest. Moreover, compared with the method of the present invention, there are a large number of white background noise points in the plant leaf part of the 3D point cloud generated by Colmap, which significantly damages the detail performance of the reconstruction result.

[0355] In terms of the storage space size of the files generated after plant scenario reconstruction, the files generated by the VisualSFM method occupy the smallest storage space. However, the 3D point cloud it reconstructs is extremely sparse, with poor quality and low integrity. The MVSFormer method has cases of reconstruction failure, and the average storage space occupied by the generated files is the largest, about 14 times that of the method of the present invention. The reconstruction file generated by the method of the present invention occupies 58MB, which indicates that the method of the present invention can still effectively control the file size and optimize storage resources while maintaining a high 3D reconstruction quality.

[0356] According to the experimental results, the multi-view Figure Three 3D reconstruction method based on deep learning proposed by the present invention, in the test of the real plant scenario dataset, not only has a short reconstruction time, but also can generate a complete, dense, less noisy and detail-rich 3D point cloud while keeping the storage space occupied by the generated reconstruction file less. Therefore, this method has great potential in practical applications.

[0357] 5.1 Framework of plant phenotype measurement method

[0358] In [reference], a method for measuring plant phenotype parameters based on 3D point cloud is mainly introduced, which includes two modules: a point cloud processing module and a plant phenotype measurement module. First, the point cloud processing module processes the obtained plant point cloud, including downsampling and coordinate correction operations to ensure that the growth direction of the plant point cloud is consistent with the z-axis direction of the coordinate axis. Then, using point cloud feature segmentation and point cloud denoising operations, the phenotype part of the plant is extracted from the original point cloud data. Finally, using the plant phenotype measurement algorithm based on 3D point cloud, the measurement of parameters such as plant height, crown width and surface area is completed. The plant phenotype measurement framework is as Figure 38 shown.

[0359] 5.2 Preprocessing of plant point cloud

[0360] The point cloud data after 3D reconstruction of plants is relatively dense. To more quickly and accurately segment plant features and reduce the processing difficulty, it is necessary to perform point cloud downsampling on it. In addition, since there are deviations in the spatial direction and scale between the reconstructed plant point cloud and the real plant, it is necessary to correct its coordinates to eliminate these differences. Subsequently, to more accurately calculate plant phenotypic data, it is necessary to use a point cloud feature segmentation algorithm to segment the plant from the background and the ground. However, the segmented plant point cloud may still contain a small amount of noise, and it is necessary to denoise it to eliminate interference.

[0361] 5.2.1 Point cloud downsampling

[0362] To efficiently measure plant phenotypic parameters, it is first necessary to process the dense 3D plant point cloud data. Directly correcting the coordinates of the plant point cloud, segmenting features, and measuring its phenotypic parameters will consume a large amount of time and memory resources. Therefore, a point cloud downsampling method is adopted to reduce the number of points in the point cloud while retaining the outer contour of the plant point cloud, so as to improve the measurement efficiency. The present invention uses a uniform downsampling method to process the plant point cloud, which maintains the distribution characteristics of the original point cloud and avoids information loss. The downsampling result is as Figure 39 shown.

[0363] When performing uniform downsampling, first determine the sampling rate r, which represents the proportion of points finally retained in the original point cloud. Then, calculate the interval k according to the sampling rate r, that is, how many points to select and retain in the original point cloud at intervals, and the calculation formula is shown in Equation (5-1).

[0364]

[0365] In the formula, n is the number of points in the original point cloud, r is the sampling rate, represents rounding x down. Then, traverse each point in the original point cloud, starting from the first point, and select and retain a point every k points. This operation can ensure that the selected points are evenly distributed in the entire point cloud model, and maximize the retention of the structure and characteristics of the original point cloud.

[0366] 5.2.2 Point cloud coordinate correction

[0367] (1) Coordinate ratio correction

[0368] There is a certain proportional scaling relationship between the actual plant and its 3D point cloud model. To accurately determine the ratio between them, it is necessary to perform coordinate ratio calibration on the point cloud. The present invention calculates the ratio relationship based on the plant flower pot, as shown in Equation (5-2).

[0369]

[0370] Among them, R is the actual ratio, H realis the true height of the flowerpot, H reconstructed is the height of the flowerpot in the 3D point cloud.

[0371] (2) Coordinate transformation and correction

[0372] Since there are deviations in the spatial direction and scale between the reconstructed point cloud and the real plant, in order to accurately measure the plant phenotype parameters, it is necessary to correct the coordinates of the reconstructed plant point cloud to solve the coordinate axis error between it and the actual plant growth direction. A rotation and translation matrix is used for coordinate adjustment to ensure that the z-axis direction of the point cloud is consistent with the plant growth direction. The results of the coordinate transformation and correction of the point cloud are as Figure 40 shown.

[0373] The z-axis transformation and correction process is as follows:

[0374] ① Use the Random Sample Consensus (RANSAC) algorithm to identify the model in the point cloud data and detect the geometry of the ground.

[0375] ② After detecting the ground, the normal vector m of the ground can be obtained, and m represents the orientation of the ground.

[0376] ③ Calculate the rotation angle θ between the ground normal vector m and the z-axis normal vector n(0, 0, 1). The calculation formula is shown in Equation (5-3).

[0377]

[0378] ④ After obtaining the rotation angle and the rotation axis, the present invention uses the Rodriguez rotation formula to calculate the rotation matrix

[72] . The calculation formula is shown in Equation (5-4).

[0379] R mt = E×cosθ+(m·n)×d×(1 - cosθ)+(m·n)×sinθ (5-4)

[0380] where R mt is the rotation matrix, E is the third-order identity matrix, θ is the rotation angle, and d=(d1, d2, d3) is the unit vector of m·n.

[0381] ⑤ In the last step, apply the point cloud data to the rotation matrix so that the z-axis in the space of the point cloud is corrected to the direction perpendicular to the ground, as Figure 40 (b) shown. The red line, green line, and blue line in the figure represent the directions of the x-axis, y-axis, and z-axis in space respectively. The calculation formula for the coordinate transformation and correction of the point cloud is shown in Equation (5-5).

[0382] T a = R mt T o (5-5)

[0383] where Ta is the plant point cloud after transformation, R mt is the rotation matrix, T o is the original plant point cloud.

[0384] 5.2.3 Feature Segmentation of Plant Point Cloud

[0385] The main task of plant point cloud segmentation is to effectively separate the main parts of the plant (trunk, leaves, etc.) from the background such as flower pots and the ground. When performing plant phenotype measurement, the operation of removing background point cloud information can reduce unnecessary calculations and improve accuracy.

[0386] The present invention adopts a method based on point cloud color information to realize the segmentation of plant point cloud. First, the RGB color space is converted into the HSV color space, which can better capture the information of the green part of the plant. In the HSV color space, S represents saturation, V represents value, and H represents hue. The H channel can directly describe the types of colors and is easier to distinguish different colors. Therefore, for color-based segmentation tasks, HSV segmentation usually has better effects

[73] . Then, histogram analysis is performed on the H (hue) channel in the HSV color space, which enables the intuitive finding of the highest peak of the green part of the plant, and this peak will be used as the segmentation threshold, which can accurately capture the characteristics of the green part of the plant and achieve an accurate segmentation effect. This method can not only efficiently segment the main part of the plant from the overall point cloud, but also is not affected by factors such as the shape of the point cloud, so it has strong applicability and robustness. The segmentation results are as Figure 41 shown.

[0387] 5.2.4 Point Cloud Denoising

[0388] The denoising results are as Figure 42 shown.

[0389] 5.3 Extraction of Plant Three-Dimensional Phenotype Parameters

[0390] 5.3.1 Calculation Methods for Plant Height and Crown Width

[0391] The present invention uses the Axis-Aligned Bounding Box (AABB) method to calculate the plant height and crown width parameters of the plant point cloud model. Its principle is to use an axis-aligned rectangular bounding box to approximately represent the boundary range of the plant point cloud model. The process of calculating the AABB is to find the minimum and maximum coordinate values in the point cloud data, and then determine the position and size of the bounding box according to these coordinate values. After determining the AABB bounding box, the plant height and crown width information of the plant point cloud model can be extracted according to the size of the bounding box. Figure 43 is a schematic diagram for the extraction of plant height and crown width parameters.

[0392] 5.3.2 Plant surface area calculation method

[0393] The specific triangular meshing process is as follows:

[0394] ① Determine the Alpha shape parameter: First, an appropriate Alpha shape parameter (α) needs to be selected. This parameter determines the neighborhood size around each point, thus affecting the density of the generated triangular mesh. Usually, the value of α is adjusted according to the density of the point cloud and the required density of the triangular mesh. A larger α value will result in a sparse triangular mesh, while a smaller α value will result in a dense triangular mesh.

[0395] ② Construct the Alpha shape: For each point, construct its Alpha shape. Specifically, for each point, a spherical neighborhood with a radius of α is constructed centered at that point. All points within this spherical neighborhood are considered adjacent to that point.

[0396] ③ Determine the connection relationship: Next, the connection relationship between all points within the Alpha shape of each point needs to be determined. This invention accomplishes this by performing Delaunay triangulation. Delaunay triangulation is a method of generating triangles in a point set where no point lies inside the circumcircle of any triangle. By performing Delaunay triangulation, the connection relationship between points within the neighborhood of each point can be determined, thereby generating a triangular mesh.

[0397] ④ Generate the triangular mesh: Finally, based on the determined connection relationship, the point cloud can be converted into a triangular mesh. For every three connected points, a triangle is created and added to the final triangular mesh. In this way, a triangular mesh model with a continuous surface can be generated from the plant point cloud.

[0398] Figure 44 is the result of point cloud triangular meshing. According to Figure 45 observation, the plant surface is composed of multiple small triangles, and the surface area of the entire mesh can be obtained by calculating the area of each triangle and summing up the areas of all triangles. The specific steps are as follows:

[0399] ① Obtain the vertex coordinates of the triangle: For each triangle, obtain the coordinate information of the three vertices from the triangular mesh.

[0400] ② Calculate the edge vectors and normal vectors of the triangle: For each triangle, calculate the vectors of its three edges. Then, based on the edge vectors of the triangle, calculate the normal vector of the triangle using the cross product. If the vertices of the triangle are arranged in a counterclockwise direction, the normal vector points outward, otherwise inward.

[0401] ③ Calculate the area of the triangle: The area of the triangle can be calculated using the normal vector of the triangle and any two side vectors. The calculation formula for the area is shown in Equation (5-6).

[0402]

[0403] Among them, AB and AC are the two side vectors of the triangle, × represents the cross product of vectors, and ∥·∥ represents the modulus of the vector.

[0404] ④ Accumulate the areas of each triangle: Accumulate the areas of all triangles to obtain the surface area of the entire mesh model.

[0405] 5.4 Plant Phenotype Parameter Measurement Experiment and Analysis

[0406] 5.4.1 Evaluation Index

[0407] For the plant height and surface area parameters, an algorithm is used for measurement, and the results are compared and analyzed with the manually measured values. When evaluating the accuracy of the measurement method, three evaluation indexes, namely the root mean square error (RMSE), the mean absolute percentage error (MAPE), and the coefficient of determination (R 2 ) are adopted. RMSE represents the average deviation between the algorithm measurement value and the manually measured value, and the smaller the value, the higher the measurement accuracy. MAPE represents the average relative error between the algorithm measurement value and the manually measured value, expressed as a percentage, and the smaller the value, the higher the measurement accuracy. R 2 represents the degree of correlation between the algorithm measurement value and the manually measured value. The closer it is to 1, the higher the linear correlation between the two. Its calculation is shown in Equations

[74] (5-7),

[75] (5-8), and

[76] (5-9).

[0408]

[0409]

[0410] Among them, n is the number of samples, x ai is the three-dimensional measurement data, x mi is the manually measured data, Cov(X,Y) is the covariance of X and Y, and Var[X] and Var[Y] are the variances of X and Y.

[0411] 5.4.2 Plant Height Measurement Results

[0412] The present invention adopts two methods for measuring the height of plants, namely algorithm measurement and manual measurement. In the algorithm extraction, the present invention adopts the axis-aligned bounding box (AABB) method to calculate the plant height of the plant point cloud model. In order to compare the accuracy of the algorithm measurement, 20 plants are selected for the experiment, and the measurement results are shown in Table 5.1. And a scatter plot is used to display the comparison results, asFigure 46 As shown. The horizontal axis of the scatter plot represents the manually measured values, and the vertical axis represents the algorithm-measured values. Each data point represents the measurement result of a plant. Among them, RMSE is 0.38, MAPE is 2.47%, and R 2 is 0.98. According to the result analysis, the values of RMSE and MAPE are small, indicating that the deviation and error between the algorithm-measured values and the manually measured values are small, showing that the algorithm has high measurement accuracy. At the same time, R 2 close to 1 indicates a high linear correlation between the algorithm-measured values and the manually measured values, further verifying the accuracy and reliability of the algorithm measurement.

[0413] Table 5.1 Comparison between Manually Measured Values and Algorithm-Measured Values of Plant Heights

[0414]

[0415] 5.4.3 Measurement Results of Plant Surface Areas

[0416] For the measurement of the surface area of plants, the present invention also adopts two methods: algorithm measurement and manual measurement. The algorithm measurement method uses the method of triangular meshing to perform surface reconstruction on the point cloud data of plants, generate a triangular mesh model, and then calculate the surface area of the plant. In order to compare the measurement accuracy of the algorithm, 20 plants were selected for experiments, and the measurement results are shown in Table 5.2. And a scatter plot is used to display the comparison results, as Figure 47 shown. The horizontal axis of the scatter plot represents the manually measured values, and the vertical axis represents the algorithm-measured values. Each data point represents the measurement result of a plant. Among them, RMSE is 11.87, MAPE is 1.87%, and R 2 is 0.98. According to the result analysis, the values of RMSE and MAPE are small, indicating that the deviation and error between the algorithm-measured values and the manually measured values are small, showing that the algorithm has high measurement accuracy. At the same time, R2 close to 1 indicates a high linear correlation between the algorithm-measured values and the manually measured values, further verifying the accuracy and reliability of the algorithm measurement.

[0417] Table 5.2 Comparison between Manually Measured Values and Algorithm-Measured Values of Plant Surface Areas

[0418]

[0419]

[0420] A method for extracting plant phenotypic parameters is designed for plant three-dimensional point clouds. For the plant point clouds obtained by three-dimensional reconstruction, processes such as point cloud downsampling, coordinate correction, feature segmentation, and denoising are carried out to extract plant phenotypic characteristics. Then, the axis-aligned bounding box (AABB) method is used to obtain the plant height and crown width information, and the plant surface area is calculated by the point cloud triangulation method. Next, the obtained plant height and surface area parameters are compared with the manually measured values, and evaluation indexes such as the mean absolute percentage error, root mean square error, and coefficient of determination are calculated to evaluate the effectiveness of the method of the present invention. The results show that the method can quickly and accurately extract plant phenotypic parameter information.

[0421] The present invention develops a smart greenhouse monitoring system. The design of the system aims to provide a convenient and fast intelligent agriculture monitoring platform for users. The system mainly uses the Python programming language, combines the Flask framework and the Bootstrap framework, and integrates multiple third-party libraries such as Open3D, OpenCV, ECharts, and PyMySQL, etc. to complete the development of the system. The system is deployed on the Windows operating system and adopts the B / S mode. The functional modules mainly include real-time data monitoring of the internal environment of the greenhouse, historical data query, data prediction, and plant phenotypic analysis based on three-dimensional point clouds, etc. The function realization and principle of each module of the system will be elaborated.

[0422] 6.1 System Requirement Analysis

[0423] System requirement analysis is a crucial step in the system design and development. It helps to clarify the needs and expectations of users, and provides the basis and direction for system design and implementation. Through system requirement analysis, the requirements in aspects such as the functions, performance, and interface design of the system can be accurately defined, thus providing clear working goals. At the same time, system requirement analysis can also identify and solve potential problems and risks to ensure meeting the expectations of users.

[0424] 6.1.1 Non-functional Requirement Analysis

[0425] (1) Performance Requirements

[0426] In terms of performance requirements, the system needs to be able to quickly respond to user requests. Especially for the acquisition of real-time data and prediction results, low latency should be ensured. At the same time, the system should have good concurrent processing capabilities, be able to effectively handle the requests of multiple users under high load, and maintain the stability and fluency of the system. In addition, the system needs to reasonably utilize computing resources to improve the performance and throughput of the system while saving resource costs as much as possible.

[0427] (2) Security Requirements

[0428] In terms of security, the system should have a strict access control mechanism to ensure that only authorized users can access sensitive data and functions, preventing unauthorized access. In addition, the system needs to take measures to protect the confidentiality and integrity of user data, including encrypted storage, transmission, and access control, etc., to prevent data leakage or tampering. The system should also be capable of guarding against common network attacks to ensure the security and stability of the system.

[0429] (3) Maintainability requirements

[0430] In terms of maintainability, the system should be configurable, facilitating administrators to configure and adjust the system according to needs to cope with different business requirements and environmental changes. At the same time, the system design should have good scalability, making it easy to add new functions or modules to meet the ever-changing needs of users. The system should record the logs of key operations and events for easy troubleshooting and system maintenance to ensure the stability and reliability of the system.

[0431] (4) User experience requirements

[0432] In terms of user experience, the system interface should be simple, intuitive, and easy to operate, conforming to user usage habits and reducing the learning cost and operation complexity of users. The system should support multiple devices and screen sizes to ensure good display effects on different devices and enhance the user experience. The system should provide clear and complete user documentation and help information to facilitate users to understand and use the system and reduce the usage difficulty and communication cost of users.

[0433] 6.1.2 Functional requirement analysis

[0434] (1) Basic module

[0435] The basic module provides user registration, login, and logout functions. Login verifies user credentials and allows access to the system. The logout function allows users to safely exit the system.

[0436] (2) Real-time data monitoring module

[0437] The real-time data monitoring module ensures that users obtain the latest data on the internal environment of the greenhouse. The monitoring frequency is updated hourly, including soil temperature and humidity, salinity, air temperature and humidity, light intensity, evaporation, CO2, and PM2.5 concentration.

[0438] (3) Historical data query module

[0439] The historical data query module facilitates users to consult environmental data records and flexibly query historical data according to time and data type.

[0440] (4) Data prediction module

[0441] The data prediction module aims to provide users with the predicted trends of environmental data over a period of time in the future. The system should utilize historical data and deep learning models to predict the future change trends of environmental data, so as to help users make decisions and adjustments.

[0442] (5) 3D Point Cloud Plant Phenotyping Analysis Module

[0443] The 3D point cloud plant characterization analysis module is designed to process 3D point cloud data of plants, calculate parameters such as plant height, crown width, and surface area, accurately analyze the growth status and morphological characteristics, and present them to users in an intuitive way.

[0444] 6.2 System Design

[0445] 6.2.1 System Function Design

[0446] The overall function design framework of the intelligent greenhouse monitoring system includes five major modules, namely the basic module, real-time data monitoring module, historical data query module, data prediction module, and 3D point cloud plant phenotyping analysis module. The system function design framework diagram is as Figure 48 shown.

[0447] The core functions of the basic module include user registration, login, and logout. The registration function provides a simple interface, requires filling in information such as username and password, and verifies to ensure the uniqueness of the username and the strength of the password. The registration information is securely stored in the MySQL database, the password is hashed to enhance security, and security mechanisms such as preventing SQL injection are also adopted. After successful login, the server saves the user login status to support subsequent operations and permission control. The logout function allows users to safely exit the system, clears the login status on the server side, and ensures the security of user information.

[0448] The real-time data monitoring module aims to ensure that users can obtain the latest data inside the greenhouse in a timely manner. The module regularly monitors key data, such as soil temperature, humidity, salinity, air temperature and humidity, light intensity, evaporation, CO2, and PM2.5 concentration, and guarantees the real-time and accuracy of the data with a frequency of updating once per hour. Data collection is completed through sensor devices, and after being processed and formatted, it is stored in the database. The module designs clear data display charts to help users understand the internal environment of the greenhouse in a timely manner.

[0449] The historical data query module facilitates users to query the historical environmental data of the greenhouse. The user interface is designed to be intuitive and simple, including date and data type selectors, and users can select query conditions according to their needs. The module efficiently queries the historical data in the database, visually displays it in the form of a curve graph, and also provides a data export function for subsequent analysis or sharing.

[0450] The data prediction module uses the WCACM prediction model proposed in Chapter 3 to predict the future change trend of greenhouse environment data. This model has been optimized and adjusted in the training stage and has a high prediction accuracy. Each time the module makes a prediction, it extracts the latest data from the database for prediction. The user interface design allows the input of the prediction time range and variable type. The prediction results are presented in the form of a curve graph, enabling users to clearly understand the trend and changes of future environment data.

[0451] The 3D point cloud plant phenotype analysis module aims to comprehensively analyze the morphological characteristics of plants. After obtaining the plant point cloud using the SFM-MVSNet method, users can select the local point cloud file to load it into the system. After loading, the point cloud model will be visually displayed, supporting zooming and dragging operations. The module provides functions such as downsampling, coordinate correction, feature segmentation, and denoising to process the data to improve its quality and accuracy. Subsequently, by using functions such as AABB and triangular meshing, key parameters such as plant height, crown width, and surface area are calculated, and the results are visually displayed to help users deeply understand the morphological characteristics of plants.

[0452] 6.2.2 System Architecture and Technical Route

[0453] (1) System Architecture

[0454] The system architecture design divides the system into the application layer, control layer, logical business layer, and data layer from top to bottom. Each layer communicates through interfaces to ensure the smooth operation of the system. The system architecture design is as Figure 49 shown.

[0455] The application layer is the top layer of the system, responsible for providing visible and directly operable functions and services to users. It usually includes the user interface and user interaction parts, presenting the functions and data of the system to users, receiving user input requests, and then passing these requests to the next layer, namely the control layer.

[0456] The control layer is the middle layer of the system, responsible for processing the requests passed from the application layer and passing commands and data to the business logic layer or data layer. The control layer usually includes functions such as request processing and routing, scheduling and coordination of business logic, and generation and return of responses. It is the bridge between the application layer and the business logic layer, coordinating the interaction and communication between various components.

[0457] The logical business layer is the core layer of the system, responsible for implementing the core business logic and functions of the system. It includes functions such as definition and execution of business rules, call and processing of algorithms, and processing and conversion of data. The complexity and diversity at this level directly affect the overall performance and efficiency of the system.

[0458] The data layer is the bottom layer of the system, responsible for managing and processing the system's data. It covers core functions such as data storage, reading, updating, and deletion, as well as data management and maintenance work.

[0459] (2) Technical route

[0460] The system development is mainly divided into front-end development, back-end development, and front-back end interaction design. The technical route of the system is as Figure 50 shown.

[0461] For the front-end development of the system, technologies such as HTML, CSS, and JavaScript are used for page layout and interaction design. Combining the Bootstrap framework provides an aesthetically pleasing interface design and responsive layout, while using the jQuery library simplifies the front-end development process. To achieve data visualization, the system uses the ECharts chart library and uses the Open3D library for three-dimensional point cloud display. The comprehensive application of these technologies aims to provide users with a friendly interface and a smooth interaction experience.

[0462] For back-end development, Python is used as the main programming language, and the Flask framework is combined to build a Web application. Additionally, Node.js is used for asynchronous processing and event-driven programming, MySQL database is used to store and manage data, and the PyMySQL library is used to implement the interaction with the database. At the same time, deep learning models are integrated to meet the core function requirements of the system. The selection of these technologies aims to provide stable and efficient data processing and server-side logic to ensure the reliability and performance of the system.

[0463] The front-back end interaction of the system is achieved through HTTP requests and APIs. The front end sends requests to the back end, and the back end returns data in JSON format after processing according to the business logic. The Ajax technology enables asynchronous requests, updating data without page refresh, enhancing the user experience. Additionally, the WebSocket technology is adopted to establish long connections to support real-time data push, further enhancing the interactivity and real-time nature of the system. Furthermore, the RESTful API design style is adopted to simplify the interface design, improving the communication efficiency and maintainability of the system.

[0464] 6.2.3 Database design

[0465] In this system, the design of the database is a key link in system development, and MySQL is selected as the database platform for the system. As one of the basic modules of the system, the database undertakes the functions of providing data storage and query for the logical business layer to ensure the reliability and stability of data. The database mainly focuses on operations related to user login and registration, data storage, display, and data prediction. Considering the complexity and operating requirements of the system, multiple database tables are designed, and the structures of some basic data tables will be described in detail. The database structures corresponding to the system function design are shown in Tables 6.1 and 6.2.

[0466] Table 6.1 User Login and Registration Table

[0467]

[0468] Table 6.2 Basic Table of Environmental Data in the Greenhouse

[0469]

[0470] 6.3 System Implementation

[0471] The visualization interface is designed based on the PyCharm development environment, and the Bootstrap and Flask frameworks are mainly used to achieve the separation of the front and back ends. This design method realizes the decoupling of the front and back end applications and improves the development efficiency. As the back-end framework, Flask focuses on the processing of data logic and connects to the MySQL database to achieve data storage and management. Bootstrap is used to design and render the front-end interface, and the two cooperate with each other to achieve the interface presentation and function realization of the system.

[0472] 6.3.1 Registration and Login Module

[0473] The intelligent greenhouse monitoring system includes two user roles, namely system users and system administrators. Regardless of the user identity, the login process is completed through the same interface.

[0474] 6.3.2 Home Page Module

[0475] The main purpose of the home page module is to display the overall structure of the intelligent greenhouse monitoring system, and through the home page module, the various functions of the system can be quickly understood. The home page module mainly includes a real-time data monitoring module, a historical data query module, a data prediction module, and a three-dimensional point cloud plant characterization analysis module, etc.

[0476] 6.3.3 Real-time Monitoring Module

[0477] The main purpose of the real-time monitoring module is to ensure that users can obtain the data collected by various sensors in the greenhouse in a timely manner and present it to users in the form of a curve graph. At the same time, the system will also automatically store the collected data in the database for subsequent data query and analysis.

[0478] 6.3.4 Historical Data Query Module

[0479] The main purpose of the historical data query module is to enable users to conveniently view the historical greenhouse internal environment data. Users can select the variables and dates they want to view in order to view the data they are interested in. After selecting the variables and dates, users can click the "Query" button, and the interface will present the data in the form of a curve graph. In addition, users can also click the "Data Export" button to export the desired data as a file.

[0480] 6.3.5 Data Prediction Module

[0481] The main function of the data prediction module is to predict the trends and changes of various variables of the greenhouse internal environment at different future time periods. This module relies on the PyTorch framework of Python and incorporates a trained deep learning model into the system. Each time a prediction is made, the model extracts the latest data from the database as input. After the user selects the "Prediction Variable" and "Prediction Duration", and clicks the "Start Prediction" button, the system will perform the prediction operation and present the prediction results in the form of a curve graph on the interface. At the same time, after the prediction is completed, users can export the prediction data as a file by clicking "Data Export".

[0482] 6.3.6 3D Point Cloud Plant Phenotype Analysis Module

[0483] The main purpose of the 3D point cloud plant characterization analysis module is to comprehensively analyze the morphological characteristics of plants in the greenhouse to help users deeply understand the morphological characteristics of plants. The specific functions of this module will be introduced below.

[0484] (1) Point Cloud Loading

[0485] Clicking "Select File" in the option bar will pop up an interface where different types of point cloud files in the system can be read.

[0486] (2) Point Cloud Display

[0487] After successfully loading the point cloud, the 3D point cloud will be directly displayed on the page, and the loading result will be intuitively feedback to the message box. Clicking the left mouse button can rotate the point cloud model, clicking the right mouse button can pan the point cloud model, and scrolling the mouse wheel can zoom in and out of the point cloud model.

[0488] (3) Point Cloud Downsampling

[0489] When the user clicks "Point Cloud Downsampling" in the option bar, a message pop-up window will be triggered, prompting the user to enter the retention percentage. The user enters the downsampling percentage as needed and submits it. The system will perform uniform downsampling on the point cloud model and visualize the downsampled model on the interface, and the processing result will be intuitively feedback to the message box.

[0490] (4) Point Cloud Coordinate Correction

[0491] When the user clicks "Point Cloud Scale Correction" in the option bar, a message pop-up window will be triggered, prompting the user to enter the scale factor. The user enters the scale factor according to the actual value and submits it. The system will perform point cloud coordinate scale correction on the point cloud model and visualize the corrected model on the interface, and the processing result will be intuitively feedback to the message box.

[0492] After the user clicks "Point Cloud Z-axis Correction" in the option bar, the system will perform point cloud coordinate transformation correction on the point cloud model and visualize the corrected model on the interface, and the processing result will be intuitively feedback to the message box.

[0493] (5) Point Cloud Feature Segmentation

[0494] When the user clicks "Point Cloud Segmentation" in the option bar, a message pop-up window will be triggered, prompting the user to enter the HSV color threshold range. The user enters the threshold range according to the actual situation and submits it. The system will perform point cloud feature segmentation on the point cloud model and visualize the segmented model on the interface, and the processing result will be intuitively feedback to the message box.

[0495] (6) Point Cloud Denoising

[0496] There are still a small number of noise points after point cloud segmentation, so denoising operation needs to be performed on the point cloud. When the user clicks "Point Cloud Denoising" in the option bar, a message pop-up window will be triggered, prompting the user to enter the number of neighbors and the standard deviation ratio. The user enters the threshold range according to the demand and submits it. The system will perform statistical filtering denoising processing on the point cloud model and visualize the denoised model on the interface, and the processing result will be intuitively feedback to the message box.

[0497] (7) Calculation of Plant Height and Crown Width

[0498] When the user clicks "Calculate Length, Width and Height" in the option bar, the system will automatically calculate the crown width length, width and plant height of the plant point cloud, and display the calculation results in the message box.

[0499] (8) Calculation of Point Cloud Triangulation Area

[0500] When the user clicks "Point Cloud Meshing" in the option bar, a message pop-up window will be triggered, prompting the user to input the alpha parameter to control the triangulation degree. The user inputs the parameter according to the requirement and submits it. The system will perform triangular meshing on the point cloud model and load the triangulated model, and the processing result will be visually reflected in the message box.

[0501] After performing point cloud triangular meshing, when the user clicks "Calculate Surface Area" in the option bar, the system will automatically calculate the surface area of the plant point cloud and display the calculation result in the message box.

[0502] The above is only a specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be covered by the protection scope of the present invention.

Claims

1. A smart greenhouse monitoring system based on deep learning, characterized in that: include: Data acquisition module, data preprocessing module, main control module, prediction module, 3D reconstruction module, measurement module, and analysis module; The data acquisition module is connected to the data preprocessing module and is used to collect the crop growth environment data inside the greenhouse; The data preprocessing module is connected to the data acquisition module and the main control module and is used to preprocess the collected environmental data containing noise; The main control module is connected with the data preprocessing module, prediction module, 3D reconstruction module, measurement module and analysis module to control the normal operation of each module; The prediction module is connected to the main control module and is used to make short-term and medium-term predictions of the acquired environmental factor data through the WCACM combined neural network prediction model; A three-dimensional reconstruction module, connected to the main control module, is used to perform three-dimensional reconstruction of plants or crops in the greenhouse through the SfM-MVSNet neural network three-dimensional reconstruction model; A measuring module, connected to the main control module, is used to measure and extract plant phenotypic parameter information; The analysis module is connected to the main control module and is used to analyze the needs of the monitoring system and design the system functions and architecture; The prediction module method: Using the WCACM combined neural network prediction model, the data is first processed through wavelet packet denoising and complete population empirical mode decomposition, and then the AutoFormer neural network model is used for short- and medium-term predictions; finally, it is trained and tested on the greenhouse environmental factor data set; The three-dimensional reconstruction module method: Firstly, the camera’s internal and external parameters are estimated by using the incremental Structure from Motion (SfM) method through multi-view image acquisition. Then, the MVSNet deep learning framework is used to construct the 3D point cloud of the plant. Finally, the method is tested on the DTU public dataset and the self-made plant dataset.

2. The deep learning-based smart greenhouse monitoring system according to claim 1, characterized in that: The data preprocessing module method: Redundant value processing, outlier elimination, missing value filling and data standardization are performed on noisy environmental data.

3. The smart greenhouse monitoring system based on deep learning as claimed in claim 1, characterized in that: The measurement module method: Firstly, the 3D point cloud data of plants were preprocessed, including point cloud downsampling, coordinate correction, feature segmentation, and denoising operations. Then, the axis-aligned bounding box method was used to calculate the plant height and crown width parameters, and the triangulated meshing method was used to calculate the plant surface area parameters. Finally, 20 plants were selected for manual measurement and algorithm measurement comparison experiments.

4. The smart greenhouse monitoring system based on deep learning as claimed in claim 1, characterized in that: The analysis module method: First, the needs of the monitoring system are analyzed, and the corresponding system functions and architecture are designed. Secondly, the front-end and back-end of the system are built based on the Bootstrap and Flask frameworks. Then, the deep learning algorithm is deployed into the system to build a complete greenhouse monitoring system covering real-time data monitoring, environmental factor prediction, and plant three-dimensional model phenotypic parameter measurement functions. Finally, the various functions of the system are functionally tested and verified.

5. A deep learning-based smart greenhouse monitoring method for implementing the deep learning-based smart greenhouse monitoring system according to any one of claims 1 to 4, characterized in that: The smart greenhouse monitoring method based on deep learning includes: Step 1: Collect the crop growth environment data inside the greenhouse through the data acquisition module; and pre-process the collected environmental data containing noise through the data pre-processing module; Step 2: The main control module uses the WCACM combined neural network prediction model through the prediction module to make short-term and medium-term predictions on the acquired environmental factor data; Step 3, using the SfM-MVSNet neural network 3D reconstruction model through the 3D reconstruction module to perform 3D reconstruction of the plants or crops in the greenhouse; Step 4, extracting plant phenotypic parameter information through measurement module; Step 5: Use the analysis module to analyze the monitoring system requirements and design the system functions and architecture.

6. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning as claimed in claim 5.

7. A computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning as described in claim 5.

8. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the smart greenhouse monitoring system based on deep learning as described in any one of claims 1-4.

Images