Intelligent greenhouse monitoring system based on deep learning

Abstract

The invention belongs to the technical field of greenhouse monitoring, and discloses an intelligent greenhouse monitoring system based on deep learning, and the system comprises a data collection module, a data preprocessing module, a main control module, a prediction module, a three-dimensional reconstruction module, a measurement module, and an analysis module. According to the method, a method combining SFM and deep learning MVSNet is adopted, reconstruction of a three-dimensional model of a plant or a crop is taken as a core, and phenotypic characteristics of the plant or the crop are extracted by performing preprocessing such as down-sampling, segmentation and noise reduction on the three-dimensional point cloud of the plant or the crop, so that phenotypic parameters of the plant or the crop are accurately acquired. According to the method, phenotypic parameters of plants or crops can be obtained more accurately and efficiently, reliable data processing method reference is provided for research in the fields of botany, agronomy and the like, the breeding process of the plants or crops is accelerated, the growth environment is optimized, development of agricultural production is promoted, and the production efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of greenhouse monitoring, and in particular relates to a smart greenhouse monitoring system based on deep learning. Background Art

[0002] Domestic and foreign research on greenhouse environmental factor prediction shows that traditional prediction models perform well in short-term prediction, but have limitations in long-term prediction and nonlinear sequence analysis. Artificial neural networks (ANN), radial basis function neural networks (RBF), long short-term memory neural networks (LSTM) and other methods have achieved certain results in improving prediction accuracy, but there are still problems such as initial weight sensitivity and high training complexity. In recent years, the combination of neural networks and traditional methods has significantly improved the prediction accuracy.

[0003] In terms of crop phenotyping based on three-dimensional data, binocular stereo vision, structured light, time-of-flight technology (ToF), lidar, multi-view stereo vision and other technologies are widely used. Binocular stereo vision relies on the light environment and the surface texture of the object, and it is difficult to accurately measure plant phenotypic parameters in complex environments. Structured light technology is easily affected by ambient light and has limited imaging distance. ToF technology is expensive and susceptible to noise. Although lidar technology has accurate measurement and high resolution, the equipment is expensive. Multi-view stereo vision three-dimensional reconstruction technology is highly accurate and lossless, but it has problems such as long reconstruction time and difficult geometric calibration. Summary of the invention

[0004] In response to the problems existing in the prior art, the present invention provides a smart greenhouse monitoring system based on deep learning.

[0005] The present invention is implemented in this way: a smart greenhouse monitoring system based on deep learning includes:

[0006] Data acquisition module, data preprocessing module, main control module, prediction module, 3D reconstruction module, measurement module, analysis module;

[0007] The data acquisition module is connected to the data preprocessing module and is used to collect the crop growth environment data inside the greenhouse;

[0008] The data preprocessing module is connected to the data acquisition module and the main control module and is used to preprocess the collected environmental data containing noise;

[0009] The main control module is connected with the data preprocessing module, prediction module, 3D reconstruction module, measurement module and analysis module to control the normal operation of each module;

[0010] The prediction module is connected to the main control module and is used to make short-term and medium-term predictions of the acquired environmental factor data through the WCACM combined neural network prediction model;

[0011] A three-dimensional reconstruction module, connected to the main control module, is used to perform three-dimensional reconstruction of plants or crops in the greenhouse through the SfM-MVSNet neural network three-dimensional reconstruction model;

[0012] A measuring module, connected to the main control module, is used to measure and extract plant phenotypic parameter information;

[0013] The analysis module is connected to the main control module and is used to analyze the needs of the monitoring system and design the system functions and architecture.

[0014] Further, the data preprocessing module method:

[0015] Redundant value processing, outlier elimination, missing value filling and data standardization are performed on noisy environmental data.

[0016] Further, the prediction module method:

[0017] Using the WCACM combined neural network prediction model, the data was first processed through wavelet packet denoising and complete population empirical mode decomposition. Then, the AutoFormer neural network model was used for short- and medium-term predictions. Finally, it was trained and tested on the greenhouse environmental factor dataset.

[0018] Furthermore, the three-dimensional reconstruction module method:

[0019] Firstly, the camera’s internal and external parameters are estimated by using the incremental Structure from Motion (SfM) method through multi-view image acquisition. Then, the MVSNet deep learning framework is used to construct the 3D point cloud of the plant. Finally, the method is tested on the DTU public dataset and the self-made plant dataset.

[0020] Further, the measurement module method:

[0021] Firstly, the 3D point cloud data of plants were preprocessed, including point cloud downsampling, coordinate correction, feature segmentation, and denoising operations. Then, the axis-aligned bounding box method was used to calculate the plant height and crown width parameters, and the triangulated meshing method was used to calculate the plant surface area parameters. Finally, 20 plants were selected for manual measurement and algorithm measurement comparison experiments.

[0022] Further, the analysis module method:

[0023] First, the needs of the monitoring system are analyzed, and the corresponding system functions and architecture are designed. Secondly, the front-end and back-end of the system are built based on the Bootstrap and Flask frameworks. Then, the deep learning algorithm is deployed into the system to build a complete greenhouse monitoring system covering real-time data monitoring, environmental factor prediction, and plant three-dimensional model phenotypic parameter measurement functions. Finally, the various functions of the system are functionally tested and verified.

[0024] Another object of the present invention is to provide a smart greenhouse monitoring method based on deep learning, including:

[0025] Step 1: Collect the crop growth environment data inside the greenhouse through the data acquisition module; and pre-process the collected environmental data containing noise through the data pre-processing module;

[0026] Step 2: The main control module uses the WCACM combined neural network prediction model through the prediction module to make short-term and medium-term predictions on the acquired environmental factor data;

[0027] Step 3, using the SfM-MVSNet neural network 3D reconstruction model through the 3D reconstruction module to perform 3D reconstruction of the plants or crops in the greenhouse;

[0028] Step 4, extracting plant phenotypic parameter information through measurement module;

[0029] Step 5: Use the analysis module to analyze the monitoring system requirements and design the system functions and architecture.

[0030] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning.

[0031] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the deep learning-based smart greenhouse monitoring method.

[0032] Another object of the present invention is to provide an information data processing terminal, which is used to implement the deep learning-based smart greenhouse monitoring system.

[0033] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0034] First, the present invention is based on the existing three-dimensional reconstruction method. It will adopt a method combining SFM and deep learning MVSNet, with the reconstruction of the three-dimensional model of plants or crops as the core. By downsampling, segmenting and denoising the three-dimensional point cloud of plants or crops, and embedding it into the agricultural Internet of Things system, the phenotypic characteristics are automatically extracted, thereby achieving accurate acquisition of plant or crop phenotypic parameters, changing the traditional monitoring method of three-dimensional reconstruction of plant or crop phenotypic parameters, and using a deep learning model to improve the traditional three-dimensional reconstruction of plants or crops. This method can more accurately, efficiently and automatically obtain plant or crop phenotypic parameters, providing a reliable data processing method reference for research in botany, agronomy and other fields, helping to accelerate the process of plant or crop breeding, and optimizing the growth environment, promoting the development of agricultural production and improving production efficiency.

[0035] First, the present invention uses the acquisition system to obtain the greenhouse environmental data, and cleans and normalizes the data. Secondly, the present invention proposes a WCACM combined neural network prediction model, which first processes the data through wavelet packet denoising and complete population empirical mode decomposition, and then uses the AutoFormer neural network model for short- and medium-term predictions. Finally, training and testing are performed on the greenhouse environmental factor data set. Qualitative and quantitative results show that this method outperforms other TransFormer-based prediction models.

[0036] The present invention proposes a plant 3D reconstruction method based on SfM-MVSNet and multi-view images, that is, dense 3D reconstruction of plants is achieved by combining the SfM algorithm and the deep learning algorithm MVSNet. First, through multi-view image acquisition, the incremental motion recovery structure method (SfM) is used to estimate the internal and external parameters of the camera. Then, the MVSNet deep learning framework is used to construct the 3D point cloud of the plant. Finally, this method is tested on the DTU public dataset and the self-made plant dataset. The experimental results show that this method is superior to other advanced methods and traditional methods in terms of plant reconstruction quality and reconstruction speed.

[0037] The present invention proposes a three-dimensional point cloud phenotyping method based on plants. First, the three-dimensional point cloud data of plants is preprocessed, and the method includes operations such as point cloud downsampling, coordinate correction, feature segmentation, and denoising. Then, the axis-aligned bounding box method is used to calculate the plant height and crown width parameters, and the triangulated meshing method is used to calculate the plant surface area parameters. Finally, 20 plants are selected for manual measurement and algorithm measurement comparison experiments. The experimental results show that this method can quickly and accurately extract plant phenotypic parameter information.

[0038] First, the present invention conducts a demand analysis on the monitoring system and designs the corresponding system functions and architecture. Secondly, the present invention implements the front-end and back-end construction of the system based on the Bootstrap and Flask frameworks. Then, in combination with the above research content, the deep learning algorithm is deployed into the system to build a complete greenhouse monitoring system covering functions such as real-time data monitoring, environmental factor prediction, and plant three-dimensional model phenotypic parameter measurement. Finally, the various functions of the system are functionally tested and verified, and the results show that the system can be further deployed and applied to the intelligent monitoring system of the agricultural Internet of Things.

[0039] This paper mainly studies the greenhouse environmental factor prediction model based on the WCACM neural network and the plant three-dimensional reconstruction method based on SfM-MVSNet and multi-view images, as well as the plant phenotypic parameter measurement method based on three-dimensional point cloud and the design and implementation of the greenhouse monitoring system. The WCACM neural network prediction model and the plant three-dimensional point cloud phenotypic measurement method reconstructed based on the SfM-MVSNet method are integrated into the greenhouse monitoring system to realize a multifunctional greenhouse monitoring system to meet the needs of modern agricultural development. The main research work includes the following aspects:

[0040] (1) First, the internal environmental data of the greenhouse is collected using the acquisition system. Then the collected data is cleaned and preprocessed to ensure the quality, comprehensibility and comparability of the data.

[0041] (2) The WCACM combined neural network prediction model is built by combining wavelet packet denoising and complete population empirical mode decomposition with AutoFormer, and the model is used to make short- and medium-term predictions for the collected data. The experimental results show that this model outperforms the standard TransFormer, LogTrans, InFormer, and the AutoFormer network model used alone in the prediction of the next 24, 96, 192, and 336 hours.

[0042] (3) High-precision dense 3D reconstruction of plants is achieved. First, the present invention uses the SfM algorithm to restore the internal and external parameters of the camera and the sparse 3D point cloud data of the plant. Next, combined with the deep learning algorithm MVSNet, by generating masks and depth maps and fusing them with pairing information, dense 3D reconstruction of plants is achieved. Experimental results show that the framework proposed in the present invention can quickly and efficiently generate high-quality plant 3D models compared to other methods.

[0043] (4) Design a plant phenotypic parameter extraction method based on a three-dimensional model. First, the plant three-dimensional point cloud model is preprocessed, including point cloud downsampling, coordinate correction, feature segmentation, denoising and other operations. Then, the axis-aligned bounding box method is used to calculate the plant height and crown width, and the plant surface area parameters are calculated using the triangulated meshing method.

[0044] (5) We conducted a demand analysis on the monitoring system and designed the system functions and architecture. In combination with the above research contents, we built a complete monitoring system that covers functions such as data monitoring, environmental factor prediction, plant three-dimensional reconstruction, and phenotypic parameter measurement.

[0045] Second, the expected benefits and commercial value of the technical solution of the present invention after transformation are: for the whole country, the cost savings and increased output are at the level of tens of millions of yuan. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a structural block diagram of a smart greenhouse monitoring system based on deep learning provided by an embodiment of the present invention.

[0047] Figure 2 It is a flow chart of a smart greenhouse monitoring method based on deep learning provided in an embodiment of the present invention.

[0048] Figure 3 It is a technical roadmap provided by the embodiments of the present invention.

[0049] Figure 4 It is a greenhouse internal environment data transmission path diagram provided by an embodiment of the present invention.

[0050] Figure 5 It is a data cleaning flow chart provided by an embodiment of the present invention.

[0051] Figure 6 It is a filling principle diagram provided by an embodiment of the present invention.

[0052] Figure 7 It is a basic structure diagram of the BP neural network provided by an embodiment of the present invention.

[0053] Figure 8 It is a basic structure diagram of the RNN neural network provided by an embodiment of the present invention.

[0054] Fig. 9 This is a LSTM neural network unit diagram provided by an embodiment of the present invention.

[0055] Fig.10 It is a basic structure diagram of the TransFormer model provided by an embodiment of the present invention.

[0056] Fig.11 It is a basic end-to-end structure diagram provided by an embodiment of the present invention.

[0057] Fig.12 It is an implementation diagram of the Self-Attention mechanism provided by an embodiment of the present invention.

[0058] Fig.13It is a structural diagram of the AutoFormer model provided by an embodiment of the present invention.

[0059] Fig.14 2 are autocorrelation (left) and time-delayed aggregation (right) graphs provided by embodiments of the present invention.

[0060] Fig.15 It is a self-correlation attention and self-attention series diagram provided by an embodiment of the present invention.

[0061] Fig.16 It is a structural diagram of the WCACM prediction model provided by an embodiment of the present invention.

[0062] Fig.17 It is a graph of the experimental results of 24-hour prediction of air temperature by different models provided in the embodiments of the present invention.

[0063] Fig.18 This is a graph showing the 24-hour prediction experiment results of light intensity using different models provided in an embodiment of the present invention.

[0064] Fig.19 It is a graph showing the experimental results of 24-hour soil moisture prediction using different models provided in an embodiment of the present invention.

[0065] Fig. 20 It is a graph showing the experimental results of a 96-hour prediction of air temperature using different models provided in an embodiment of the present invention.

[0066] Fig.21 This is a graph showing the 96-hour prediction experiment results of light intensity using different models provided in an embodiment of the present invention.

[0067] Fig. 22 It is a graph showing the 96-hour prediction experimental results of soil moisture using different models provided in an embodiment of the present invention.

[0068] Fig.23 It is a graph showing the experimental results of a 192-hour prediction of air temperature using different models provided in an embodiment of the present invention.

[0069] Fig.24 This is a graph showing the experimental results of 192 hours of prediction of light intensity using different models provided in an embodiment of the present invention.

[0070] Fig.25 It is a graph showing the 192-hour prediction experiment results of soil moisture using different models provided in an embodiment of the present invention.

[0071] Fig.26 It is a graph showing the experimental results of 336-hour prediction of air temperature using different models provided in an embodiment of the present invention.

[0072] Fig. 27This is a graph showing the experimental results of 336 hours of prediction of light intensity using different models provided in an embodiment of the present invention.

[0073] Fig.28 It is a graph showing the 336-hour prediction experiment results of soil moisture using different models provided in an embodiment of the present invention.

[0074] Fig.29 This is an example diagram of a DTU data set provided by an embodiment of the present invention.

[0075] Fig.30 This is an example diagram of a self-collected plant dataset provided by an embodiment of the present invention.

[0076] Fig.31 It is a flow chart of SfM-MVSNet 3D reconstruction provided by an embodiment of the present invention.

[0077] Fig.32 It is a framework diagram of the incremental SfM algorithm provided by an embodiment of the present invention.

[0078] Fig.33 It is a diagram of the MVSNet network framework provided by an embodiment of the present invention.

[0079] Fig.34 It is a depth map fusion map provided by an embodiment of the present invention.

[0080] Fig.35 The following are examples of test results provided by the embodiments of the present invention: (a) 3D reconstruction quality comparison of Scan1 (b) 3D reconstruction quality comparison of Scan13 (c) 3D reconstruction quality comparison of Scan32 (d) 3D reconstruction quality comparison of Scan77.

[0081] Fig.36 The following are typical results of 3D reconstruction experiments of images provided by the embodiments of the present invention: (a) 3D reconstruction quality comparison of the plant Pothos sinensis (b) 3D reconstruction quality comparison of the plant Dieffenbachia cerasifera.

[0082] Fig.37 This is a diagram of the experimental results of three-dimensional reconstruction of self-collected images provided by an embodiment of the present invention.

[0083] Fig.38 1 is a diagram of a plant phenotype measurement framework provided by an embodiment of the present invention. (a) Original plant point cloud (b) Downsampled plant point cloud.

[0084] Fig.39 : This is a down-sampling diagram of a plant point cloud provided by an embodiment of the present invention. (a) Point cloud before coordinate transformation (b) Point cloud after coordinate transformation.

[0085] Fig.40 1 is a point cloud coordinate correction diagram provided by an embodiment of the present invention. (a) Before point cloud segmentation (b) After point cloud segmentation.

[0086] Fig.41 : This is a plant point cloud segmentation map provided by an embodiment of the present invention. (a) Point cloud before denoising (b) Point cloud after denoising.

[0087] Fig.42 This is a point cloud denoising image provided by an embodiment of the present invention.

[0088] Fig.43 This is a plant height and crown width extraction diagram provided by an embodiment of the present invention. (a) Plant point cloud (b) Point cloud triangulation.

[0089] Fig.44 It is a point cloud triangulated mesh diagram provided by an embodiment of the present invention.

[0090] Fig.45 This is a plant grid detail diagram provided by an embodiment of the present invention.

[0091] Fig.46 It is a plant height measurement comparison scatter plot provided by an embodiment of the present invention.

[0092] Fig.47 It is a surface area measurement comparison scatter plot provided by an embodiment of the present invention.

[0093] Fig.48 It is a system function design framework diagram provided by an embodiment of the present invention.

[0094] Fig.49 It is a system architecture design diagram provided by an embodiment of the present invention.

[0095] Fig.50 This is the system technology roadmap provided by the embodiment of the present invention. (a) User login page (b) System registration page.

[0096] In the figure: 1. Data acquisition module; 2. Data preprocessing module; 3. Main control module; 4. Prediction module; 5. 3D reconstruction module; 6. Measurement module; 7. Analysis module. DETAILED DESCRIPTION

[0097] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0098] like Figure 1 As shown, a smart greenhouse monitoring system based on deep learning provided by an embodiment of the present invention includes:

[0099] Data acquisition module 1, data preprocessing module 2, main control module 3, prediction module 4, three-dimensional reconstruction module 5, measurement module 6, analysis module 7.

[0100] The data acquisition module 1 is connected to the data preprocessing module 2 and is used to collect the crop growth environment data inside the greenhouse;

[0101] The data preprocessing module 2 is connected to the data acquisition module 1 and the main control module 3, and is used to preprocess the collected environmental data containing noise;

[0102] The main control module 3 is connected with the data preprocessing module 2, the prediction module 4, the 3D reconstruction module 5, the measurement module 6 and the analysis module 7, and is used to control the normal operation of each module;

[0103] The prediction module 4 is connected to the main control module 3 and is used to make short-term and medium-term predictions on the acquired environmental factor data through the WCACM combined neural network prediction model;

[0104] A three-dimensional reconstruction module 5 is connected to the main control module 3 and is used to perform three-dimensional reconstruction of plants or crops in the greenhouse through the SfM-MVSNet neural network three-dimensional reconstruction model;

[0105] The measuring module 6 is connected to the main control module 3 and is used to measure and extract plant phenotypic parameter information;

[0106] The analysis module 7 is connected to the main control module 3 and is used to perform demand analysis on the monitoring system and design system functions and architecture.

[0107] The data preprocessing module method provided by the embodiment of the present invention:

[0108] Redundant value processing, outlier elimination, missing value filling and data standardization are performed on noisy environmental data.

[0109] The prediction module method provided by the embodiment of the present invention:

[0110] Using the WCACM combined neural network prediction model, the data was first processed through wavelet packet denoising and complete population empirical mode decomposition. Then, the AutoFormer neural network model was used for short- and medium-term predictions. Finally, it was trained and tested on the greenhouse environmental factor dataset.

[0111] The three-dimensional reconstruction module method provided by the embodiment of the present invention:

[0112] Firstly, the camera’s internal and external parameters are estimated by using the incremental Structure from Motion (SfM) method through multi-view image acquisition. Then, the MVSNet deep learning framework is used to construct the 3D point cloud of the plant. Finally, the method is tested on the DTU public dataset and the self-made plant dataset.

[0113] The measurement module method provided by the embodiment of the present invention:

[0114] Firstly, the 3D point cloud data of plants were preprocessed, including point cloud downsampling, coordinate correction, feature segmentation, and denoising operations. Then, the axis-aligned bounding box method was used to calculate the plant height and crown width parameters, and the triangulated meshing method was used to calculate the plant surface area parameters. Finally, 20 plants were selected for manual measurement and algorithm measurement comparison experiments.

[0115] The analysis module method provided by the embodiment of the present invention:

[0116] First, the needs of the monitoring system are analyzed, and the corresponding system functions and architecture are designed. Secondly, the front-end and back-end of the system are built based on the Bootstrap and Flask frameworks. Then, the deep learning algorithm is deployed into the system to build a complete greenhouse monitoring system covering real-time data monitoring, environmental factor prediction, and plant three-dimensional model phenotypic parameter measurement functions. Finally, the various functions of the system are functionally tested and verified.

[0117] like Figure 2 As shown, a smart greenhouse monitoring method based on deep learning provided by an embodiment of the present invention includes:

[0118] S101, collecting crop growth environment data inside the greenhouse through a data collection module; and preprocessing the collected environmental data containing noise through a data preprocessing module;

[0119] S102, the main control module uses the WCACM combined neural network prediction model through the prediction module to perform short-term and medium-term predictions on the acquired environmental factor data;

[0120] S103, performing three-dimensional reconstruction of the plants or crops in the greenhouse using a three-dimensional reconstruction model of an SfM-MVSNet neural network through a three-dimensional reconstruction module;

[0121] S104, extracting plant phenotypic parameter information through measurement module;

[0122] S105, performing demand analysis on the monitoring system and designing system functions and architecture through the analysis module.

[0123] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning.

[0124] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the deep learning-based smart greenhouse monitoring method.

[0125] Another object of the present invention is to provide an information data processing terminal, which is used to implement the deep learning-based smart greenhouse monitoring system.

[0126] The main research contents of the present invention are divided into the following five parts:

[0127] (1) Data collection and preprocessing. An environmental factor data collection system was built using sensors, monitoring stations, gateways and other terminal equipment produced by Shenzhen Xinli Technology Co., Ltd. to collect crop growth environment data inside the greenhouse. The environmental data containing noise obtained by the collection system was preprocessed, mainly including redundant value processing, outlier elimination, vacancy filling and data standardization operations.

[0128] (2) Prediction of greenhouse environmental factors. A WCACM combined neural network prediction model is proposed, and the model is used to make short- and medium-term predictions of the environmental factor data obtained.

[0129] (3) 3D reconstruction of plants or crops. A SfM-MVSNet neural network 3D reconstruction model is proposed, which is used to perform 3D reconstruction of plants or crops in the greenhouse.

[0130] (4) Measurement of plant or crop phenotypic parameters. A method for extracting plant phenotypic parameters based on a three-dimensional model was designed. First, the three-dimensional point cloud model of the plant or crop was preprocessed, including operations such as point cloud downsampling, coordinate correction, feature segmentation, denoising, and triangulation. Then, based on the preprocessed model, the parameters of the plant or crop, such as plant height, crown width, and surface area, were calculated.

[0131] (5) Design and implementation of the smart greenhouse monitoring system. Conduct demand analysis on the monitoring system and design system functions and architecture. Combined with the above research content, a complete monitoring system covering data monitoring, environmental factor prediction, and measurement of phenotypic parameters of plant or crop three-dimensional models was constructed. Figure 3 Technology Roadmap

[0132] 2. Data collection and preprocessing

[0133] 2.1 Data Collection

[0134] Data acquisition and preprocessing are indispensable steps in data science and machine learning. They provide reliable and efficient training data for the model by ensuring data quality, eliminating bias, improving generalization ability, reducing computing costs, and improving features, thereby improving model performance and interpretability.

[0135] 2.1.1 Principle of data collection of internal environment of IoT greenhouse

[0136] The data collection process is as follows Figure 4 shown.

[0137] 2.1.2 Introduction to the lower computer data acquisition system

[0138] (1) Gateway device

[0139] The gateway used in this experimental acquisition system is XL90, and the XL90 intelligent gateway is the core of the intelligent sensor network.

[0140] (2) Monitoring station equipment

[0141] The XL68 environmental monitoring terminal also provides a variety of wireless RF module options, including 433MHz, LoRa, 2.4G, etc.

[0142] (3) Wireless communication module

[0143] LoRa was chosen as the method of wireless communication.

[0144] 2.2 Data Preprocessing

[0145] The main purpose of data preprocessing is to optimize the original data to meet the needs of the prediction model and improve the performance of the model. It usually includes data cleaning, data integration, data transformation and data reduction, aiming to improve the accuracy, stability and generalization ability of the prediction model and ensure that the research results are more credible and reliable.

[0146] 2.2.1 Data cleaning

[0147] The collected greenhouse environmental factor data will be cleaned up using three steps: redundant value processing, outlier elimination, and missing value filling. The data cleaning example process is as follows Figure 5 shown.

[0148] The redundant value processing, outlier removal and missing value filling in data preprocessing are introduced as follows:

[0149] (1) Redundant value processing: Redundant values ​​refer to data that appear multiple times in a data set but have the same or highly similar values ​​in different entries.

[0150] (2) Outlier elimination: Outliers refer to values ​​that are significantly deviated from other data in the data set.

[0151] Following are the detailed steps to identify outliers using MAD:

[0152] First, calculate the median of a certain environmental feature data set X. Here, the Median function is used to calculate the median. It is expressed as formula (2-1).

[0153] median=Median(X) (2-1)

[0154] Second, for each data point x in the dataseti Calculate its absolute deviation (absolute deviationi) from the median median, expressed as formula (2-2).

[0155] absolute deviation i =|x i -median| (2-2)

[0156] The above calculations yield an absolute deviation set (Absolute Deviation). Then, the median of all absolute deviation values ​​is calculated to obtain the median absolute deviation MAD, which is expressed as formula (2-3).

[0157] MAD=Median(Absolute Deviation)(2-3)

[0158] Finally, a threshold is set to judge outliers. In the experiment, if the absolute deviation of a data point exceeds three times the MAD of the median, it is considered an outlier.

[0159] (3) Gap filling: Gap filling is a method of estimating missing data points by using the relationship between known data points. Linear interpolation is usually used to fill in missing values ​​in a data set. The only problem in the data set after the two-step processing (1) and (2) is data gaps. Therefore, the present invention uses linear interpolation to fill in the gaps. The algorithm principle used is as follows:

[0160] If it is known that there are two gaps between points C and D that need to be filled, and the C value is less than the D value, the original data is as follows Figure 6 shown.

[0161] If points C and D are marked on a one-dimensional axis and two new data points are inserted between them so that the interval [C, D] is divided into three segments, each segment is of length portion, expressed as formula (2-4).

[0162]

[0163] Then insert the value data1 at position 2, expressed as formula (2-5).

[0164] data1=C+portion*1 (2-5)

[0165] Insert the value data2 at position 3, expressed as formula (2-6).

[0166] data2=C+portion*2 (2-6)

[0167] According to the previous settings, considering the case where the left endpoint of the missing data is C, the right endpoint is D, and the number of missing data in the middle is n, the following formula can be generalized: the interval formed by the two points C and D is divided into n+1 segments, and the length of each segment is represented by portion, which is expressed as formula (2-7).

[0168]

[0169] Then insert the value data of the kth vacant position k , expressed as formula (2-8).

[0170] data k =C+portion*k (2-8)

[0171] 2.2.2 Data Normalization

[0172] Data normalization methods include: Min-Max normalization, linear scaling transformation method and Z-Score normalization. All three methods can eliminate the impact of the dimension and magnitude of the data on subsequent analysis. The specific calculation method is as follows:

[0173] Min-Max normalization: as shown in formula (2-9).

[0174]

[0175] Linear scaling method: The core idea is to use linear transformation to scale the data proportionally. However, its disadvantage is that it is sensitive to extreme values ​​and may cause data distribution deviation, as shown in formula (2-10).

[0176]

[0177] Z-Score standardization: The core of this method is to map the original data to a standard normal distribution with a mean of 0 and a standard deviation of 1 for data transformation, which helps to eliminate the scale differences between different features and makes the importance of each variable approach the same after transformation, as shown in formula (2-11).

[0178]

[0179] In formula (2-9), (2-10), and (2-11), x i It represents the i-th data in the data set, min(x) is the minimum value of the data set, max(x) is the maximum value of the data set, mean(x) is the mean value of the data set, δ is the standard deviation of the data set, and yi is the result of different normalization of the data set.

[0180] The principle of wavelet packet denoising is as follows:

[0181] (1) Wavelet packet decomposition: Wavelet packet decomposition uses wavelet packet transform to decompose the original signal into sub-signals of different scales and frequency bands. It is similar to continuous wavelet transform and discrete wavelet transform, but wavelet packet transform is more flexible and can decompose the signal in more detail. x(t) is the original signal, Ψ j.k (t) is the wavelet packet basis function, d j,k is the decomposed wavelet packet coefficient, where j represents the scale (level) and k represents the frequency band at the scale. The decomposition is shown in formula (3-1).

[0182] d j,k =∫x(t)·ψ j,k (t)dt (3-1)

[0183] (2) Threshold processing: The series of wavelet packet coefficients d obtained by decomposition j,k Threshold processing is performed to reduce or zero the noise in the wavelet packet coefficients. Threshold processing is usually divided into hard threshold processing and soft threshold processing. In hard threshold processing, smaller coefficients are set to zero, while larger coefficients remain unchanged. In soft threshold processing, smaller coefficients are reduced to zero, while larger coefficients are reduced. The goal of this step is to reduce or zero the noise coefficient d. Soft threshold processing is shown in equation (3-2), and hard threshold processing is shown in equation (3-3).

[0184]

[0185] D j,k =sign(d j,k )·max(|d j,k |-T,0) (3-3)

[0186] Where D j,k is the wavelet packet coefficient after threshold processing, T is the threshold, sign(d j,k ) is d j,k The sign function is 1 for positive numbers, -1 for negative numbers, and 0 for zero.

[0187] (3) Signal reconstruction: Perform inverse wavelet packet transform on the sub-signal after threshold processing to obtain the denoised signal x denoised (t). The inverse wavelet packet transform can be obtained by iteratively merging the sub-signals, as shown in equation (3-4).

[0188]

[0189] The complete population empirical mode decomposition (CEEMD) steps are as follows:

[0190] The specific decomposition process is as follows:

[0191] ① In each iteration, a set of positive noise sequences with normal distribution is added to the original signal X(t) and negative noise sequence The calculation formulas are shown in formulas (3-5) and (3-6).

[0192]

[0193] ②Yes and Perform empirical mode decomposition separately to obtain two sets of IMFs + and IMFs - The decomposition result is shown in equations (3-7) and (3-8).

[0194]

[0195] ③ Take IMFs + With IMFs - The mean IMF is taken as the decomposition result of the complete overall empirical state decomposition. The calculation formula is shown in formula (3-9).

[0196] IMF=AVG(IMFs + +IMFs - ) (3-9)

[0197] The basic structure of BP neural network is as follows Figure 7 As shown, each node represents a neuron.

[0198] The LSTM neural network unit structure is as follows Fig. 9 shown.

[0199] LSTM neural network units replace the hidden units in the general recurrent network to form the Figure 8 The basic structure of an RNN neural network is shown, where LSTM units are connected to each other through recurrent links.

[0200] 3.2.4 TransFormer-based neural network Fig.10

[0201] To solve the inherent problems of traditional end-to-end models, the TransFormer model introduces a series of innovative structures and mechanisms. These include the following key innovations. The traditional end-to-end model consists of two parts: Encoder and Decoder, such as the RNN-based end-to-end LSTM model. The Encoder, as an encoder, can encode the input into a semantic vector; the Decoder, as a decoder, decodes the output passed by the Encoder. The model structure is as follows: Fig.11 shown.

[0202] (1) Self-Attention Mechanism:

[0203] The Self-Attention mechanism allows the model to focus on different parts of the input sequence at each time step, solving the problem that fixed-length semantic vectors cannot carry complete information and improving long-term memory capabilities. The implementation of the Self-Attention mechanism is as follows: Fig.12 shown.

[0204] The self-attention mechanism calculates the attention score through the weight matrix of Query (Q), Key (K) and Value (V). First, the query vector Q is multiplied by the key vector K, and then divided by the constant term To stabilize the calculation. Then, the normalized attention weights are calculated by the Softmax function. Finally, the normalized weights are multiplied by the value matrix V to obtain the final self-attention output. This process allows the model to weight different parts of the input in order to capture important information. The specific formula is expressed as formula (3-8).

[0205]

[0206] Where Q, K and V are matrices consisting of query vector, key vector and value vector respectively, k is the dimension of the input vector.

[0207] (2) Multi-headed Attention:

[0208] In the TransFormer model, 8 attention heads are usually used, each with its own set of weight matrices W i , multiply it with the input vector and then merge it. Finally, the merged attention representation is sent to the feedforward layer for the next step of calculation. The calculation formula of multi-head attention is expressed as formula (3-9) and (3-10).

[0209]

[0210] MultiHead(Q,K,V)=Concat(head1,…,head h )W o (3-10)

[0211] in, and is the parameter matrix of Q, K and V matrices, W 0 is an additional matrix.

[0212] (3) Positional encoding

[0213] Sine and cosine functions are used to generate position codes, and the calculation formulas are expressed as formulas (3-11) and (3-12).

[0214]

[0215] Among them, pos is the position in the sequence, i is the dimension index of the position encoding, and d model is the dimension of the model embedding vector.

[0216] (4) Feedforward Neural Networks and Layer Normalization

[0217] The Feed Forward Network (FFN) aims to provide nonlinear transformation to enhance the model's expressiveness, while layer normalization aims to normalize the input of each layer to accelerate training and improve model robustness, as expressed in formula (3-13).

[0218] FFN=max(0,xW1+b1)W2+b2 (3-13)

[0219] The TransFormer model uses a layer normalization method, and the calculation formula is expressed as formula (3-14).

[0220]

[0221] 3.3 AutoFormer Neural Network

[0222] 3.3.1 AutoFormer Model Structure

[0223] The AutoFormer model structure is as follows Fig.13 shown.

[0224] (1) Series decomposition block

[0225] The sequence decomposition block smoothes the periodic fluctuations in the sequence by adjusting the moving average. In this method, the window size of the moving average is dynamically adjusted according to the characteristics and needs of the sequence. The sequence decomposition block can better highlight the long-term trend of the sequence while reducing short-term fluctuations. For an input sequence X∈R of length L L ×d, the decomposition formula is shown in formula (3-15).

[0226]

[0227] Among them, X s and X t They refer to the seasonal component and the cyclical trend component in the time series, respectively. Both are input sequences R L×dIn addition, in order to maintain the consistency of sequence length, the model performs moving average processing through AvgPool(·) and performs corresponding filling operations. In this study, X s , X t = SeriesDecomp(X) to summarize the above equations, which constitute an in-model block.

[0228] (2) Model inputs

[0229] In the encoder, the input consists of data from the past I time steps, denoted as X en ∈R I ×d. Decomposition architecture such as Fig.13 As shown, the input of the AutoFormer decoder includes the season part X des and the trend cycle part X det , respectively belong to Each initialization consists of two parts: the first part comes from the encoder input X en The components decomposed from the second half have a length of Used to provide the most recent information; the second part comes from a scalar padding placeholder of length O. This setting allows the decoder to make full use of the past information provided by the encoder to better predict future sequences. This is shown in formula

[59] (3-16).

[0230]

[0231] Among them, X ens and X ent Respectively represent X en The seasonal and cyclical parts of X0 and X Mean Represented by zero and X en The placeholder filled with the mean value.

[0232] (3) Encoder

[0233] The overall calculation of the lth encoder layer is shown in formula (3-17).

[0234]

[0235] The specific details are shown in formula

[59] (3-18).

[0236]

[0237] Among them, "_" represents the trend part that is eliminated. represents the output of the encoder at layer l, Represents the embedded X en , They represent the seasonal components of the i-th sequence decomposition block in the l-th layer respectively.

[0238] (4) Decoder

[0239] like Fig.13 As shown in Figure 2. Each decoder layer has two autocorrelation functions: inner autocorrelation and encoder-decoder autocorrelation. Using the latent variables from the encoder The equation for the first encoder layer can be summarized as The decoder can be expressed as formula

[59] (3-19).

[0240]

[0241] in, represents the output of the lth decoder layer, From X des Embedded in for depth transformation, Indicates the initial value used for accumulation, and They represent the seasonal data component and trend cycle component after the i-th sequence decomposition block in the first layer, respectively, and W l,i ,i∈{1,2,3} represents the i-th extracted trend The final prediction result is the sum of the two refined decomposition components. Among them, W s Indicates the amount of seasonal data that will be deeply transformed Project to the target dimension.

[0242] 3.3.2 Autocorrelation mechanism

[0243] The autocorrelation mechanism discovers the periodic dependence of seasonal data by analyzing the autocorrelation of the series and merges similar subsequences through time-delay aggregation. Its structure is as follows: Fig.14 shown.

[0244] (1) Based on periodic dependence

[0245] Inspired by the theory of random processes, for a real discrete-time process {X t}, the autocorrelation R can be obtained by the following equation (3-20): XX (τ).

[0246]

[0247] R XX (τ) reflects {X t} and its τ-lag sequence {X t-τ The autocorrelation function R(τ) is used as the unstandardized confidence level for estimating the cycle length τ. Then, the most likely k cycle lengths τ1,…,τ are selected. k Based on these estimated period lengths, the periodic dependencies in the time series are established and can be weighted by the corresponding autocorrelation functions.

[0248] (2) Time-delayed aggregation

[0249] The confidence normalized by SoftMax is used to weight the aggregated subsequences. In the single-head case, the time series X of length L is passed through the projector to obtain the query Q, key K and value V, so that it can seamlessly replace the self-attention mechanism. The self-correlation mechanism process is expressed as formula

[59] (3-21).

[0250]

[0251] In this process, argTopk(·) is a function used to obtain the Topk autocorrelation parameters, where c is a hyperparameter. Q,K represents the autocorrelation relationship between sequences Q and K. Roll(X,τ) represents an operation on sequence X with a time lag τ, during which the operation reintroduces elements beyond the first position of the sequence to the last position of the sequence. For encoder-decoder autocorrelation models (such as Fig.14 As shown), the key K and value V come from the output of the encoder They will be resized to length O; the query Q comes from the previous block of the decoder.

[0252] (3) Efficient computing

[0253] Since the aggregation length is O(logL) sequence, the complexity of formula (3-21) and its subsequent (3-22) is O(LlogL). For the autocorrelation calculation formula (3-20), given the time series {X t}, R XX (τ) can be calculated by fast Fourier transform (FFT), and the calculation formula is expressed as formula

[59] (3-22).

[0254]

[0255] Where F represents FFT, F -1 represents the inverse operation of FFT, * represents the conjugate operation, S XX(f) Represent the autocorrelation in the frequency domain. Importantly, FFT allows the autocorrelation of the sequence for all time lags in {1,···,L} to be computed at once. Therefore, the autocorrelation computation can be achieved with O(LlogL) complexity.

[0256] (4) Attention Mechanism VS Self-Attention Mechanism Series

[0257] The difference between self-correlation attention and self-attention series is that their connection forms are different. Self-correlation attention shows a sequential connection, such as Fig.15 shown.

[0258] 3.4 Prediction model based on WCACM neural network

[0259] 3.4.1 Model structure

[0260] The overall structure of the WCACM greenhouse internal environment data factor prediction model proposed in this invention is as follows: Fig.16 As shown. First, the data of the previous four days (96 hours) are used as input to predict the data of the next 24 hours, 96 hours, 192 hours and 336 hours respectively. The algorithm flow of the model is as follows:

[0261] ① Wavelet packet denoising (WPD) is used to process the environmental factor time series after data preprocessing.

[0262] ② The time series obtained after WPD denoising is used as the input of complete population empirical mode decomposition (CEEMD) to decompose and obtain the IMF component.

[0263] ③ Input the IMF components after CEEMD decomposition into the AutoFormer neural network model for prediction.

[0264] ④The AutoFormer neural network model takes the data of the previous 4 days (96 hours) as input to predict the environmental factor data of the next 24 hours, 96 hours, 192 hours and 336 hours.

[0265] ⑤Finally, the predicted component signals are integrated and superimposed to obtain the final prediction result.

[0266] 3.4.2 Experimental plan

[0267] In the experiment, the data set was divided into training set, validation set and test set in a ratio of 7:1:2 for model training, validation and testing. Considering that the internal environmental data factors of each greenhouse are time series data, the experimental data were processed by sliding window sampling to complete the sequence prediction task. In addition, neural network models such as LSTM, TransFormer, LogTrans, InFormer and AutoFormer were used for experimental comparison with the WCACM model; and four days (96 hours) of data were used as input to predict the data of the next 24 hours, 96 hours, 192 hours and 336 hours respectively to evaluate their performance in task implementation.

[0268] 3.5 Experimental Results and Analysis

[0269] This section uses the WCACM model to predict the collected greenhouse internal environment dataset. Considering that the dataset contains multiple data types, the experiment will select air temperature, light intensity and soil moisture data as examples, and display and analyze the experimental results.

[0270] 3.5.1 Experimental setup

[0271] The experimental environment is configured as follows: NVIDIA A100 GPU is used, equipped with 80GB video memory, and the operating system is Ubuntu 20.04. In addition, this experiment selects PyTorch as the algorithm framework, and uses Python language to write the algorithm code in the experiment.

[0272] After many experiments and comparisons, the present invention sets the hyperparameters of the model as follows: the input sequence length is 96 steps, the prediction sequence lengths are 24, 96, 192 and 336 steps respectively, and the label length is 48 steps. The training batch size is 32 to ensure that the model processes an appropriate amount of data in each iteration, thereby maintaining training speed and stability. The learning rate is 0.0001 to ensure that the optimizer is stable when adjusting the model weights. The model uses the Adam optimizer, whose adaptive learning rate strategy adjusts the gradient between different layers to improve training efficiency and stability. The loss rate of the fully connected layer and other layers is 0.05 to prevent the model from overfitting and enhance generalization ability. The model dimension is 512, converting the input sequence data into a 512-dimensional internal representation to better capture the complex features in the data. The number of attention heads is 8, which improves the performance of the model in calculating attention. The number of encoder layers is 2 and the number of decoder layers is 1, ensuring that the model maintains a moderate depth when processing time series data. The model selects the GELU activation function to improve the nonlinear performance of the model.

[0273] 3.5.2 Model Performance Evaluation Metrics

[0274] In order to more accurately quantify and evaluate the prediction accuracy of the experimental results, the experiment of the present invention selects Mean Squared Error (MSE) and Mean Absolute Error (MAE) as the main evaluation indicators for verifying the performance of the model. MAE is often used to measure the prediction accuracy of the model. The smaller the value, the better, reflecting that the overall prediction error of the model is small. In contrast, MSE focuses more on evaluating the stability of the prediction error, is more sensitive to outliers, and the smaller the value, the better. The present invention focuses on the two indicators of MSE and MAE because they can comprehensively evaluate the prediction performance of the model. By analyzing these indicators, we can have a deeper understanding of the performance of the model in the time series prediction task, and further optimize and improve the model design. The two formulas are respectively expressed as formula

[60] (3-23) and formula

[61] (3-24).

[0275]

[0276] Among them, n represents the number of samples, yi represents the true value, Represents the predicted value.

[0277] In the next experiment, the prediction results of the test set in different network models are shown. The orange line represents the predicted value of the model, while the blue line represents the actual observed value.

[0278] 3.5.3 Comparative experiment of prediction model for the next 24 hours

[0279] After data preprocessing, the air temperature, light intensity, and soil moisture data sets each have 4728 data points, or 197 days. The hourly data of the 197 days are divided according to the division method in Section 3.4.2. They are applied to the same prediction task scenario, that is, to conduct prediction experiments on air temperature, light intensity, and soil moisture in the next 24 hours. Fig.17 , 18 As observed in Figure 19, the trend of the predicted data of the WCACM network model is basically consistent with that of the actual data, and the predicted results are closest to the real values. It should be noted that after the LSTM model was tested for the next 24 hours of prediction experiment, it can be clearly observed that there is a significant difference between the real value and the predicted value in the experimental results. Whether in the trend or in the extreme point, the deviation between the two is very obvious, and the gap is huge. This shows that the LSTM model fails to accurately fit the real value, that is, it is not suitable for short- and medium-term time series prediction tasks. Therefore, in the subsequent experimental presentation, the experimental results processed by the LSTM model will no longer be displayed and analyzed.

[0280] Table 3.1 Comparison of evaluation indicators of models

[0281]

[0282]

[0283] Table 3.1 shows the mean square error (MSE) and mean absolute error (MAE) results of different network prediction models on different data test sets. It can be seen from the table that the WCACM network model proposed in the present invention has lower MSE and MAE values ​​than other network models, performs best, and has better prediction effect.

[0284] 3.5.4 Comparative test of the prediction model for the next 96 hours

[0285] The data set division is the same as Section 3.4.2. They are applied to the same task scenario to conduct prediction experiments on air temperature, light intensity and soil moisture data for the next 96 hours. Fig. 20 , 21 As observed in Figure 22, the trends of the predicted data of the WCACM network model are basically consistent with those of the actual data, and the predicted results are closest to the true values.

[0286] Table 3.2 Comparison of evaluation indicators of each model

[0287]

[0288] Table 3.2 shows the mean square error (MSE) and mean absolute error (MAE) results of different network models on the test set. It can be seen from the table that the WCACM network model proposed in the present invention has lower MSE and MAE values ​​than other network models, performs best, and has better prediction effect.

[0289] 3.5.5 Comparison experiment of the prediction model for the next 192 hours

[0290] The data set division is the same as Section 3.4.2. They are applied to the same task scenario to conduct prediction experiments on air temperature, light intensity and soil moisture data for the next 192 hours. Fig.23 , 24 As observed in Figures 25 and 25, the trends of the predicted data of the WCACM network model are basically consistent with the actual data, and the predicted results are closest to the real values.

[0291] Table 3.3 Comparison of evaluation indicators of each model

[0292]

[0293] Table 3.3 shows the mean square error (MSE) and mean absolute error (MAE) results of different network models on the test set. It can be seen from the table that the WCACM network model proposed in the present invention has lower MSE and MAE values ​​than other network models, performs best, and has better prediction effect.

[0294] 3.5.6 Comparison experiment of models predicting the next 336 hours

[0295] The data set division is the same as Section 3.4.2. They are applied to the same task scenario to conduct prediction experiments on air temperature, light intensity and soil moisture for the next 336 hours. Fig.26 , 27 It is observed in Figure 28 that the trends of the predicted data of the WCACM network model are basically consistent with the actual data, and the predicted results are closest to the real values.

[0296] Table 3.4 Comparison of evaluation indicators of each model

[0297]

[0298] Table 3.4 shows the mean square error (MSE) and mean absolute error (MAE) results of different network models on the test set. It can be seen from the table that the WCACM network model proposed in the present invention has lower MSE and MAE values ​​than other network models, performs best, and has better prediction effect.

[0299] Multi-view stereo matching algorithms can be divided into four types: point cloud-based, voxel-based, surface evolution-based, and depth map prediction-based. The essence of 3D reconstruction based on depth map prediction is to use image information from multiple perspectives, calculate the disparity between corresponding pixels in the image, and then infer the depth information of each point in the scene, and finally generate a depth map. Compared with other methods, this method has stronger flexibility and scalability

[64] . Therefore, the present invention adopts a depth estimation method to perform multi-view stereo matching. Figure 3 Reconstruction.

[0300] Plane scanning is a computer vision technology used to reconstruct three-dimensional scene structures from two-dimensional images. Its essence is to divide the reference camera space into multiple parallel planes, scan each plane in turn from far to near, and then project and match each point to finally obtain the depth map of the reference view. During the scanning process, it is necessary to calculate the projection points of each point on different views and compare the similarity scores to determine the depth value of each point. In this way, the depth estimation of each pixel on the reference view can be achieved, and then the depth map can be obtained.

[0301] MVSNet (Multi-View Stereo Network) is a multi-view stereo matching algorithm based on deep learning. It uses a neural network model to directly learn the depth information of the scene from images of multiple views, and realizes end-to-end multi-view 3D reconstruction

[45] . Its basic principles will be introduced in detail in Section 4.3.

[0302] Compared with traditional multi-view stereo matching algorithms, MVSNet has the advantages of end-to-end learning, accurate depth estimation, and strong generalization ability. By directly learning depth information from images from multiple perspectives, the process can be simplified and efficiency can be improved. Deep learning models can learn richer feature representations from large amounts of data, so MVSNet can produce more accurate depth estimation results. In addition, the features trained by MVSNet have good generalization and can adapt to various new scenarios, showing strong versatility and adaptability

[45] .

[0303] 4.2 Dataset Preparation

[0304] The present invention uses the large-scale and publicly available DTU

[65] multi-view 3D reconstruction dataset as experimental training data. This dataset contains images of various objects taken under different lighting conditions and viewing angles, as well as the corresponding real 3D point cloud data. Researchers can evaluate the accuracy of the algorithm by comparing the algorithm output with the real point cloud data. The DTU dataset is widely used in the fields of computer vision and 3D reconstruction, providing a standardized evaluation platform for algorithm testing and optimization. Related examples include Fig.29 shown.

[0305] For the experiment, the dataset was divided into training set, validation set and test set in a ratio of 7:1:2, with 87 scenes used for training, 12 for validation and 25 for testing. In addition to using the DTU dataset, real data of plants in the greenhouse were also collected to further verify and compare the performance of the proposed method with the traditional multi-view 3D reconstruction method in actual scenes. Examples of self-collected datasets are as follows: Fig.30 shown.

[0306] 4.3 Introduction to plant 3D reconstruction method based on SfM-MVSNet

[0307] The overall process of plant 3D reconstruction based on SfM-MVSNet is as follows Fig.31 As shown in the figure, the incremental SfM algorithm and MVSNet deep learning algorithm are mainly used to reconstruct the three-dimensional models of different plants.

[0308] First, feature points are extracted from images taken at multiple angles and matched to establish correspondences between images. Next, the incremental SfM algorithm is used to infer camera parameters based on their correspondences, determine the position and posture of the camera, and obtain sparse 3D point cloud data of the plant. On this basis, the depth range can be determined based on the sparse point cloud model. Subsequently, the MVSNet deep learning algorithm framework is used to generate a depth map for each image and a mask for the area of ​​interest, combining the previously calculated camera internal and external parameters and depth range. Finally, by combining the previously obtained pairing information with the depth map, a dense 3D point cloud model of the plant is generated, which can more accurately present the morphology and structure of the plant.

[0309] 4.3.1 Sparse Reconstruction Based on Multi-View Views

[0310] The MVSNet (Multi-View Stereo Network) algorithm is a multi-view stereo matching method based on deep learning, which is used to reconstruct the depth information of the scene from images of multiple perspectives. Fig.32 shown.

[0311] MVSNet is a deep learning network that is essentially used for image depth estimation. It accepts images from reference and source perspectives as input and outputs depth maps for each reference perspective. Its essence is to generate point clouds by fusing the traditional MVS method with the depth map, rather than directly building a 3D model. Therefore, MVSNet completes end-to-end optimization of the point cloud reconstruction process on the depth map through end-to-end learning. Finally, the generated point cloud reconstruction can be compared using standard metrics.

[0312] First, a two-dimensional convolutional neural network (CNN) is used to extract features from the reference view and source view of the input network. Fig.33 As shown in the figure, the network uses a two-dimensional CNN structure with 8 layers to extract image features. To capture features of different scales, the stride of the 3rd and 6th layers is set to 2, and the stride of other layers is set to 1. At each scale, two convolutional layers are used to extract features from the output of the previous layer. The convolution result is then batch normalized (BN) and nonlinearly activated by the linear rectifier unit activation function (ReLU). Finally, the two-dimensional CNN network outputs a feature map of 32 channels. In addition, in order to improve learning efficiency, the feature extraction network of each image uses the same weight parameter.

[0313] After image feature extraction, the feature map generated by the 2D CNN network is combined with the camera parameters obtained by the incremental SfM algorithm. With the help of differentiable homography transformation technology, the feature maps of multiple source view images are mapped to the feature map of the reference view to construct the 3D matching cost. Assume that I1 is the reference view to be predicted. Represents the source view for multi-view stereo matching with I1. Each source view has a corresponding camera intrinsic parameter matrix K i , the rotation matrix R i and the translation matrix t. The present invention uses homography to transform each source view I i The corresponding feature map F i Switch to the reference view I i The corresponding feature map F1 is on the plane. The homography transformation formula is shown in formula

[45] (4-1).

[0314]

[0315] Where d is the uniformly sampled value within the depth range determined during the incremental SfM algorithm, and n1 is the direction of the main axis of the reference camera. For the feature map F1 corresponding to the reference view I1, a 3×3 unit matrix is ​​used as the homography transformation matrix. This setting ensures that the homography transformation process remains consistent and universally applicable when processing all views.

[0316] After homography transformation, all source views are integrated into the three-dimensional space to form a feature map, which is expressed as If the height and width of each source view are H and W respectively, the number of depth samples is D, and the feature map output by the 2D convolutional network contains F channels. Because two convolution operations with a stride of 2 are performed, the feature map volume V generated by the homography transformation is expressed as It should be noted that since the feature map of the reference image is used as a benchmark, the feature map volume for the reference view is directly copied from its feature map at each depth.

[0317] In order to enable the network to adapt to any number of input views, the present invention adopts a variance-based mapping relationship To calculate the matching cost body C, the calculation formula is shown in formula

[45] (4-2).

[0318]

[0319] Where N is the number of input views, is the mean of all feature maps. In the cost volume, the values ​​of each point at different depths and channels represent the variance of the points in the feature map of the corresponding depth and channel. The variance in the cost volume reflects the consistency of the depth prediction values ​​of feature points at different depths on a certain channel of the feature map. In short, the smaller the variance, the higher the consistency of the depth prediction of the points in the feature map at that depth, that is, the predicted depth value is more likely to be close to the true value. Therefore, by observing the variance of each point in the cost volume, the accuracy and reliability of the depth prediction of feature points at different depths can be evaluated, which helps to select the depth value that best matches the actual situation.

[0320] Image noise, mismatching, occlusion and reflection of real scenes, and geometric distortion may interfere with multi-view stereo matching. In order to reduce noise and unnecessary details, the cost volume C needs to be regularized to improve the accuracy and stability of depth estimation, and effectively reduce memory consumption and computational costs. The present invention regularizes the cost volume through a multi-scale 3D CNN network and aggregates neighboring information using an encoder-decoder structure. The network reduces the number of channels from 32 to 8, and the number of convolutional layers at each scale from 3 to 2, and finally outputs a single channel. Probabilistic normalization is used in depth to obtain the probability space P for depth estimation. For any point (x, y, d) on the probability space P, its corresponding value P (x,y,d) The probability that the pixel (x, y) corresponds to a depth value of d is P (x,y,d) , which reflects the confidence of this depth estimate.

[0321] Before constructing the depth map D, the depth is first estimated by analyzing the probability space P pixel by pixel. However, the depth sample value with the highest probability at each pixel cannot be simply selected as the depth estimate. This is because the depth sample values ​​are uniformly sampled, so the resulting depth estimate is discrete. In addition, since the argmax operation cannot be differentiated, the model cannot use error backpropagation to learn network parameters.

[0322] In view of the above problems, the softargmin operation is used to estimate the depth value. This operation essentially calculates the expected value in the depth direction to initialize the depth map. The softargmin operation is expressed as formula

[45] (4-3).

[0323]

[0324] Where d is the depth sampling value, P(d) is the probability estimate of the pixel at depth d, and d max is the maximum depth, d minis the minimum depth, and D is the expected value in the depth direction. Through the soft argmin operation, the model can achieve continuous depth estimation. And this operation is differentiable, so end-to-end network training can be performed through error back propagation to improve model performance.

[0325] Next, the depth map is optimized. After obtaining the initial depth map, there are still two problems. The first is that for the pixels in the background area of ​​the reference view, their probability distributions at different depth sampling values ​​are scattered and do not gather to an obvious peak. Therefore, the present invention uses the probability distribution of depth estimation to measure the confidence of the depth estimation result. The confidence of the depth estimation value of each point is calculated by selecting pixel points around the background area of ​​the reference view and considering the sum of the depth estimation probabilities corresponding to the four nearest depth sampling values ​​around these points. Next, points with confidence levels lower than a preset threshold (set to 0.8 in this experiment) are regarded as invalid points and are removed from the depth map. For the remaining valid points, their original depth estimation values ​​are multiplied by the corresponding confidence levels to obtain a depth map from which invalid points have been removed.

[0326] Second, to solve the problem that the initial depth map may be over-smoothed, the present invention uses a deep residual network to fine-tune the initial estimated depth map, with the reference view as the standard. First, the initial estimated depth map is connected to the reference view to form a four-channel input. Then, it is processed by a deep residual network, which contains three 32-channel convolutional layers and one 1-channel convolutional layer to perform residual learning and improve the accuracy of the depth map. This processing makes the boundaries of the depth map clearer.

[0327] After training the multi-view matching network, the depth map can be predicted for the reference view. However, directly fusing these depth maps may lead to redundancy. In order to obtain a more accurate 3D reconstruction model, the present invention combines the depth matching information generated in SfM, verifies the consistency of the depth value through projection and back-projection between the reference view and the 3D space, and fuses the depth maps of different views to generate a 3D point cloud. The fusion process is as follows: Fig.34 The calculation of depth map conversion to three-dimensional point cloud is shown in formula (4-4).

[0328]

[0329] Among them, X is the coordinate of the three-dimensional point cloud in space, P is the homogeneous coordinate form of the pixel point, and C i is the camera center coordinate of the current viewing angle, is the inverse matrix of the camera intrinsic parameters, is the rotation matrix transpose operation, and d is the depth value.

[0330] First, use formula (4-4) to convert C in the reference view iA pixel point captured by the perspective is projected into the three-dimensional space, and the three-dimensional point X is obtained based on its depth estimation value. Then X is re-projected to C i In the image of the adjacent shooting angle, the corresponding depth value d is calculated. If C i The original depth estimation value d corresponding to the adjacent shooting perspective images orig If the difference with the depth value d just calculated is within 1%, that is, it satisfies the constraints described in formula (4-5), then this point is retained, otherwise it is ignored. Finally, all depth maps are projected into three-dimensional space and merged into a three-dimensional point cloud.

[0331] ∣d orig -d| / d<0.01 (4-5)

[0332] In order to evaluate the performance of the proposed method in 3D reconstruction, the algorithm was first tested on the DTU test set, and the overall 3D reconstruction performance was objectively evaluated using three quantitative indicators, namely accuracy, completeness and comprehensive performance. In addition, the proposed method was compared with advanced methods based on deep learning (such as MVSFormer

[66] ) and traditional methods (such as VisualSFM

[67] , Bundler

[68] , OpenMVG

[69] , MVE

[70] , and Colmap

[71] ) to analyze their differences in visual effects, generated file size, and reconstruction time.

[0333] The test results show that the deep learning method used in the present invention is superior to traditional algorithms in terms of the integrity, accuracy and robustness of the reconstruction results. Although slightly higher than the current leading deep learning method in terms of evaluation indicators, the method of the present invention has significant advantages in terms of reconstruction file size and reconstruction time. By visually comparing the reconstruction results, it is found that the model effect obtained by the method of the present invention is basically equivalent to that of the current leading method. However, in practical applications, due to the large file size and long reconstruction time, the memory usage of the plant characterization analysis system is too high, the efficiency is low, and the reconstruction efficiency is not high. In contrast, the method of the present invention is more suitable for practical application in plant characterization analysis systems, further proving its superiority.

[0334] In order to evaluate the performance of the proposed method in a real environment, real data of plants from multiple angles in a greenhouse were collected and experimentally tested and compared with advanced methods based on deep learning and various traditional methods. The test results show that the proposed method performs equally well in a real environment.

[0335] The present invention uses the Matlab test script provided by the DTU dataset to evaluate the accuracy, completeness and comprehensive indicators of the 3D point cloud. The accuracy indicator determines the accuracy of the position by calculating the average distance from the reconstructed point cloud to the reference model. The completeness indicator evaluates the integrity of the model by measuring the average distance from the reference model to the reconstructed point cloud. The shorter the two distance indicators, the higher the accuracy and completeness. The comprehensive indicator, that is, the average of these two items, is used to comprehensively measure the multi-view Figure 3 The quality of the reconstructed model.

[0336] This paper tests the multi-view 3D reconstruction model based on deep learning on the DTU public dataset and self-made real dataset, and compares and analyzes it with other advanced deep learning methods and traditional methods. The following is an analysis of the test results.

[0337] (1) Test results of DTU dataset

[0338] After training the network model, the present invention first tested the model on the DTU test set. The three quantitative indicators of accuracy, completeness and comprehensive performance are used to objectively evaluate the overall 3D reconstruction performance of the algorithm. There are many reasons for choosing the DTU dataset for testing: although the DTU dataset lacks plant scenes, it contains various types of scenes and objects, and is widely representative and universal; it also provides rich real-world scene data and high-quality 3D reconstruction results, which can be used as the "true value" standard when testing the algorithm; it helps to evaluate the generalization ability of the model in diverse scenes and unknown objects, thereby verifying the versatility and robustness of the model. Examples of test results are as follows: Fig.35 As shown. The experimental data show that the reconstructed point cloud has an average accuracy of 0.396 mm and an average completeness of 0.527 mm, and its comprehensive performance reaches 0.462 mm. The results prove that the method proposed in the present invention has high completeness and accuracy in reconstructing the target object.

[0339] Subsequently, for scenes with different textures and non-diffuse reflections, the method of the present invention is compared with the advanced method based on deep learning (MVSFormer) and the traditional method based on VisualSFM, MVE, Bundler, Colmap, OpenMVG in terms of visual effect, storage space occupied by the generated file, and reconstruction time. Due to space limitations, the comparative experimental results show the comparison between the present invention and MVSFormer, VisualSFM, MVE, Bundler and Colmap. The reconstruction effect comparison under different scenes is shown in the figure below. Fig.36 As shown, the storage space occupied by the generated file and the reconstruction time are shown in Table 4.2.

[0340] according to Fig.36As shown in the figure, the method of the present invention shows obvious advantages over traditional methods when dealing with scenes with different textures and non-diffuse reflection characteristics. The reconstructed 3D model is superior to traditional methods in terms of density and details. Compared with advanced methods, the method of the present invention is clearer in details and has less noise. The experimental results show that the method of the present invention can achieve complete and accurate 3D reconstruction when dealing with various objects.

[0341] The speed of 3D reconstruction and the size of the storage space occupied by the files generated after reconstruction are both important indicators for evaluating the 3D reconstruction system. Under the experimental environment described in Section 4.4.3, the present invention uses the DTU test data set to test the method of the present invention with the advanced method based on deep learning (MVSFormer) and the traditional method based on VisualSFM, MVE, Bundler, Colmap, OpenMVG. The average time consumed by each method for 3D reconstruction of a single set of data and the average storage space occupied by the files generated after reconstruction are recorded and compared. The experimental results are shown in Table 4.2.

[0342] Table 4.2 Average reconstruction time and average size of generated files for a single scene using different methods

[0343]

[0344]

[0345] As can be seen from Table 4.2, for the average 3D reconstruction time of a single scene, the method of the present invention is far superior to other methods. Among the traditional 3D reconstruction methods, Colmap has the best reconstruction quality, but its reconstruction time is the longest; and compared with the method of the present invention, its reconstructed 3D point cloud is slightly sparse and lacks some details.

[0346] Among the average storage space sizes of files generated after reconstruction of a single scene, the files generated by the VisualSFM method occupy the smallest storage space. However, the three-dimensional point cloud it reconstructs is relatively sparse, of poor quality and low integrity. On the contrary, the files generated by the MVSFormer method occupy the largest storage space on average, which is about 18 times that of the method of the present invention. This will result in excessive memory usage and low efficiency in subsequent plant phenotyping analysis systems. In addition, the three-dimensional point cloud it reconstructs contains more noise, which will affect the detailed analysis of the point cloud.

[0347] According to the experimental results, the multi-view based on deep learning proposed in this paper Figure 3 The 3D reconstruction method not only has a shorter reconstruction time in the public dataset test, but also can generate a complete, dense, less cluttered and detailed 3D point cloud while keeping the generated reconstruction file occupying less storage space. Therefore, this method has strong potential for practical application.

[0348] (2) Test results of real scenarios (self-made dataset)

[0349] The present invention uses an ordinary monocular camera to shoot different plants in the greenhouse along a ring. The first set of data contains 150 images of potted green radish, and the second set of data contains 150 images of potted dill. Then, these image data are tested, compared and analyzed using the method of the present invention and the advanced method based on deep learning (MVSFormer) and traditional methods (VisualSFM, MVE, Bundler, Colmap and OpenMVG) to evaluate their performance in terms of visual effect, storage space occupied by generated files, and reconstruction time. Due to space limitations, the comparative experimental results only show the comparison between the present invention and MVSFormer, VisualSFM, MVE, Bundler and Colmap. Comparison of reconstruction effects under different plant scenes is shown in the following figure. Fig.37 As shown, the storage space occupied by the generated file and the reconstruction time are shown in Table 4.3.

[0350] Depend on Fig.37 It can be seen that the method of the present invention shows obvious advantages over traditional methods when dealing with plants in real scenes. The reconstructed three-dimensional model of plants is superior to traditional methods in terms of density and details. Compared with advanced methods, the details of the reconstructed three-dimensional model of plants are clearer in the method of the present invention, and the noise points are significantly less. It should be noted that advanced methods still have the situation of reconstruction failure. Experimental results show that the method of the present invention can achieve complete and accurate three-dimensional reconstruction in real plant scenes, is not restricted by strict reconstruction environment, and has high robustness.

[0351] The present invention uses a self-made plant data set to test the method of the present invention with an advanced method based on deep learning (MVSFormer) and a traditional method based on VisualSFM, MVE, Bundler, Colmap, OpenMVG. The average time consumed by each method for three-dimensional reconstruction of plant data and the average storage space occupied by the files generated after reconstruction are recorded and compared. The experimental results are shown in Table 4.3.

[0352] Table 4.3 Average reconstruction time and average file size of plant scenes by different methods

[0353]

[0354] As can be seen from Table 4.3, the method of the present invention is far superior to other methods in terms of the average 3D reconstruction time for a single plant scene. Colmap has the best reconstruction quality among traditional 3D reconstruction methods, but it takes the longest time to reconstruct. Compared with the method of the present invention, the plant leaf part of the 3D point cloud generated by Colmap has a large amount of white background noise, which obviously damages the details of the reconstruction result.

[0355] In terms of the storage space size of the files generated after the reconstruction of the plant scene, the files generated by the VisualSFM method occupy the smallest storage space. However, the reconstructed three-dimensional point cloud is extremely sparse, of poor quality, and low in completeness. The MVSFormer method, on the other hand, has reconstruction failures, and the generated files occupy the largest storage space on average, which is about 14 times that of the method of the present invention. The reconstructed file generated by the method of the present invention occupies 58MB, which shows that the method of the present invention can effectively control the file size and optimize storage resources while maintaining a high three-dimensional reconstruction quality.

[0356] According to the experimental results, the multi-view based on deep learning proposed in this paper Figure 3 In the test of the real plant scene dataset, the proposed 3D reconstruction method not only has a shorter reconstruction time, but also can generate a complete, dense, less cluttered and detailed 3D point cloud while keeping the generated reconstruction file occupying less storage space. Therefore, this method has great potential in practical applications.

[0357] 5.1 Framework of plant phenotyping methods

[0358] In this paper, a method for measuring plant phenotypic parameters based on three-dimensional point clouds is mainly introduced, which includes two modules: a point cloud processing module and a plant phenotypic measurement module. First, the point cloud processing module processes the acquired plant point cloud, including downsampling and coordinate correction operations to ensure that the growth direction of the plant point cloud is consistent with the z-axis direction of the coordinate axis. Then, the phenotypic part of the plant is extracted from the original point cloud data by using point cloud feature segmentation and point cloud denoising operations. Finally, the plant phenotypic measurement algorithm based on three-dimensional point clouds is used to complete the measurement of parameters such as plant height, crown width and surface area. The plant phenotypic measurement framework is as follows: Fig.38 shown.

[0359] 5.2 Plant point cloud preprocessing

[0360] The point cloud data after plant 3D reconstruction is relatively dense. In order to segment plant features more quickly and accurately and reduce processing difficulty, it is necessary to perform point cloud downsampling. In addition, since the reconstructed plant point cloud deviates from the real plant in spatial direction and scale, it is necessary to perform coordinate correction to eliminate these differences. Subsequently, in order to more accurately calculate the plant phenotypic data, the point cloud feature segmentation algorithm is used to segment the plant from the background and the ground. However, the segmented plant point cloud may still contain a small amount of noise, which needs to be denoised to eliminate interference.

[0361] 5.2.1 Point Cloud Downsampling

[0362] In order to efficiently measure plant phenotypic parameters, it is first necessary to process dense plant three-dimensional point cloud data. Directly performing coordinate correction, feature segmentation, and measuring phenotypic parameters on plant point clouds will consume a lot of time and memory resources. Therefore, a point cloud downsampling method is used to reduce the number of point clouds while retaining the outer contour of the plant point cloud to improve measurement efficiency. The present invention uses a uniform downsampling method to process plant point clouds, which maintains the distribution characteristics of the original point cloud and avoids information loss. The downsampling results are shown in Figure 1. Fig.39 shown.

[0363] When performing uniform downsampling, first determine the sampling rate r, which represents the proportion of the points that are finally retained in the original point cloud. Then, calculate the interval k based on the sampling rate r, that is, how many points in the original point cloud are selected to be retained, and the calculation formula is shown in formula (5-1).

[0364]

[0365] Where n is the number of points in the original point cloud, r is the sampling rate, Represents rounding x down. Then, traverse each point in the original point cloud, starting from the first point, and select one point every k points to keep. This operation ensures that the selected points are evenly distributed in the entire point cloud model, and the structure and characteristics of the original point cloud are retained to the maximum extent.

[0366] 5.2.2 Point cloud coordinate correction

[0367] (1) Coordinate scale correction

[0368] There is a certain proportional scaling relationship between the actual plant and its three-dimensional point cloud model. In order to accurately determine the ratio between them, the point cloud needs to be calibrated. The present invention calculates the proportional relationship based on the plant pot as a reference, as shown in formula (5-2).

[0369]

[0370] Among them, R is the actual ratio, H realis the actual flower pot height, H reconstructed is the height of the flower pot in the 3D point cloud.

[0371] (2) Coordinate transformation correction

[0372] Since there are deviations between the reconstructed point cloud and the real plant in terms of spatial direction and scale, in order to accurately measure the plant phenotypic parameters, the coordinates of the reconstructed plant point cloud need to be corrected to resolve the coordinate axis error between it and the actual plant growth direction. The rotation and translation matrix is ​​used to adjust the coordinates to ensure that the z-axis direction of the point cloud is consistent with the plant growth direction. The point cloud coordinate transformation correction result is shown in the figure below. Fig.40 shown.

[0373] The z-axis conversion correction process is:

[0374] ① Use the random sampling consensus (RANSNA) algorithm to identify the model in the point cloud data and detect the geometric shape of the ground.

[0375] ② After detecting the ground, the normal vector m of the ground can be obtained, where m represents the direction of the ground.

[0376] ③ Calculate the rotation angle θ between the ground normal vector m and the z-axis normal vector n (0,0,1). The calculation formula is shown in formula (5-3).

[0377]

[0378] ④ After obtaining the rotation angle and rotation axis, the present invention uses the Rodriguez rotation formula to calculate the rotation matrix

[72] , and the calculation formula is shown in formula (5-4).

[0379] R mt =E×cosθ+(m·n)×d×(1-cosθ)+(m·n)×sinθ (5-4)

[0380] Among them, R mt is the rotation matrix, E is the third-order unit matrix, θ is the rotation angle, and d=(d1, d2, d3) is the unit vector of m·n.

[0381] ⑤The last step is to apply the point cloud data to the rotation matrix so that the z-axis of the point cloud in space is corrected to be perpendicular to the ground, such as Fig.40 As shown in (b), the red line, green line and blue line in the figure represent the directions of the x-axis, y-axis and z-axis in space respectively. The point cloud coordinate transformation correction formula is shown in formula (5-5).

[0382] T a =R mt T o (5-5)

[0383] Among them, Ta is the plant point cloud after conversion, R mt is the rotation matrix, T o It is the original point cloud of plants.

[0384] 5.2.3 Plant point cloud feature segmentation

[0385] The main task of plant point cloud segmentation is to effectively separate the main part of the plant (trunk, leaves, etc.) from the background such as flower pots and the ground. When measuring plant phenotypes, removing background point cloud information can reduce unnecessary operations and improve accuracy.

[0386] The present invention uses a method based on point cloud color information to achieve plant point cloud segmentation. First, the RGB color space is converted to the HSV color space, which can better capture the information of the green part of the plant. In the HSV color space, S stands for saturation, V stands for value, and H stands for hue. The H channel can directly describe the type of color, making it easier to distinguish different colors. Therefore, for color-based segmentation tasks, HSV segmentation is usually more effective

[73] . Next, a histogram analysis is performed on the H (hue) channel in the HSV color space, which enables the highest peak of the green part of the plant to be found intuitively, and this peak will be used as the segmentation threshold, which can accurately capture the characteristics of the green part of the plant and achieve an accurate segmentation effect. This method can not only efficiently segment the main part of the plant from the overall point cloud, but also is not affected by factors such as the shape of the point cloud. Therefore, it has strong applicability and robustness. The segmentation results are as follows: Fig.41 shown.

[0387] 5.2.4 Point Cloud Denoising

[0388] The denoising result is as follows Fig.42 shown.

[0389] 5.3 Plant 3D phenotypic parameter extraction

[0390] 5.3.1 Calculation method of plant height and crown width

[0391] The present invention uses the axis-aligned bounding box (AABB) method to calculate the plant height and canopy parameters of the plant point cloud model. The principle is to use an axis-aligned rectangular bounding box to approximate the boundary range of the plant point cloud model. The process of calculating the AABB is to find the minimum and maximum coordinate values ​​in the point cloud data, and then determine the position and size of the bounding box according to these coordinate values. After determining the AABB bounding box, the plant height and canopy information of the plant point cloud model can be extracted according to the size of the bounding box. Fig.43 Schematic diagram of plant height and crown width parameter extraction.

[0392] 5.3.2 Calculation method of plant surface area

[0393] The specific triangulation process is as follows:

[0394] ① Determine the Alpha shape parameter: First, you need to select an appropriate Alpha shape parameter (α). This parameter determines the size of the neighborhood around each point, which affects the density of the generated triangular mesh. Usually, the value of α is adjusted according to the density of the point cloud and the required triangular mesh density. A larger α value will result in a sparse triangular mesh, while a smaller α value will result in a dense triangular mesh.

[0395] ② Construct Alpha shape: For each point, construct its Alpha shape. Specifically, for each point, construct a spherical neighborhood with a radius of α with the point as the center. All points in this spherical neighborhood are considered as points adjacent to the point.

[0396] ③ Determine the connection relationship: Next, it is necessary to determine the connection relationship between all points within the Alpha shape of each point. The present invention is accomplished by performing Delaunay triangulation. Delaunay triangulation is a method of generating triangles in a point set, in which no point is within the circumscribed circle of any triangle. By performing Delaunay triangulation, the connection relationship between points in the neighborhood of each point can be determined, thereby generating a triangular mesh.

[0397] ④ Generate triangular mesh: Finally, according to the determined connection relationship, the point cloud can be converted into a triangular mesh. For each three connected points, a triangle is created and added to the final triangular mesh. In this way, a triangular mesh model with a continuous surface can be generated from the plant point cloud.

[0398] Fig.44 is the result of point cloud triangulation. Fig.45 Observe that the plant surface is composed of multiple small triangles. The surface area of ​​the entire grid can be obtained by calculating the area of ​​the triangle and adding up the areas of all triangles. The specific steps are as follows:

[0399] ① Get the vertex coordinates of the triangle: For each triangle, get the coordinate information of the three vertices from the triangle mesh.

[0400] ② Calculate the edge vector and normal vector of the triangle: For each triangle, calculate the vectors of its three edges. Then, use the cross product to calculate the normal vector of the triangle based on the edge vector of the triangle. If the vertices of the triangle are arranged in a counterclockwise direction, the normal vector is facing outward, otherwise it is facing inward.

[0401] ③ Calculate the area of ​​a triangle: Using the normal vector of a triangle and any two edge vectors, the area of ​​the triangle can be calculated. The area calculation formula is shown in formula (5-6).

[0402]

[0403] Among them, AB and AC are two side vectors of the triangle, × represents the cross product of the vector, and ∥·∥ represents the modulus of the vector.

[0404] ④ Accumulate the area of ​​each triangle: Add the areas of all triangles to get the surface area of ​​the entire mesh model.

[0405] 5.4 Plant phenotypic parameter measurement experiment and analysis

[0406] 5.4.1 Evaluation indicators

[0407] The plant height and surface area parameters were measured using the algorithm, and the results were compared and analyzed with the manual measurements. The root mean square error (RMSE), mean absolute percentage error (MAPE), and coefficient of determination (R 2 ) three evaluation indicators. RMSE represents the average deviation between the algorithm measurement value and the manual measurement value. The smaller the value, the higher the measurement accuracy. MAPE represents the average relative error between the algorithm measurement value and the manual measurement value, expressed as a percentage. The smaller the value, the higher the measurement accuracy. 2 It indicates the degree of correlation between the algorithm measurement value and the manual measurement value. The closer it is to 1, the higher the linear correlation between the two. Its calculation is shown in formula

[74] (5-7), formula

[75] (5-8) and formula

[76] (5-9).

[0408]

[0409]

[0410] Where n is the number of samples, x ai is the three-dimensional measurement data, x mi is artificial measurement data, Cov(X,Y) is the covariance of X and Y, Var[X] and Var[Y] are the variances of X and Y.

[0411] 5.4.2 Plant height measurement results

[0412] The present invention adopts two methods for plant height measurement: algorithm measurement and manual measurement. In the algorithm extraction, the present invention adopts the axis-aligned bounding box (AABB) method to calculate the plant height of the plant point cloud model. In order to compare the accuracy of the algorithm measurement, 20 plants were selected for the experiment, and the measurement results are shown in Table 5.1. The comparison results are shown in the scatter plot, as shown in Table 5.1. Fig.46 As shown. The horizontal axis of the scatter plot represents the manual measurement value, the vertical axis represents the algorithm measurement value, and each data point represents the measurement result of a plant. The RMSE is 0.38, the MAPE is 2.47%, and the R 2 According to the analysis of the results, the values ​​of RMSE and MAPE are small, indicating that the deviation and error between the algorithm measurement value and the manual measurement value are small, indicating that the algorithm has a high measurement accuracy. 2 A value close to 1 indicates that there is a high linear correlation between the algorithm measurement and the manual measurement, which further verifies the accuracy and reliability of the algorithm measurement.

[0413] Table 5.1 Comparison of manual and algorithmic measurements of plant height

[0414]

[0415] 5.4.3 Plant surface area measurement results

[0416] The present invention also uses two methods, algorithmic measurement and manual measurement, for measuring the surface area of ​​plants. The algorithmic measurement method uses a triangular meshing method to reconstruct the surface of the plant point cloud data, generate a triangular mesh model, and then calculate the surface area of ​​the plant. In order to compare the accuracy of the algorithmic measurement, 20 plants were selected for the experiment, and the measurement results are shown in Table 5.2. A scatter plot is used to show the comparison results, as shown in Table 5.2. Fig.47 As shown. The horizontal axis of the scatter plot represents the manual measurement value, the vertical axis represents the algorithm measurement value, and each data point represents the measurement result of a plant. The RMSE is 11.87, the MAPE is 1.87%, and the R 2 is 0.98. According to the analysis of the results, the values ​​of RMSE and MAPE are small, indicating that the deviation and error between the algorithm measurement value and the manual measurement value are small, indicating that the algorithm has high measurement accuracy. At the same time, R2 is close to 1, indicating that there is a high linear correlation between the algorithm measurement value and the manual measurement value, which further verifies the accuracy and reliability of the algorithm measurement.

[0417] Table 5.2 Comparison of manual and algorithmic measurements of plant surface area

[0418]

[0419]

[0420] A method for extracting plant phenotypic parameters was designed for plant three-dimensional point clouds. For the plant point clouds obtained by three-dimensional reconstruction, point cloud downsampling, coordinate correction, feature segmentation and denoising were performed to extract plant phenotypic characteristics. Then, the axis-aligned bounding box (AABB) method was used to obtain plant height and crown width information, and the plant surface area was calculated by the point cloud triangulation method. Then, the obtained plant height and surface area parameters were compared with the manual measurement values, and the mean absolute percentage error, root mean square error and determination coefficient evaluation index were calculated to evaluate the effectiveness of the method of the present invention. The results show that the method can quickly and accurately extract plant phenotypic parameter information.

[0421] The present invention develops a smart greenhouse monitoring system. The system is designed to provide users with a convenient and fast smart agriculture monitoring platform. The system mainly uses Python programming language, combines Flask framework and Bootstrap framework, and integrates multiple third-party libraries such as Open3D, OpenCV, ECharts and PyMySQL to complete the development of the system. The system is deployed on the Windows operating system and adopts B / S mode. The functional modules mainly include real-time data monitoring of the internal environment of the greenhouse, historical data query, data prediction, and plant phenotyping analysis based on three-dimensional point cloud. The functional implementation and principle of each module of the system will be explained.

[0422] 6.1 System Requirements Analysis

[0423] System requirements analysis is a crucial step in system design and development. It helps to clarify user needs and expectations and provides a basis and direction for system design and implementation. Through system requirements analysis, the requirements of system functions, performance, interface design, etc. can be accurately defined, thus providing clear work goals. At the same time, system requirements analysis can also identify and solve potential problems and risks to ensure that user expectations are met.

[0424] 6.1.1 Non-functional requirements analysis

[0425] (1) Performance requirements

[0426] In terms of performance requirements, the system needs to be able to respond quickly to user requests, especially for the acquisition of real-time data and prediction results, and ensure low latency. At the same time, the system should have good concurrent processing capabilities, be able to effectively handle multiple user requests under high load, and maintain system stability and smoothness. In addition, the system needs to make rational use of computing resources to improve system performance and throughput, while saving resource costs as much as possible.

[0427] (2) Safety requirements

[0428] In terms of security, the system should have a strict access control mechanism to ensure that only authorized users can access sensitive data and functions and prevent unauthorized access. In addition, the system needs to take measures to protect the confidentiality and integrity of user data, including encrypted storage, transmission and access control, to prevent data leakage or tampering. The system should also have the ability to prevent common network attacks to ensure the security and stability of the system.

[0429] (3) Maintainability requirements

[0430] In terms of maintainability, the system should be configurable, so that administrators can configure and adjust the system as needed to meet different business needs and environmental changes. At the same time, the system design should have good scalability to facilitate the addition of new functions or modules to meet the changing needs of users. The system should record logs of key operations and events to facilitate troubleshooting and system maintenance, ensuring the stability and reliability of the system.

[0431] (4) User experience requirements

[0432] In terms of user experience, the system interface should be simple, intuitive, and easy to operate, in line with user habits, reducing the user's learning cost and operation complexity. The system should support a variety of devices and screen sizes, ensure good display effects on different devices, and enhance the user experience. The system should provide clear and complete user documentation and help information to facilitate users to understand and use the system, reducing the user's difficulty of use and communication costs.

[0433] 6.1.2 Functional Requirements Analysis

[0434] (1) Basic Module

[0435] The base module provides user registration, login, and logout functionality. Login verifies user credentials and allows access to the system. Logout functionality allows the user to exit the system safely.

[0436] (2) Real-time data monitoring module

[0437] The real-time data monitoring module ensures that users can obtain the latest data on the greenhouse environment. The monitoring frequency is updated every hour, including soil temperature and humidity, salinity, air temperature and humidity, light intensity, evaporation, CO2 and PM2.5 concentration.

[0438] (3) Historical data query module

[0439] The historical data query module allows users to easily access environmental data records and flexibly query historical data by time and data type.

[0440] (4) Data prediction module

[0441] The data prediction module aims to provide users with the forecast trend of environmental data in the future. The system should use historical data and deep learning models to predict the future trend of environmental data to help users make decisions and adjustments.

[0442] (5) 3D point cloud plant phenotyping module

[0443] The 3D point cloud plant characterization and analysis module is designed to process plant 3D point cloud data, calculate parameters such as plant height, crown width and surface area, so as to accurately analyze the growth status and morphological characteristics, and present them to users in an intuitive way.

[0444] 6.2 System Design

[0445] 6.2.1 System Function Design

[0446] The overall functional design framework of the smart greenhouse monitoring system includes five modules, namely the basic module, real-time data monitoring module, historical data query module, data prediction module and three-dimensional point cloud plant phenotype analysis module. Fig.48 shown.

[0447] The core functions of the basic module include user registration, login and logout. The registration function provides a simple interface, requires user name and password, and verifies to ensure the uniqueness of the user name and password strength. The registration information is securely stored in the MySQL database, the password is hashed to enhance security, and security mechanisms such as SQL injection prevention are adopted. After successful login, the server saves the user login status to support subsequent operations and permission control. The logout function allows users to safely log out of the system, clear the server login status, and ensure the security of user information.

[0448] The real-time data monitoring module is designed to ensure that users can obtain the latest data inside the greenhouse in a timely manner. The module regularly monitors key data such as soil temperature, humidity, salinity, air temperature and humidity, light intensity, evaporation, CO2 and PM2.5 concentrations, and updates them once an hour to ensure real-time and accuracy of the data. Data collection is completed through sensor devices, and is stored in the database after processing and formatting. The module is designed with clear data display charts to help users understand the internal environment of the greenhouse in a timely manner.

[0449] The historical data query module allows users to query greenhouse historical environmental data. The user interface is intuitive and concise, including date and data type selectors, allowing users to select query conditions as needed. The module efficiently queries historical data in the database and displays it intuitively in the form of a curve chart. It also provides a data export function for subsequent analysis or sharing.

[0450] The data prediction module uses the WCACM prediction model proposed in Chapter 3 to predict the future trend of greenhouse environmental data. The model has been optimized and adjusted during the training phase and has a high prediction accuracy. The module extracts the latest data from the database for each prediction, and the user interface design allows the input of the prediction time range and variable type. The prediction results are presented in the form of a curve chart, allowing users to clearly understand the trends and changes of future environmental data.

[0451] The 3D point cloud plant phenotyping module is designed to conduct a comprehensive analysis of plant morphological characteristics. After obtaining the plant point cloud using the SFM-MVSNet method, the user can select a local point cloud file to load into the system. After loading, the point cloud model will be displayed intuitively, supporting zooming and dragging operations. The module provides functions such as downsampling, coordinate correction, feature segmentation, and denoising to process data to improve quality and accuracy. Subsequently, by using functions such as AABB and triangulated meshing, key parameters such as plant height, crown width, and surface area are calculated, and the results are displayed intuitively to help users gain a deeper understanding of plant morphological characteristics.

[0452] 6.2.2 System Architecture and Technical Route

[0453] (1) System architecture

[0454] The system architecture design divides the system from top to bottom into application layer, control layer, logical business layer and data layer. The layers communicate through interfaces to ensure the smooth operation of the system. Fig.49 shown.

[0455] The application layer is the top layer of the system, responsible for providing visible, directly operable functions and services to users. It usually includes the user interface and user interaction parts, presenting system functions and data to users, receiving user input requests, and then passing these requests to the next layer, the control layer.

[0456] The control layer is the middle layer of the system, responsible for processing requests from the application layer and passing commands and data to the business logic layer or data layer. The control layer usually includes functions such as request processing and routing, scheduling and coordination of business logic, and generation and return of responses. It is a bridge between the application layer and the business logic layer, coordinating the interaction and communication between various components.

[0457] The logical business layer is the core layer of the system, responsible for implementing the core business logic and functions of the system. It includes the definition and execution of business rules, the call and processing of algorithms, the processing and conversion of data, etc. The complexity and diversity of this layer directly affects the overall performance and efficiency of the system.

[0458] The data layer is the bottom layer of the system and is responsible for managing and processing the system's data. It covers core functions such as data storage, reading, updating, and deleting, as well as data management and maintenance.

[0459] (2) Technical route

[0460] System development is mainly divided into front-end development, back-end development and front-end and back-end interaction design. The system's technical route is as follows: Fig.50 shown.

[0461] The front-end development of the system uses technologies such as HTML, CSS, and JavaScript for page layout and interactive design. The Bootstrap framework provides beautiful interface design and responsive layout, while the jQuery library simplifies the front-end development process. In order to achieve data visualization, the system uses the ECharts chart library and uses the Open3D library for 3D point cloud display. The comprehensive application of these technologies is designed to provide users with a friendly interface and smooth interactive experience.

[0462] Backend development uses Python as the main programming language and combines it with the Flask framework to build web applications. In addition, Node.js is used for asynchronous processing and event-driven programming, MySQL database is used to store and manage data, and PyMySQL library is used to interact with the database. At the same time, deep learning models are integrated to meet the core functional requirements of the system. The selection of these technologies is intended to provide stable and efficient data processing and server-side logic to ensure the reliability and performance of the system.

[0463] The front-end and back-end interactions of the system are realized through HTTP requests and APIs. The front-end sends requests to the back-end, and the back-end returns JSON format data after processing according to business logic. Ajax technology implements asynchronous requests, and data can be updated without page refresh, which improves the user experience. In addition, WebSocket technology is used to establish a long connection to support real-time data push, further enhancing the interactivity and real-time performance of the system. In addition, the RESTful API design style is adopted to simplify the interface design and improve the communication efficiency and maintainability of the system.

[0464] 6.2.3 Database Design

[0465] In this system, database design is a key link in system development, and MySQL is selected as the database platform of the system. As one of the basic modules of the system, the database is responsible for providing data storage and query functions for the logical business layer to ensure the reliability and stability of the data. The database mainly focuses on user login registration, data storage, display, data prediction and other related operations. Considering the complexity and operation requirements of the system, multiple database tables are designed, and the structure of some basic data tables will be described in detail. The database structure corresponding to the system function design is shown in Tables 6.1 and 6.2.

[0466] Table 6.1 User login registration form

[0467]

[0468] Table 6.2 Basic table of greenhouse environment data

[0469]

[0470] 6.3 System Implementation

[0471] The visual interface is designed based on the PyCharm development environment, and mainly uses Bootstrap and Flask frameworks to achieve front-end and back-end separation. This design method achieves the decoupling of front-end and back-end applications and improves development efficiency. Flask, as a back-end framework, focuses on data logic processing and connects to the MySQL database to achieve data storage and management. Bootstrap is used to design and render the front-end interface. The two work together to achieve the system interface presentation and function implementation.

[0472] 6.3.1 Registration and Login Module

[0473] The smart greenhouse monitoring system includes two user roles, namely system user and system administrator. Regardless of the user identity, the login process is completed through the same interface.

[0474] 6.3.2 Home page module

[0475] The main purpose of the homepage module is to display the overall structure of the smart greenhouse monitoring system. Through the homepage module, you can quickly understand the various functions of the system. The homepage module mainly includes real-time data monitoring module, historical data query module, data prediction module and 3D point cloud plant representation analysis module.

[0476] 6.3.3 Real-time monitoring module

[0477] The main purpose of the real-time monitoring module is to ensure that users can obtain the data collected by each sensor in the greenhouse in a timely manner and present it to users in the form of a curve chart. At the same time, the system will automatically store the collected data in the database for subsequent data query and analysis.

[0478] 6.3.4 Historical data query module

[0479] The main purpose of the historical data query module is to allow users to easily view historical greenhouse internal environmental data. Users can select variables and dates to view in order to view the data they are interested in. After selecting the variables and dates, users can click the "Query" button and the interface will present the data in the form of a curve graph. In addition, users can also click the "Data Export" button to export the desired data as a file.

[0480] 6.3.5 Data Prediction Module

[0481] The main function of the data prediction module is to predict the trends and changes of various variables in the greenhouse internal environment at different time periods in the future. This module relies on Python's PyTorch framework and integrates a trained deep learning model into the system. Each time a prediction is made, the model extracts the latest data from the database as input. After the user selects the "prediction variable" and "prediction duration", click the "Start Prediction" button, the system will perform the prediction operation, and present the prediction results in the form of a curve graph in the interface. At the same time, after the prediction is completed, the user can export the predicted data in the form of a file by clicking "Data Export".

[0482] 6.3.6 3D Point Cloud Plant Phenotyping Module

[0483] The main purpose of the 3D point cloud plant characterization analysis module is to conduct a comprehensive analysis of the morphological characteristics of plants in the greenhouse, helping users to gain a deeper understanding of the morphological characteristics of plants. The specific functions of this module will be introduced below.

[0484] (1) Point cloud loading

[0485] Clicking "Select File" in the options bar will pop up an interface where you can read different types of point cloud files in the system.

[0486] (2) Point cloud display

[0487] After the point cloud is loaded successfully, the 3D point cloud will be directly displayed on the page, and the loading results will be intuitively fed back to the message box. Click the left mouse button to rotate the point cloud model, click the right mouse button to translate the point cloud model, and slide the mouse wheel to zoom in and out the point cloud model.

[0488] (3) Point cloud downsampling

[0489] When the user clicks "Point Cloud Downsampling" in the options bar, a pop-up message window will be triggered, prompting the user to enter the retention percentage. The user enters the downsampling percentage as needed and submits it. The system will uniformly downsample the point cloud model and visualize the downsampled model on the interface. The processing results will be intuitively fed back to the message box.

[0490] (4) Point cloud coordinate correction

[0491] When the user clicks "Point Cloud Scale Correction" in the option bar, a pop-up message window will be triggered, prompting the user to enter the scale factor. The user enters the scale factor according to the actual value and submits it. The system will correct the point cloud coordinate scale of the point cloud model and visualize the corrected model on the interface. The processing results will be intuitively fed back to the message box.

[0492] After the user clicks "Point cloud z-axis correction" in the option bar, the system will perform point cloud coordinate conversion correction on the point cloud model, and visualize the corrected model on the interface. The processing results will be intuitively fed back to the message box.

[0493] (5) Point cloud feature segmentation

[0494] When the user clicks "Point Cloud Segmentation" in the option bar, a pop-up message window will be triggered, prompting the user to enter the HSV color threshold range. The user enters the threshold range according to the actual situation and submits it. The system will perform point cloud feature segmentation on the point cloud model and visualize the segmented model on the interface. The processing results will be intuitively fed back to the message box.

[0495] (6) Point cloud denoising

[0496] After the point cloud is segmented, there are still a small amount of noise points, so the point cloud needs to be denoised. When the user clicks "Point Cloud Denoising" in the option bar, a message pop-up window will be triggered, prompting the user to enter the number of neighbors and the standard deviation ratio. The user enters the threshold range according to the requirements and submits it. The system will perform statistical filtering and denoising on the point cloud model, and visualize the denoised model on the interface. The processing results will be intuitively fed back to the message box.

[0497] (7) Calculation of plant height and crown width

[0498] When the user clicks "Calculate Length, Width and Height" in the option bar, the system will automatically calculate the crown length, width and plant height of the plant point cloud, and display the calculation results in the message box.

[0499] (8) Calculate area by triangulating point cloud

[0500] When the user clicks "Point Cloud Meshing" in the options bar, a message pop-up window will be triggered, prompting the user to enter the alpha parameter to control the degree of triangulation. The user enters the parameters as required and submits. The system will triangulate the point cloud model and load the triangulated model. The processing results will be intuitively fed back to the message box.

[0501] After triangulating the point cloud, the user clicks "Calculate Surface Area" in the option bar, and the system will automatically calculate the surface area of ​​the plant point cloud and display the calculation results in the message box.

[0502] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A smart greenhouse monitoring system based on deep learning, characterized in that: include: Data acquisition module, data preprocessing module, main control module, prediction module, 3D reconstruction module, measurement module, analysis module; The data acquisition module is connected to the data preprocessing module and is used to collect the crop growth environment data inside the greenhouse; The data preprocessing module is connected to the data acquisition module and the main control module and is used to preprocess the collected environmental data containing noise; The main control module is connected with the data preprocessing module, prediction module, 3D reconstruction module, measurement module and analysis module to control the normal operation of each module; The prediction module is connected to the main control module and is used to make short-term and medium-term predictions of the acquired environmental factor data through the WCACM combined neural network prediction model; A three-dimensional reconstruction module, connected to the main control module, is used to perform three-dimensional reconstruction of plants or crops in the greenhouse through the SfM-MVSNet neural network three-dimensional reconstruction model; A measuring module, connected to the main control module, is used to measure and extract plant phenotypic parameter information; The analysis module is connected to the main control module and is used to analyze the needs of the monitoring system and design the system functions and architecture.

2. The deep learning-based smart greenhouse monitoring system according to claim 1, characterized in that: The data preprocessing module method: Redundant value processing, outlier elimination, missing value filling and data standardization are performed on noisy environmental data.

3. The smart greenhouse monitoring system based on deep learning as claimed in claim 1, characterized in that: The prediction module method: Using the WCACM combined neural network prediction model, the data was first processed through wavelet packet denoising and complete population empirical mode decomposition. Then, the AutoFormer neural network model was used for short- and medium-term predictions. Finally, it was trained and tested on the greenhouse environmental factor dataset.

4. The smart greenhouse monitoring system based on deep learning as claimed in claim 1, characterized in that: The three-dimensional reconstruction module method: Firstly, the camera’s internal and external parameters are estimated by using the incremental Structure from Motion (SfM) method through multi-view image acquisition. Then, the MVSNet deep learning framework is used to construct the 3D point cloud of the plant. Finally, the method is tested on the DTU public dataset and the self-made plant dataset.

5. The deep learning-based smart greenhouse monitoring system according to claim 1, characterized in that: The measurement module method: Firstly, the 3D point cloud data of plants were preprocessed, including point cloud downsampling, coordinate correction, feature segmentation, and denoising operations. Then, the axis-aligned bounding box method was used to calculate the plant height and crown width parameters, and the triangulated meshing method was used to calculate the plant surface area parameters. Finally, 20 plants were selected for manual measurement and algorithm measurement comparison experiments.

6. The deep learning-based smart greenhouse monitoring system according to claim 1, characterized in that: The analysis module method: First, the needs of the monitoring system are analyzed, and the corresponding system functions and architecture are designed. Secondly, the front-end and back-end of the system are built based on the Bootstrap and Flask frameworks. Then, the deep learning algorithm is deployed into the system to build a complete greenhouse monitoring system covering real-time data monitoring, environmental factor prediction, and plant three-dimensional model phenotypic parameter measurement functions. Finally, the various functions of the system are functionally tested and verified.

7. A deep learning-based smart greenhouse monitoring method for implementing the deep learning-based smart greenhouse monitoring system according to any one of claims 1 to 6, characterized in that: The smart greenhouse monitoring method based on deep learning includes: Step 1: Collect the crop growth environment data inside the greenhouse through the data acquisition module; and pre-process the collected environmental data containing noise through the data pre-processing module; Step 2: The main control module uses the WCACM combined neural network prediction model through the prediction module to make short-term and medium-term predictions on the acquired environmental factor data; Step 3, using the SfM-MVSNet neural network 3D reconstruction model through the 3D reconstruction module to perform 3D reconstruction of the plants or crops in the greenhouse; Step 4, extracting plant phenotypic parameter information through measurement module; Step 5: Use the analysis module to analyze the monitoring system requirements and design the system functions and architecture.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning as claimed in claim 7.

9. A computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the smart greenhouse monitoring method based on deep learning as described in claim 7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the smart greenhouse monitoring system based on deep learning as described in any one of claims 1-6.

Images