Oat plant height estimation method and system based on machine vision
The oat plant height estimation system based on machine vision utilizes a depth camera and a dual-output regressive convolutional neural network to solve the problem of time-consuming and labor-intensive traditional measurement, and realizes automated and real-time accurate estimation of oat plant height. It is suitable for large-scale unmanned farms and intelligent spraying systems.
Patent Information
- Application Number
- CN202511719748.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-02-17
AI Technical Summary
Traditional crop height measurement is time-consuming, labor-intensive, and easily affected by subjective factors, making it difficult to automate and achieve real-time measurement.
An oat plant height estimation system was constructed using a machine vision-based approach, utilizing a depth camera and a dual-output regressive convolutional neural network. The system includes depth image acquisition, preprocessing, and model training. Modified EfficientNet V2 L was selected as the final estimation model, and automated estimation was achieved using LabVIEW software.
It enables automatic, real-time, and accurate estimation of oat plant height, reduces human intervention, improves efficiency, and is suitable for large-scale unmanned farm management and intelligent spraying systems, with broad application prospects.
Smart Images
Figure CN121545005A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of crop height measurement technology, specifically relating to a method and system for estimating oat plant height based on machine vision. Background Technology
[0002] Oats are a dual-purpose crop used for both food and feed, ranking third in global grain production after wheat, rice, and corn. Plant height is a key parameter in oat growth, directly impacting oat yield, disease incidence, and field management practices such as irrigation and fertilization.
[0003] In traditional crop production, crop height is usually measured by agricultural workers using a measuring tape, which is time-consuming and labor-intensive, increases agricultural labor costs, and the measurement results are easily affected by subjective factors.
[0004] Therefore, this study proposes a method and system for automatic and real-time acquisition and estimation of oat plant height parameters based on machine vision, which has important practical significance. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a machine vision-based method and system for estimating oat plant height, thereby at least partially solving the aforementioned problems.
[0006] To achieve the above objectives, the technical solution of the present invention is as follows:
[0007] This invention provides a machine vision-based method for estimating oat plant height, comprising:
[0008] S1. Construct an oat depth image acquisition system and acquire oat depth images and plant height data;
[0009] S2. After preprocessing the acquired depth image, it is input into a modified version of a variety of classic feature extraction networks to construct an initial dual-output regression convolutional neural network model for training and testing. ModifiedEfficientNet V2 L is selected as the final estimation model.
[0010] S3. At the same time, an oat plant height estimation system was built based on the LabVIEW software development platform to determine the average plant height and the highest plant height of the oats to be tested.
[0011] This invention utilizes stereo vision and a dual-output regressive convolutional neural network to estimate the height of oats in a field environment. The entire estimation process can be automated without human intervention, and the estimation speed can meet real-time requirements while the estimation accuracy can meet production requirements.
[0012] The beneficial effects of this invention are as follows:
[0013] 1. In this invention, the average and maximum plant height of oats can be automatically, in real time and accurately estimated by a dual-output regression convolutional neural network. There is no human intervention, and agricultural workers do not need to visually or measure the plant height of oats with a ruler in agricultural production. This can effectively avoid the influence of subjective factors on the plant height and growth status of oats and improve efficiency.
[0014] 2. In this invention, the oat plant height estimation system is suitable for large-scale unmanned farms, and can provide a basis for oat fertilization and irrigation. It can be integrated into oat harvesters to realize automatic real-time adjustment of the header height. It can also be used in intelligent spraying systems to adjust the nozzle height in a timely manner to improve the quality of spraying. It can also be extended to other crops such as wheat and soybeans for plant height estimation, and has broad application prospects. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0016] Figure 1 This is a schematic diagram of the oat plant height estimation method and system flow of the present invention;
[0017] Figure 2 This is a diagram of the oat depth image acquisition system of the present invention;
[0018] Figure 3 This is a diagram of the oat depth image acquisition program of the present invention;
[0019] Figure 4 This is a sample image of the oat depth image of the present invention;
[0020] Figure 5 This is a schematic diagram of oat depth image preprocessing according to the present invention;
[0021] Figure 6 This is a schematic diagram of the modified EfficientNet V2 L dual-output plant height estimation model of the present invention;
[0022] Figure 7 This is a schematic diagram of the training and testing process of the dual-output regression network method of the modified classic feature extraction network of the present invention.
[0023] Figure 8 This is a schematic diagram of the oat plant height estimation system of the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0025] This invention can be widely applied to estimating the growth of oats at different stages, such as the growth period, jointing stage, heading stage, grain-filling stage, and maturity stage. In practical applications, when detecting specific growth stages, the sample type used during model training should be adjusted accordingly. Furthermore, this invention can also be adaptively applied to estimating the height of other crops requiring height measurement, or the knowledge can be transferred and applied to other fields.
[0026] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:
[0027] Figure 1 The flowchart of the oat plant height estimation method and system provided by this invention is as follows: Figure 1 As shown, this embodiment of the invention provides a machine vision-based method and system for estimating oat plant height, including:
[0028] Step S101: Construct an oat depth image acquisition system.
[0029] Specifically, the system includes a RealSense D435 depth camera, a USB cable, an adjustable stand, and a laptop computer, and also includes an oat depth image acquisition program developed based on the LabVIEW software platform.
[0030] Specifically, Figure 2 A diagram of an oat depth image acquisition system, such as... Figure 2 As shown, the depth camera transmits image data to a laptop via a USB cable and is mounted on an adjustable stand. Considering the growth height of oats and the optimal accuracy range of the depth camera, the camera is adjusted to be 1.5m above the ground and parallel to the ground plane each time a depth image is acquired. At this height, the actual field of view of the camera is approximately 2.78m × 1.58m.
[0031] Specifically, Figure 3 A diagram of the oat depth image acquisition program, such as... Figure 3As shown, clicking the "Capture" button on the program interface allows you to acquire depth images of the oats. All acquired images are in "APD" format and have a resolution of 1280×720 pixels. After image acquisition is complete, clicking the "Stop" button will terminate the program. The value of each pixel in the depth image represents the distance from the object to the depth camera, measured in millimeters. In the image, redder colors indicate greater distance from the camera, while greener colors indicate closer proximity.
[0032] Step S102: Obtain oat depth images and plant height data.
[0033] Specifically, oat depth images were acquired using a constructed oat depth image acquisition system, and plant height parameters were measured using a measuring tape. Depth images and plant height data from 10cm to maturity (120cm) were acquired using a depth camera, and modeling and test data were collected at different shooting angles.
[0034] Specifically, in this study, oat plant height was defined as the distance from the ground to the highest point of the oat plant in its natural state. Five oat plants were randomly selected within the field of view of the depth camera, and their heights were measured. The average height of these five plants was taken as the average plant height of the oats within that depth image (hereinafter referred to as the 5-plant averaging method). The height of the tallest oat plant within the field of view was taken as the highest plant height in that depth image. To clarify the error when using the 5-plant averaging method to obtain the average plant height of oats, samples were taken from the image range at different growth stages of oats to compare the difference between the average plant height obtained using the 5-plant averaging method and the true average plant height.
[0035] Specifically, Figure 4 This is a sample image of a portion of the oat depth image, such as... Figure 4 As shown, each depth image is assigned two labels: its average plant height and its highest plant height. To apply this technology to field management equipment for oat plant height estimation, depth images were collected from specific sample types at different oat growth stages, and the angle between the camera and the oat rows was randomized, so that oat plant height could be estimated at any oat growth stage and from any sampling angle.
[0036] Specifically, the modeling data was further divided into training and validation data according to the K-fold cross-validation method (K was set to 5 in this study considering the amount of data collected). The training data was used to train the constructed dual-output regression convolutional neural network model. After each training iteration, the validation data was used to examine the effectiveness of the training, and the model parameters with the highest accuracy were saved. The test data was not used in model training. After all model training was completed, it was used to examine the generalization performance of the model.
[0037] Step S103: Preprocess the depth image.
[0038] It should be noted that preprocessing depth image data can improve model training efficiency. To ensure that the data preprocessing process does not affect the final training accuracy of the model, all data are preprocessed in the same way.
[0039] Specifically, Figure 5 This is a schematic diagram of oat depth image preprocessing, as shown below. Figure 5 As shown, the original depth image acquired has a resolution of 1280×720 pixels, and the pixel value of each point in the image represents the distance from the object to the depth camera in millimeters.
[0040] Specifically, to ensure that the depth image represents the true height of an object, depth restoration processing is performed on the original image. The calculation formula is as follows:
[0041]
[0042] In the formula This represents the pixel values in the depth-reconstructed image. These are the pixel values from the original image. At this point, the pixel values represent the object's true height, and the image is still in "APD" format. The pixel values represent the object's true height, ranging from 0 to 1500. In the image, redder colors indicate a higher height, and greener colors indicate a lower height.
[0043] To convert the image into a format recognizable by the TensorFlow deep learning platform, the image was further processed into grayscale. The calculation formula is as follows:
[0044]
[0045] In the formula The image consists of pixel values in a grayscale image. After grayscale conversion, the pixel value range becomes 0-255, at which point the image can be saved as a "jpg" file. The grayscale images are then uniformly scaled to 224×224 pixels to serve as input for each model. Finally, each image is labeled with two tags: the average and maximum oat plant heights in the image.
[0046] Step S104: Based on the modified classic feature extraction networks, an initial dual-output regression convolutional neural network model is constructed for training and testing.
[0047] It should be noted that this study uses a convolutional neural network to estimate the average and maximum oat plant height in a dual-output manner. Four lightweight classic feature extraction networks (MobileNet V3 Small, NasNet Mobile, RegNet Y002, and EfficientNet V2 B0) and four slightly larger networks (MobileNet V3 Large, NasNet Large, RegNet Y008, and EfficientNet V2 L) were selected and modified for oat plant height estimation.
[0048] It should be noted that all oat plant height estimation models are built on Python 3.9.16, TensorFlow-GPU-2.10.0 and Keras 2.10.0, and run on a desktop computer equipped with an Intel i7-12700k processor, 64GB of memory, an Nvidia GeForce RTX 3090 24 GB graphics card and a 64-bit Windows 11 system.
[0049] Specifically, Figure 6 This is a schematic diagram of the modified EfficientNet V2 L dual-output plant height estimation model, as shown below. Figure 6 As shown, the process of modifying the feature extraction model of this invention is explained in conjunction with the specific model structure. Each classic feature extraction network is a single-output classification network, so each model is modified: first, the last layer (classification layer) of the original network model is removed, and then two fully connected layers with single nodes and no activation function are added respectively. The two fully connected layers correspond to the outputs of the average plant height and the highest plant height of oats in the depth image.
[0050] It should be noted that the specific structure of the above model is only used as a specific example to illustrate the present invention. In the actual application of the present invention, the model structure can be adjusted according to actual needs, and the present invention does not limit it in this regard.
[0051] Specifically, Figure 7 This is a schematic diagram illustrating the training and testing process of the modified classic feature extraction network with dual-output regression network, as shown below. Figure 7As shown, training an oat plant height estimation model is essentially a process of continuously adjusting model parameters to make the estimation results increasingly accurate. Using K-fold cross-validation on small datasets allows the dataset to be divided into training and validation data multiple times for training, avoiding the impact of imbalanced data in a single split on the training results, thus effectively improving the model's learning ability and ultimately enhancing its generalization performance. Fine-tuning methods, on the other hand, can continue training the constructed model based on the parameters of the model trained on ImageNet, offering advantages such as fast training speed, low computational cost, and high-quality trained models.
[0052] Specifically, considering the scale and balance of the collected data, this study employed a 5-fold cross-validation and fine-tuning method for training. During training, the mean squared error (MSE) function was used to evaluate the accuracy of each model in estimating oat plant height and to select the optimal model. A lower MSE indicates higher model accuracy. The calculation formula is as follows:
[0053]
[0054]
[0055]
[0056] MSE A MSE M These represent the root mean square errors for estimating the average and highest plant heights of oats, respectively, where N is the number of validation data points. The average plant height of oats measured in the nth image. The average plant height estimated by the model in the nth image. This represents the highest measured plant height of oats in the nth image. The highest number of plants estimated by the model in the nth image.
[0057] It should be noted that, to ensure that the estimation performance of each model is evaluated under the same conditions, all models are trained using the same hyperparameters. The Adam optimization function, which has fast convergence speed and is easy to tune, and can automatically adjust the learning rate, is selected to adjust the parameters of the feature extraction model. Its configuration parameters are shown in Table 1, and all are default values with good training results. The batch size of each model is 32 during training, and each fold training is performed 300 times, retaining the model parameters with the minimum MSE.
[0058] Initial learning rate Weight decay coefficient First-order moment attenuation coefficient Beta-1 Second-order moment attenuation coefficient Beta-2 Denominator adjustment parameter Epsilon 0.001 0.01 0.9 0.999 1×10-8
[0059] Table 1. Adam Optimization Function Parameter Configuration Table
[0060] Specifically, after each model was trained, the optimal model was selected based on the lowest MSE (Mean Sequence Equation), and this model was retrained using all the modeling data. During the model retraining process, the Adam optimization function was used to adjust the model parameters, keeping their configuration unchanged. The model accuracy was evaluated using MSE, and the optimal model parameters were saved. After the optimal model was trained, the mean relative error (MRE), mean absolute error (MAE), root mean square error (RMSE), and mean estimation time (MET) for estimating the average and highest oat plant heights were examined using test data.
[0061] Step S105: Select Modified EfficientNet V2 L as the final estimation model.
[0062] Specifically, the minimum MSE and MET of each model in each training fold are shown in Table 2. All models achieved high estimation accuracy and could realize real-time estimation of oat plant height. Among them, the Modified EfficientNet V2 L model had the highest estimation accuracy, with an MSE of 0.002 in 5 fold training. Its average estimation time was the longest at 52.14ms, but it could still meet the requirements of real-time estimation. Therefore, it was selected as the final oat plant height estimation model.
[0063] Table 2. Minimum MSE and MET for each model during each training fold.
[0064] Estimation model Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Mean Squared Error (MSE) Mean Estimated Time (MET) Modified MobileNet V3 Small 0.046 0.051 0.075 0.184 0.131 0.098 8.53 ms Modified NasNet Mobile 0.054 0.038 0.035 0.262 0.063 0.090 46.02 ms Modified RegNet Y002 0.206 0.262 0.182 0.232 0.109 0.198 13.78 ms Modified EfficientNet V2 B0 0.036 0.021 0.052 0.064 0.035 0.042 28.59 ms Modified MobileNet V3 Large 0.034 0.040 0.523 0.016 0.011 0.125 12.05 ms Modified NasNet Large 0.065 0.007 0.008 0.423 0.231 0.147 50.12 ms Modified RegNet Y008 0.018 0.013 0.016 0.080 0.009 0.027 14.41 ms Modified EfficientNet V2 L 0.003 0.001 0.003 0.002 0.001 0.002 52.14 ms
[0065] It should be noted that, to examine the generalization performance of the Modified EfficientNet V2 L model in estimating oat plant height, the trained model was tested using test data not used in model training. The model's mean absolute error (MAE), root mean square error (RMSE), and mean relative error (MRE) for estimating average oat plant height were 2.30 cm, 2.90 cm, and 4.4%, respectively; for estimating the highest plant height, the MAEs were 2.24 cm, 2.82 cm, and 4.1%, respectively. The average estimation time was 52.14 ms. The accuracy of this method in estimating crop plant height is similar to existing methods, and the average estimation time meets the real-time requirements for crop plant height acquisition.
[0066] Step S106: Construct an oat plant height estimation system based on LabVIEW to determine the average plant height and the highest plant height of the oats to be tested.
[0067] Specifically, Figure 8 A schematic diagram of an oat plant height estimation system, such as... Figure 8As shown, this invention, based on LabVIEW software and combined with a depth camera and an oat plant height estimation model, constructs an oat plant height estimation system, achieving real-time, accurate, and automatic estimation of the average and maximum plant height of oats. After the depth camera acquires a depth image of the oats, the oat plant height estimation system can accurately estimate the average and maximum plant height within 0.1 seconds, without any human intervention. This system can provide a basis for crop fertilization and irrigation, can be integrated into oat harvesters to achieve automatic real-time adjustment of the header height, and can also be used in intelligent spraying systems to adjust the nozzle height in a timely manner to improve spraying quality, demonstrating broad application prospects.
Claims
1. A method and system for estimating oat plant height based on machine vision, characterized in that, The method comprises the following steps: S1, constructing an oat depth image acquisition system and obtaining oat depth images and plant height data. S2, after preprocessing the obtained depth images, inputting them into an initial double-output regression convolutional neural network model constructed based on a modified multiple classical feature extraction network for training and testing, and selecting Modified EfficientNet V2L as the final estimation model. S3, simultaneously constructing an oat plant height estimation system based on a LabVIEW software development platform to determine the average plant height and the maximum plant height of the oat to be measured.
2. The machine vision-based oat plant height estimation method and system of claim 1, wherein: In the step S1 of constructing the oat depth image acquisition system, the specific steps are as follows: the system comprises a RealSense D435 depth camera, a USB transmission line, an adjustable support and a notebook computer, and an oat depth image acquisition program is developed based on the LabVIEW software platform.
3. The oat depth image acquisition program developed based on the LabVIEW software platform according to claim 2, characterized in that: The oat depth images can be obtained by clicking the "Capture" button on the program interface, and the obtained images are in "APD" format and have 1280×720 pixels. After the image acquisition is completed, the program running can be terminated by clicking the "Stop" button.
4. The machine vision-based oat plant height estimation method and system of claim 1, wherein: In the step S1 of obtaining the oat depth images and the plant height data, the specific steps are as follows: the oat depth images are obtained by the constructed oat depth image acquisition system, and the plant height parameters are measured by a ruler. The depth images and the plant height data from 10 cm to the mature stage (120 cm) are collected by the depth camera, and the modeling data and the test data are collected under different shooting angles.
5. The machine vision-based oat plant height estimation method and system of claim 1, wherein: In the step S2 of preprocessing the obtained depth images, the specific steps are as follows: in order to make the depth images represent the true height of the object, the original images are subjected to depth restoration processing. After the grayscale processing, the image pixel value range becomes 0-255, at which time the images can be stored in "jpg" format. Then, the grayscale images are uniformly scaled to 224×224 pixels in size as the input of each model. Finally, two labels are given to each image, which are the average value and the maximum value of the oat plant height in the image.
6. The machine vision-based oat plant height estimation method and system of claim 1, wherein: In the step S2 of constructing the initial double-output regression convolutional neural network model based on the modified multiple classical feature extraction network, the specific steps are as follows: each classical feature extraction network is a single-output classification network, so each model is modified: firstly, the last layer (classification layer) of the original network model is removed, and then two single-node full connection layers without activation functions are added, and the two full connection layers correspond to the outputs of the average plant height and the maximum plant height of the oat in the depth image respectively.
7. The machine vision-based oat plant height estimation method and system of claim 1, wherein: In the step S2 of training and testing the initial double-output regression convolutional neural network model constructed based on the modified multiple classical feature extraction network, the specific steps are as follows: considering the size and balance of the collected data, the 5-fold cross-validation and fine-tuning methods are used for training and verification. In the training process, the Adam optimization function with automatic learning rate adjustment is selected for model parameter adjustment, the mean square error function (MSE) is used to evaluate the accuracy of the model in estimating the oat plant height, and the optimal model is selected.
8. The machine vision-based oat plant height estimation method and system of claim 1, wherein: The Modified EfficientNet V2 L is selected as the final estimation model in the step S2, and the specific steps are as follows: when the average plant height of oats is estimated, the average absolute error (MAE), the root mean square error (RMSE) and the average relative error (MRE) are 2.30 cm, 2.90 cm and 4.4% respectively; when the highest plant height is estimated, the average absolute error (MAE), the root mean square error (RMSE) and the average relative error (MRE) are 2.24 cm, 2.82 cm and 4.1% respectively; and the average estimation time of the model is 52.14 ms. The precision of the method for estimating the plant height of crops is similar to that of the existing method, and the average estimation time can meet the real-time requirement of obtaining the plant height of crops.
9. The machine vision-based oat plant height estimation method and system of claim 1, wherein: In the step S3, the oat plant height estimation system is constructed based on the LabVIEW software development platform to determine the average plant height and the highest plant height of the oat to be measured, and the specific steps are as follows: based on the LabVIEW software, the oat plant height estimation system is constructed by combining a depth camera and an oat plant height estimation model, and the real-time, accurate and automatic estimation of the average plant height and the highest plant height of oats is realized. After the depth camera collects the depth image of oats, the oat plant height estimation system can accurately estimate the average plant height and the highest plant height of oats within 0.1 s, and the whole process does not need human intervention.