A method and system for measuring the height of crop plants and swath width for a combine harvester and a harvester

Abstract

The present application provides a kind of combine harvester to be harvested crop plant height and swath measurement method and system and harvester, comprising the following steps: step S1, the depth image and RGB image of the front area to be harvested are synchronously collected, and the depth image and RGB image are aligned processing;Step S2, extract the region of interest, identify crop ear through image processing, and calculate plant height in combination with depth information;S3, using the instance segmentation model based on improved deep learning, identify the boundary line of the area that has been harvested and the area that has not been harvested in RGB image;S4, based on the pixel coordinates of boundary line, corresponding depth value and camera parameter, the real-time swath width is obtained by coordinate conversion calculation;S5, obtain inertial measurement unit data, dynamically estimate and correct the attitude angle of camera through sensor fusion algorithm, and the attitude angle is used for the calculation correction of plant height and swath width.The present application realizes the synchronous, real-time and robust measurement of crop height and swath width in complex field environment.

Description

Technical Field

[0001] This invention belongs to the field of agricultural machinery measurement and control and crop information sensing technology, specifically relating to a method and system for measuring the height of crop plants to be harvested and the cutting width of a combine harvester, and the harvester itself. Background Technology

[0002] Feed rate is a core factor determining the performance of rice combine harvesters. Currently, industry monitoring and control of feed rate mainly rely on post-processing measurements of machine forward speed, torque and speed of rotating components such as the feeding auger or threshing drum, and grain flow rate into the grain bin. This feedback control method based on existing feed material has significant lag. When the harvester encounters sudden changes in crop density, the control system cannot respond in time, easily leading to unstable threshing load, decreased separation performance, and even blockage of drums or conveyor troughs. In addition, existing header height control systems mostly rely on contact or ground proximity sensors, which only trigger adjustments when the header has already contacted the crop or the ground is uneven. They lack the ability to anticipate changes in the biomass of the crop ahead. To prevent the risk of blockage, conservative operating parameters are often forced during operation, resulting in the harvester's actual operating capacity not being fully utilized.

[0003] Crop height and cutting width are key parameters that directly determine the amount of material entering the machine per unit time. Accurate and timely measurement of these two parameters allows for proactive prediction of the feed rate, providing a basis for adjusting key operational parameters such as forward speed and threshing drum rotation speed, thus achieving stable, efficient, and intelligent harvesting. Regarding measurement technology, methods for crop height measurement include monocular vision-based scale estimation, laser scanning, and UAV remote sensing. While visual methods are easily limited by camera angle and reference objects, laser and UAV methods, although potentially more accurate, also face challenges such as high system costs, complex data processing, and difficulty in integrating them into high-speed, moving harvesters for real-time measurement and control. For the detection of cutting width, existing research mainly uses methods based on machine vision, lidar, or ultrasonic sensors. Machine vision methods often use the differences in color or texture between cut and uncut areas for identification, but they are easily affected by complex field lighting, crop lodging, and soil background interference, resulting in insufficient robustness. Lidar methods have high accuracy and are less affected by light, but their point cloud data processing computation is large and costly, making it difficult to popularize in large-scale agricultural equipment. Ultrasonic methods are easily affected by crop density and field airflow, resulting in poor measurement stability.

[0004] In summary, existing technologies still have several significant limitations when applied to real-time field monitoring and control of combine harvesters. First, crop height and cutting width measurements often require separate sensors, increasing system complexity and hardware costs. Second, most studies assume that the attitude angles of the measuring cameras or sensors are constant. However, in actual field operations, due to uneven ground, varying mud depth, and continuous machine vibration, the pitch and roll angles of the sensors change in real time. This attitude fluctuation directly alters the field of view, and without dynamic compensation, it will introduce significant errors into the measurement results. Furthermore, many image recognition-based models are trained only on specific rice varieties or single harvester models, and their generalization ability faces challenges when dealing with diverse varieties, growth states, and different equipment conditions. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for measuring the height of crop plants and the cutting width of a combine harvester, as well as the harvester itself. Through multi-source information fusion and dynamic correction, it achieves synchronous, real-time, and robust measurement of crop height and cutting width in complex field environments, providing a decision-making basis for forward-looking prediction of feed amount and proactive operation control.

[0006] This invention can adapt to complex and dynamic field environments, balance accuracy, real-time performance and cost-effectiveness, and can simultaneously acquire crop height and cutting width information. It is forward-looking and of great significance for improving the intelligence level of combine harvesters and realizing active operation control based on feed rate prediction.

[0007] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0008] A method for measuring the height of crop plants to be harvested and the cutting width of a combine harvester includes the following steps:

[0009] Step S1: Simultaneously acquire depth images and RGB images of the area to be harvested in front of the combine harvester using a depth camera, and align the depth images and RGB images.

[0010] Step S2: Based on the aligned image, extract the region of interest, identify the crop spike through image processing, and calculate the plant height by combining depth information;

[0011] S3. Using an instance segmentation model improved based on deep learning, identify the boundary lines between harvested and unharvested regions in the RGB image;

[0012] S4. Based on the pixel coordinates of the boundary line, the corresponding depth value, and the camera parameters, the real-time cutting width of the combine harvester is calculated through coordinate transformation.

[0013] S5. Obtain the inertial measurement unit data of the depth camera, and dynamically estimate and correct the camera's attitude angle through a sensor fusion algorithm. The attitude angle is used to calculate and correct the plant height and the cutting width.

[0014] In the above scheme, the calculation of plant height in step S2 specifically involves: converting the RGB image of the region of interest to the HSV color space for threshold segmentation to obtain the crop spike region; mapping the pixel coordinates of the crop spike region to the depth image to obtain distance information; and calculating the plant height H according to the following formula based on the camera installation height, camera pitch angle, and distance from the spike to the camera:

[0015]

[0016] Where H1 is the camera installation height, and L i Let θ be the distance from the ear of grain corresponding to the i-th pixel to the camera, θ be the camera pitch angle, and n be the number of effective pixels.

[0017] In the above scheme, the calculation of the cutting width in step S4 specifically involves:

[0018] The specific method for calculating the cut width is as follows: calculate the horizontal absolute distance w2 of the feature points on the boundary line relative to the depth camera, and combine it with the fixed distance w1 of the left divider relative to the camera to obtain the cut width w:

[0019] .

[0020] In the above scheme, the improved instance segmentation model mentioned in step S3 is the YOLO11-SCC model. Its structural improvements include: introducing the Swin Transformer module to replace the original C3k2 module in the Neck part of YOLO11, which is used to model global context information through self-attention mechanism; adding the Context Aggregation module in the Neck, which is used to fuse multi-scale context features to enhance boundary awareness; and replacing the original C3k2 module with the lightweight module CSPPC in the Backbone part, which is used to reduce redundant computation and improve feature representation ability.

[0021] Furthermore, the training of the YOLO11-SCC model includes the following steps:

[0022] S3.1. Collect RGB images containing unharvested areas under different conditions, manually label the unharvested areas in the images, and convert the labeling data into a label file in YOLO instance segmentation format;

[0023] S3.2 Perform data augmentation processing on the image and its corresponding label, including at least one of rotation, translation, scaling and horizontal flipping, and divide it into training set, validation set and test set according to a preset ratio;

[0024] S3.3. The YOLO11-SCC model is trained using a multi-task joint loss function, wherein the loss function is:

[0025] Where, λ box =7.5, λ cls =0.5, λ df1 =1.5, λ seg =1.0;

[0026] S3.4. Stochastic gradient descent is used as the optimizer to tune the model hyperparameters, with an initial learning rate of 0.001, a batch size of 32, a momentum coefficient of 0.937, and a weight decay coefficient of 0.0005.

[0027] S3.5. Evaluate the performance of the trained model based on the test set. The evaluation metrics include at least one of the following: accuracy, recall, mean precision, mean intersection-union ratio, total floating-point operations, number of parameters, and inference speed.

[0028] In the above scheme, the coordinate transformation in step S4 specifically involves: converting the pixel coordinates (u, v) and depth value d of the boundary point into a three-dimensional point in the camera coordinate system by combining the camera intrinsic parameters; then, transforming it to the vehicle coordinate system through the rigid body transformation matrix between the camera and vehicle coordinate systems to obtain the spatial position of the boundary point, and then calculating the cut width.

[0029] In the above scheme, step S5 includes the following sub-steps:

[0030] S5.1 Obtain triaxial angular velocity from the inertial measurement unit and triaxial acceleration ;

[0031] S5.2, Based on angular velocity about the Y-axis The original pitch angle estimate is obtained by integration. ;

[0032] S5.3. The static pitch angle estimate is obtained by calculating the direction of gravity based on accelerometer data. ;

[0033] S5.4 Construct an extended Kalman filter to define the system state as including the pitch angle. Gyroscope zero bias error The two-dimensional state vector is fused with gyroscope and accelerometer data, and the optimal pitch angle estimate is obtained through prediction and update steps.

[0034] S5.5 Similarly, the optimal estimate of the roll angle φ is obtained.

[0035] A system for measuring the height of crop plants to be harvested and the cutting width of a combine harvester, used to implement the method for measuring the height of crop plants to be harvested and the cutting width of a combine harvester, includes a depth camera, an inertial measurement unit, and an image processing module;

[0036] The depth camera is used to simultaneously acquire depth images and RGB images of the area to be harvested in front of the combine harvester;

[0037] The inertial measurement unit is integrated into the depth camera or set up separately, and is used to output triaxial angular velocity and triaxial acceleration;

[0038] The image processing module includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method for measuring the height of the crop to be harvested and the cutting width of the combine harvester according to any one of claims 1 to 7, including:

[0039] The depth image acquired by the depth camera is aligned with the RGB image. Based on the aligned image, the region of interest is extracted, and crop heads are identified through image processing. The plant height is calculated by combining the depth information. An instance segmentation model based on deep learning is used to identify the boundary line between the harvested and unharvested areas in the RGB image. Based on the pixel coordinates of the boundary line, the corresponding depth value, and the camera parameters, the real-time cutting width of the combine harvester is calculated through coordinate transformation. The inertial measurement unit data of the depth camera is acquired, and the camera attitude angle is dynamically estimated and corrected through a sensor fusion algorithm. The attitude angle is used to correct the calculation of the plant height and the cutting width.

[0040] In the above scheme, the depth camera is an Intel RealSense D435i depth camera, and the processor is a Jetson series embedded processor.

[0041] A harvester includes a system for measuring the height of the crop to be harvested and the cutting width of the combine harvester.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] This invention relates to a combine harvester that measures the plant height and cutting width of the crop during operation. A depth camera simultaneously acquires depth and RGB images of the harvested area. Through image alignment and color space conversion, the rice panicle region is extracted from the RGB image, and the plant height is calculated by combining this with the depth information. An improved YOLO11-SCC instance segmentation model is used to identify the boundaries between harvested and unharvested areas, and the cutting width is calculated through coordinate transformation. Integrated IMU data from the camera is used, and a sensor fusion algorithm is employed to estimate and correct the camera attitude angle in real time, eliminating measurement errors caused by terrain and vibration. This invention enables synchronous, real-time, and high-precision measurement of crop height and cutting width, providing reliable forward-looking perception information for the combine harvester's feed rate prediction and adaptive control.

[0044] This invention achieves simultaneous acquisition of depth and color information using a single depth camera. Combined with an improved YOLO-SCC model, it enables simultaneous and accurate extraction of plant height and cutting width, with low hardware cost and high system integration.

[0045] This invention introduces a dynamic attitude correction algorithm based on inertial measurement, which uses an extended Kalman filter to fuse IMU data in real time. This effectively suppresses measurement noise caused by machine vibration and terrain undulation, and helps to fundamentally eliminate the systematic errors caused by the traditional fixed attitude assumption, thus significantly improving the robustness and reliability of measurements in unstructured field environments.

[0046] The algorithm design of this invention fully considers the computing power of embedded platforms. By setting the region of interest, optimizing the sampling frame rate strategy, and using a lightweight model, it ensures the real-time performance of the entire system under limited resources, and provides reliable forward-looking perception information for the feed amount prediction and adaptive control of combine harvesters.

[0047] Compared to lidar or discrete multi-sensor solutions, this invention uses a depth camera combined with an embedded processor, which significantly reduces system costs while ensuring measurement accuracy, making it more suitable for large-scale agricultural equipment applications. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the installation of a depth camera and the corresponding coordinate system according to an embodiment of the present invention;

[0049] Figure 2 This is a flowchart illustrating the overall method of one embodiment of the present invention;

[0050] Figure 3 This is a flowchart of a plant height measurement algorithm according to one embodiment of the present invention;

[0051] Figure 4 This is a flowchart of the YOLO11-SCC-based slit width measurement process according to one embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the attitude correction algorithm of the extended Kalman filter (EKF) according to one embodiment of the present invention;

[0053] Figure 6 This is a system software and hardware module interaction block diagram according to an embodiment of the present invention;

[0054] Figure 7 This is a YOLO11-SCC model composition according to an embodiment of the present invention;

[0055] Figure 8 This is the loss function of the YOLO11-SCC model according to one embodiment of the present invention;

[0056] Figure 9 This is a diagram illustrating the boundary line between the cut and uncut regions determined using the YOLO11-SCC model according to an embodiment of the present invention. Figure 9 (a) is a wave-like boundary line; Figure 9 (b) is a gap-shaped boundary line; Figure 9 (c) is a bent boundary line; Figure 9 (d) is a straight boundary line;

[0057] Figure 10 This is a schematic diagram of coordinate transformation according to an embodiment of the present invention;

[0058] Figure 11 This is a schematic diagram illustrating the principle of slit width measurement according to an embodiment of the present invention;

[0059] Figure 12 This invention provides a systematic comparison and error analysis of measurement results of Huiliangyou Yuehe Simiao rice under conditions of direct sunlight, backlighting, cloudy, and partly cloudy skies, according to one embodiment of the invention. Figure 12 (a) Analysis of measurement errors in rice height and cutting width under direct sunlight conditions; Figure 12 (b) Analysis of measurement errors in rice height and cutting width under backlight conditions; Figure 12 (c) Analysis of measurement errors in rice height and cutting width under cloudy conditions; Figure 12 (d) Analysis of measurement errors in rice height and cutting width under cloudy conditions;

[0060] Figure 13 In one embodiment of the present invention, a systematic measurement and comparison of the cutting width of five rice varieties—Huiliangyou Yuehe Simiao, Jinjing 818, Yangnong 1, Nanjing 9108, and Ningxiangjing 9—was conducted under two typical operating environments: direct sunlight and backlighting. Figure 13 (a) Analysis of the measurement error of the cutting width of Huiliangyou Yuehe Simiao under the conditions of front light and back light; Figure 13(b) Analysis of the cutting width measurement error of Jinjing 818 under front light and back light conditions; Figure 13 (c) Analysis of the cutting width measurement error of Yangnong No. 1 under front light and back light conditions; Figure 13 (d) Analysis of the cutting width measurement error of Nanjing 9108 under front light and back light conditions; Figure 13 (e) Analysis of the cutting width measurement error of Ningxiangjing 9 under front light and back light conditions;

[0061] Figure 14 The results of an experiment conducted according to one embodiment of the present invention, using Huiliangyou Yuehe Simiao and Jinjing 818 rice as research objects, with the camera fixed at heights of 2.5 m and 2.8 m on a combine harvester, respectively. Figure 14 (a) Analysis of the measurement error of rice height and cutting width at installation heights of 2.5 m (installation height 1) and 2.8 m (installation height 2) for Huiliangyou Yuehe Simiao rice; Figure 14 (b) Analysis of the measurement error of rice height and cutting width for Jinjing 818 at installation heights of 2.5 m (installation height 1) and 2.8 m (installation height 2);

[0062] Figure 15 Taking Nanjing 9108 and Ningxiangjing 9 rice as research objects, the measurement performance under two configuration schemes, left-side offset installation and central axis installation, was compared and analyzed. Figure 15 (a) Analysis of the measurement error of rice height and cutting width for Nanjing 9108 under the installation positions of left offset (installation position 3) and central axis (installation position 4); Figure 15 (b) Analysis of the measurement error of rice height and cutting width under the installation positions of Ningxiangjing 9 on the left side offset (installation position 3) and the central axis (installation position 4).

[0063] Figure 1 The components include: 1. Intel RealSense D435i depth camera; 2. Jetson series embedded processor. Detailed Implementation

[0064] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0065] The core of this invention's forward-looking vision measurement for combine harvesters lies in multi-source information fusion and dynamic error correction. First, depth and RGB information are simultaneously acquired using a single depth camera to build a data foundation. Second, image processing and deep learning are integrated to intelligently extract crop ear and harvesting boundary features from the RGB images and map spatial information from the depth images. Finally, an innovative dynamic attitude correction based on inertial measurement is introduced. An extended Kalman filter is used to fuse gyroscope and accelerometer data from the camera's built-in IMU in real time, optimally estimating attitude angle changes caused by terrain and vibration, and compensating for these changes in real time in the height and width calculation models. This fundamentally eliminates the systematic errors caused by traditional fixed attitude assumptions.

[0066] Figure 1 The figure shown is a preferred embodiment of the method for measuring the height of the crop to be harvested and the cutting width of the combine harvester according to the present invention, which includes the following steps:

[0067] Step S1: Simultaneously acquire depth images and RGB images of the area to be harvested in front of the combine harvester using a depth camera, and align the depth images and RGB images.

[0068] Step S2: Based on the aligned image, extract the region of interest, identify the crop spike through image processing, and calculate the plant height by combining depth information;

[0069] S3. Using an instance segmentation model improved based on deep learning, identify the boundary lines between harvested and unharvested regions in the RGB image;

[0070] S4. Based on the pixel coordinates of the boundary line, the corresponding depth value, and the camera parameters, the real-time cutting width of the combine harvester is calculated through coordinate transformation.

[0071] S5. Obtain the inertial measurement unit data of the depth camera, and dynamically estimate and correct the camera's attitude angle through a sensor fusion algorithm. The attitude angle is used to calculate and correct the plant height and the cutting width.

[0072] Specifically, in step S1, the depth camera is an Intel RealSense D435i depth camera 1. The Intel RealSense D435i depth camera 1 is installed in front of the combine harvester, and a world coordinate system is established: the forward direction is the positive X-axis, the horizontal direction perpendicular to the forward direction is the positive Y-axis, and the direction perpendicular to the ground and upward is the positive Z-axis.

[0073] like Figure 2As shown, the specific method for calculating plant height in step S2 is as follows: convert the RGB image of the region of interest to the HSV color space for threshold segmentation to obtain the rice panicle region; map the pixel coordinates of the region to the depth image to obtain distance information, based on the camera installation height, camera pitch angle, and distance from the panicle to the camera.

[0074] Specifically, the synchronous acquisition of depth images and RGB images of the area to be harvested, wherein the image resolution is 1280×720 pixels and the frame rate is 30 fps, and the two images are aligned.

[0075] Specifically, the steps for measuring the height of rice plants are as follows: Figure 3 As shown, the specific steps include the following:

[0076] S2.1: Define a 480×130 pixel region of interest as the ROI on the aligned RGB image. The ROI is calculated based on the camera's field of view, typical installation height (e.g., 250cm), and average crop height range (85cm to 110cm) to ensure that the ROI covers the ear tip layer and eliminates ground and sky interference. In practical applications, the position and size of the ROI can be dynamically adjusted according to the camera calibration parameters and prior crop knowledge.

[0077] S2.2: Convert the image within the ROI from RGB color space to HSV color space, and extract the rice spike region through global threshold segmentation;

[0078] S2.3: Perform spatial filtering on the depth image to obtain smooth depth data; map the valid pixel coordinates in the RGB image that meet the threshold condition to the depth image, obtain the depth value of the corresponding pixel, and use the depth scale (0.001) to convert the depth value into the actual distance;

[0079] S2.4: Based on the camera installation height H1, camera pitch angle θ, and distance L from the top layer of the grain to the camera. i Calculate the height of rice plants using the formula:

[0080]

[0081] Where H1 is the camera installation height, Li is the distance from the ear of rice corresponding to the i-th pixel to the camera, θ is the camera pitch angle, and n is the number of effective pixels, which is the total number of pixels within the ROI that meet the HSV threshold condition. Since the camera's pitch angle is downward when looking at the rice... Since the value is negative, this invention only considers the projection of the angle between the optical axis and the horizontal line onto the vertical direction when calculating the height; therefore, it takes... The absolute value is used to avoid the sign affecting the calculation, ensuring that the obtained plant height is positive and conforms to the actual measurement.

[0082] Step S4: Measure the cutting width. The steps are as follows: Figure 4 As shown, the specific steps include the following:

[0083] S4.1: An improved YOLO11-SCC instance segmentation model is used to process RGB images to accurately identify the boundary lines between harvested and unharvested areas;

[0084] S4.2: The YOLO11-SCC instance segmentation model adopted first introduces the SwinTransformer module into the Neck part of YOLO11, replacing the original C3k2 module. The C3k2 module is the original basic convolutional module in YOLO11, usually composed of multiple convolutional layers and cross-stage local connections (CSP), used to balance computational efficiency and expressive power during feature extraction. The Swin Transformer module first divides the image into fixed-size image patches (e.g., 4×4) through Patch Partition and maps them to a high-dimensional feature space through linear embedding. Subsequently, the model extracts features step by step through multiple stages, each stage consisting of several Swin Transformer Blocks, and uses Patch Merging at the beginning of the stage to merge adjacent image patches to achieve multi-scale feature representation. Each block contains a local window self-attention module (W-MSA) or a shift window module (SW-MSA) and a multilayer perceptron (MLP), and constructs the information flow through layer normalization and residual connections. W-MSA computes self-attention within a fixed window to reduce computational complexity, while SW-MSA achieves cross-window information interaction through window translation, cyclic shifting, and masking mechanisms, thereby efficiently capturing local and global image features. The Swin Transformer module models global contextual information through a self-attention mechanism, significantly improving the model's ability to model long-distance dependencies and enhancing the perception accuracy of crop boundaries in complex field scenes. Secondly, a ContextAggregation module is added to the Neck. This module, through an attention-guided mechanism, fuses global semantic weights with fine-grained channel information to dynamically enhance boundary regions, improving the edge accuracy of instance segmentation masks and resulting in more refined and continuous segmentation of unharvested areas. Finally, the lightweight CSPPC module replaces the original C3k2 module in the Backbone section, with the model structure as follows. Figure 7As shown, CSPPC continues the residual connection mechanism of the C3 module in its structural design and introduces partial convolution (PConv) in the main path to achieve more efficient spatial feature modeling and optimize computational resource utilization. PConv applies standard convolution operations to only a portion of the input feature map channels, while the remaining channels are passed through identity mapping or lower complexity paths. This improves feature representation capabilities while effectively reducing redundant computation, achieving a lightweight model structure, and making the model more suitable for real-time operation in embedded environments.

[0085] In one specific embodiment of this invention, a total of 2416 images of rice paddies were acquired, and the camera installation angle, installation height, and combine harvester operating parameters were recorded simultaneously. To achieve accurate identification and segmentation of unharvested areas in the rice paddy images, LabelMe software was used to manually annotate the acquired RGB images. Based on the actual harvesting status of the rice in the images, polygon outlines were manually drawn to accurately delineate the boundaries of unharvested areas. Each unharvested area was labeled as an independent instance and uniformly assigned the category label "rice" to ensure consistency of categories during subsequent model training. After annotation, the software automatically generated a corresponding JSON format annotation file, which contained the polygon coordinates and category information of the target area. To meet the training requirements of the YOLO instance segmentation model, this invention uses a data conversion program written in Python to parse the polygon annotation information in the JSON file and convert it into a YOLO format TXT tag file. During the conversion process, the polygon vertex coordinates of each unharvested area were normalized according to the image size and stored in the form of category number and polygon point sequence. Each image corresponds to a TXT tag file, realizing a one-to-one correspondence between images and instance annotation information. This data format can be directly used for instance segmentation training of YOLO models.

[0086] To improve the model's generalization ability, data augmentation was performed simultaneously on the original images and their corresponding labels, including geometric transformations such as rotation, translation, scaling, and horizontal flipping. Through data augmentation, the dataset size was expanded from 2416 images to 12080 images. This invention divides the dataset into training, validation, and test sets in a 7:2:1 ratio. The dataset includes different varieties of rice at various growth stages, as well as varying field environments such as terrain undulations and light variations. Furthermore, operational parameters such as harvester speed, cutting width, and header height also contribute to the differences in data under different operating conditions, further increasing the diversity of the dataset.

[0087] YOLO11-SCC model training configuration. The YOLO11-SCC model is trained using 640 × 640 resolution input images for 200 epochs to achieve a balance between computational efficiency and detection accuracy. To accelerate data processing, 32 data loading threads are used during training. In the segmentation task during training, the YOLO11-SCC model employs a multi-task joint loss function to simultaneously optimize target bounding box localization accuracy and segmentation mask quality.

[0088] The YOLO11-SCC model uses the following multi-task joint loss function during training:

[0089]

[0090] Where, λ box =7.5, λ cls =0.5, λ df1 =1.5, λ seg =1.0.

[0091] The hyperparameters of the YOLO11-SCC model were tuned using a validation set, primarily including the learning rate, batch size, and optimizer-related parameters. The final hyperparameter configuration was determined based on performance on the validation set. Specifically, stochastic gradient descent (SGD) was used as the optimizer, with an initial learning rate of 0.001, a batch size of 32, a momentum coefficient of 0.937, and a weight decay coefficient of 0.0005, to improve the stability of the training process and the model's generalization ability.

[0092] Model performance evaluation based on independent test sets. To comprehensively evaluate the performance of the YOLO11-SCC model in the task of detecting unharvested rice areas, precision (P), recall (R), mean average precision (mAP), average intersection over union (IoU), total floating-point operations (FLOPs), number of parameters (Params), and inference speed (Frames Per Second, FPS) are used as evaluation metrics for model precision, achieving a comprehensive measurement of model accuracy and real-time performance.

[0093]

[0094]

[0095] in, The model correctly detected the "unharvested area" of rice. The model incorrectly predicted "harvested areas" as "unharvested areas," which is a false detection. The model failed to identify the actual "uncut area," which is a missed detection. It is a function of precision and recall; This refers to the total number of target categories involved in the detection task. This invention is a single-category target detection task. .

[0096] The training environment for the model was configured as follows: Windows 11 operating system, Python 3.12 programming language, PyTorch 2.5 deep learning framework, VS Code 1.100.2 integrated development environment, Anaconda 4.12.0 environment manager, Intel® Core™ i9-14900K @ 6.00 GHz CPU, NVIDIA GeForce RTX 4080 16 GB GPU, and 64 GB of RAM.

[0097] To systematically evaluate the effectiveness of the introduced modules in rice image instance segmentation, experiments were conducted to test the impact of introducing the Swin Transformer, Context Aggregation, and CSPPC modules individually on the model performance, as well as whether there was a synergistic enhancement effect after their combined introduction. The results of the ablation experiments are shown in Table 1.

[0098] Table 1 Comparison of ablation test results

[0099]

[0100] As shown in Table 1, after introducing the Swin Transformer module (Model 1) alone, the model's mAP@0.5 and mAP@0.5:0.95 improved by 0.72% and 2.34%, respectively, and the IoU improved by 0.92%. This is because the Swin Transformer introduces a window-based self-attention mechanism, which can more effectively model long-range dependencies in images and enhance global semantic understanding capabilities. Although the model structure is complex, due to the hierarchical and local computation strategy adopted in its module design, the overall number of parameters is reduced to 2.74M, while the computational complexity is also reduced to 9.9 GFLOPs, and the FPS is increased to 282.83. The results show that this module effectively improves the modeling ability of long-range dependencies while maintaining model compactness.

[0101] After introducing the Context Aggregation module (Model 2) separately, the model's mAP@0.5:0.95 improved by 2.45%, and IoU improved by 0.62%. The number of parameters and computational complexity remained basically the same as the original model, indicating that this module can enhance the feature semantic fusion capability without increasing resource consumption, making the model more stable in boundary awareness and detail recognition.

[0102] After introducing the CSPPC module (Model 3) separately, the model's mAP@0.5:0.95 and IoU showed slight improvements of 0.21% and 0.10%, respectively. Although the improvement in accuracy was limited, the number of parameters decreased by 0.36%, GFLOPs decreased by 3.85%, and FPS increased to 255.98. These results indicate that the CSPPC module can effectively compress the network structure and reduce redundant computation while maintaining relatively stable model accuracy, thereby improving the model's running efficiency and deployment adaptability, making it suitable for subsequent real-time inference tasks on embedded platforms.

[0103] Simultaneously, by introducing the Swin Transformer and Context Aggregation (Model 4), the model's mAP@0.5:0.95 increased to 0.964, IoU increased to 0.984, GFLOPs decreased to 9.9, and FPS increased by 24.80%, further verifying the complementarity of the global attention mechanism and the local context-aware structure at the feature extraction level, and improving the continuity and accuracy of the target segmentation boundary.

[0104] The final YOLO11-SCC model integrates the three modules mentioned above, improving its mAP@0.5:0.95 to 0.966, a 2.77% improvement over the initial model, and its IoU to 0.985, a 1.23% improvement over the initial model. While maintaining the minimum number of parameters (2.73M, a 3.90% decrease), the lowest computational cost (9.6 GFLOPs, a 7.70% decrease), and the fastest inference frame rate (294.11, a 25.74% improvement), it achieves dual optimization of performance and efficiency. Comprehensive analysis shows that the improved YOLO11-SCC model achieves higher segmentation accuracy and structural continuity in boundary regions while maintaining a lower number of parameters.

[0105] For instance segmentation tasks involving harvested and unharvested rice areas, a comparison of the loss function performance of the YOLO11-SCC model before and after the improvement is shown below. Figure 8 As shown, under the same number of training rounds, the improved model exhibits faster convergence speed and lower convergence characteristics in terms of total loss value, and the loss value fluctuates less during training, demonstrating better stability and optimization efficiency.

[0106] An improved YOLO11-SCC model was used to segment rice images captured by a camera, identifying the unharvested rice region. The right edge contour was extracted to determine the boundary line between the harvested and unharvested areas. Under actual field conditions, the boundary line between the harvested and unharvested rice regions exhibits various morphologies, including undulating patterns. Figure 9 (a) Notch type ( Figure 9 (b) ), Bending type ( Figure 9 (c) and linear ( Figure 9 (d)).

[0107] In a depth camera rigidly mounted at the front end of a combine harvester, a fixed rigid transformation relationship exists between the camera coordinate system and the vehicle coordinate system. This transformation is obtained through extrinsic parameter calibration. The extrinsic parameters are determined by the rotation matrix R ∈ R 3×3 Translation vector T∈R 1×3 The structure describes the camera's attitude and position relative to the vehicle coordinate system. The rotation matrix R is used to characterize the camera's attitude relative to the vehicle coordinate system, i.e., its position around the vehicle coordinate system's X-axis (roll angle). Y-axis (pitch angle) ) and Z-axis (yaw angle) The rotation of ) is represented by the three basic rotation matrices:

[0108]

[0109]

[0110]

[0111] The rotation matrix R can be expressed as

[0112]

[0113] Translation vector The position of the camera's optical center in the vehicle coordinate system is defined as:

[0114]

[0115] in, For horizontal offset, Forward offset, Corresponding camera installation height .

[0116] Based on the extrinsic parameters obtained from the calibration results, the three-dimensional boundary points in the camera coordinate system can be... Transform to the vehicle coordinate system (world coordinate system) defined by the combine harvester. A schematic diagram of the coordinate transformation is shown below. Figure 10 As shown, the specific calculations are as follows:

[0117]

[0118] S4.5: such as Figure 11 As shown, the absolute distance w2 of the boundary point relative to the camera is calculated, and combined with the fixed distance w1 of the left divider relative to the camera, the cutting width w is obtained.

[0119]

[0120] Step S5: Dynamically correct the camera attitude angle. The process steps are as follows: Figure 5 As shown, the specific steps include the following:

[0121] S5.1: The Intel RealSense D435i depth camera used has a built-in inertial measurement unit (IMU) that can simultaneously output three-axis angular velocities. and triaxial acceleration This provides basic sensing data for dynamic estimation of attitude angles.

[0122] First, the angular velocity around the Y-axis is obtained based on the D435i's built-in IMU. The original pitch angle estimate of the camera is obtained by integration, and the calculation formula is as follows:

[0123]

[0124] This method has a fast response speed and is suitable for capturing high-frequency attitude changes, but it is susceptible to the cumulative errors of gyroscope drift and noise during long-term operation, resulting in unstable estimates.

[0125] To enhance estimation accuracy, accelerometer data is further introduced to calculate the direction of gravity. By utilizing the projection relationship of the gravitational component in the triaxial acceleration, the pitch angle is estimated under static or low-dynamic-interference scenarios. The calculation formula is as follows:

[0126]

[0127] The method of this invention has good long-term stability, but it is sensitive to instantaneous dynamic acceleration. To balance the dynamic response of the gyroscope integration method and the long-term stability of the accelerometer method, this invention introduces an extended Kalman filter (EKF) to dynamically fuse the two, construct a state-space model, and jointly update the camera pitch angle and its rate of change.

[0128] The system state is defined as a two-dimensional state vector consisting of the pitch angle and the gyroscope's zero bias error:

[0129]

[0130] in, for The system state at any given moment; For the first The pitch angle at any moment, This represents the zero-bias error of the gyroscope around the Y-axis.

[0131] Considering the zero-bias random walk characteristic of the gyroscope, the state transition function is established as follows:

[0132]

[0133] in,

[0134] The Jacobian matrix for the state transition is:

[0135]

[0136] in, for The system state at any given moment;

[0137] The process noise represents the system state at time t. Random disturbances that occur , Let be the process noise covariance matrix, take ;

[0138] The system observation is the pitch angle estimated by the accelerometer, and the observation model can be expressed as:

[0139]

[0140] in, For at any time The collected elevation angle observations; The observation function is used to map the system state to the observed values; To observe the noise, , To observe the noise covariance, take .

[0141] The Jacobian matrix of the observation model is:

[0142]

[0143] Within each acquisition cycle, the EKF is optimized in real time through prediction and updates. During the prediction phase, as shown in the formula, the pitch angle at the next time step is predicted using a state transition model in the absence of new observation data.

[0144]

[0145]

[0146] in: For the present The predicted state vector at time t represents the estimate without correction using observed data; This is the predicted state vector from the previous measurement; For the present The state prediction covariance matrix at time t represents the uncertainty of the predicted state. This is the covariance matrix of the previous measurement;

[0147] During the update phase, EKF incorporates sensor measurement data to correct the predicted state. This is done upon receiving new observation data. Then, update the estimated values ​​and covariance matrix. Based on the Kalman gain... Calculate the updated state estimate and covariance matrix.

[0148]

[0149]

[0150]

[0151] in, It is the Kalman filter gain, which controls the updating of system state variables; It is a unit diagonal matrix;

[0152] After EKF recursive processing, the optimal pitch angle estimate is obtained as follows:

[0153]

[0154] After EKF optimization, the output pitch angle By combining the high-frequency dynamic information from the gyroscope and the low-frequency stability information from the accelerometer, good dynamic tracking performance is maintained while suppressing sensor noise.

[0155] S5.4: Construct an extended Kalman filter (EKF), fuse gyroscope and accelerometer data, and process the pitch angle θ. t and gyroscope bias b ω,t Perform optimal estimation to obtain the corrected pitch angle. The roll angle φ is updated similarly.

[0156] Along the combine harvester's operating path, positioning markers are placed at fixed intervals of 5 meters as spatial reference points to record the relative positional changes of the rice's boundaries before and after harvesting. Before harvesting, the horizontal distance between each marker and the outermost edge of the adjacent rice growing area is manually measured and recorded as follows: After the harvester completes its work on this section of the work path, the horizontal distance from the marker to the edge of the unharvested rice is measured again at the same location and recorded as follows: Based on the counter-clockwise path advancing in field operations, the width of the manual cutting can be defined as the horizontal difference between the boundary lines before and after harvesting in the reference coordinate system, i.e.

[0157]

[0158] in, The cutting width is measured manually. The horizontal distance from the rice paddy boundary to the marker before harvest; This refers to the horizontal distance from the boundary of the rice paddies to the marker after harvest.

[0159] At the same time, at each marker, 20 rice plants were randomly measured with a measuring tape from the ground to the highest point of the rice ear to be taken as the actual height of the crop.

[0160] By calculating the root mean square error (RMSE), mean absolute percentage error (MAPE), mean absolute error (MAE), and maximum error (... The accuracy and stability of the model are evaluated using indicators such as [list of indicators]. The calculation formula is as follows:

[0161]

[0162]

[0163]

[0164] in, These are manually measured values. These are the model's predicted values; This represents the number of samples.

[0165] The process of comparing model measurements with actual measurements is as follows: Figure 6 As shown.

[0166] Preferably, the image processing algorithm is deployed on a Jetson (Xavier NX) embedded processor, and a multi-threaded architecture is used to ensure system real-time performance and interface responsiveness.

[0167] Preferably, to balance processing efficiency and accuracy, only one frame out of every five frames is processed from the 30 frames of images captured by the camera per second.

[0168] This invention integrates depth and color visual information, combined with dynamic camera attitude correction and deep learning-based boundary recognition, to achieve synchronous, reliable, and real-time measurement of rice plant height and cutting width, providing a precise forward-looking perception basis for the prediction and intelligent control of feed volume in combine harvesters.

[0169] Based on the data analysis in Table 2, the other four rice varieties all showed a similar accuracy improvement trend as Yangnong No. 1 under dynamic pitch angle correction. For Huiliangyou Yuehe Simiao, the accuracy of plant height measurement improved under dynamic pitch angle correction, with RMSE decreasing from 2.74 cm to 2.32 cm, MAPE decreasing from 2.15% to 1.70%, and the maximum error decreasing by 0.77 cm. For Jinjing 818, RMSE and MAPE decreased to 1.62 cm and 1.21% respectively under dynamic conditions, with the maximum error decreasing by 1.08 cm. For Nanjing 9108, RMSE, MAPE, and maximum error decreased to 1.16 cm, 1.01%, and 2.97 cm respectively. For Ningxiangjing 9, RMSE decreased to 1.08 cm, MAPE decreased to 1.02%, and the maximum error decreased from 4.74 cm to 2.72 cm under dynamic measurement conditions. Based on the combined results of five varieties, the dynamic pitch angle correction method showed superior performance compared to static conditions in the application of multiple rice varieties. It can effectively suppress measurement errors caused by changes in machine posture and improve overall monitoring accuracy.

[0170] Table 2 Comparison of Height Measurement Error Assessment Results for Different Rice Varieties

[0171]

[0172] As shown in Table 3, the other four rice varieties also exhibited superior performance in the cut width measurement using dynamic pitch angle correction compared to the fixed pitch angle method. For Huiliangyou Yuehe Simiao, under dynamic measurement conditions, the MAE of the cut width measurement data decreased from 4.52 cm to 3.30 cm, and the maximum error decreased from 8.52 cm to 6.81 cm, representing improvements of approximately 27.0% and 20.1%, respectively. For Jinjing 818, under dynamic correction conditions, the MAE decreased from 4.28 cm to 3.13 cm, and the maximum error decreased from 6.24 cm to 4.85 cm, representing improvements of 26.9% and 22.3%, respectively. Nanjing 9108 also showed a significant improvement in measurement accuracy, with the MAE decreasing from 3.74 cm to 2.76 cm and the maximum error decreasing from 6.74 cm to 4.60 cm, representing improvements of 26.2% and 31.8%, respectively. Under dynamic correction, the MAE of Ningxiangjing 9 decreased from 3.56 cm to 2.30 cm, and the maximum error decreased from 5.87 cm to 4.70 cm, representing improvements of 35.4% and 19.9% ​​respectively, indicating its effectiveness even among dwarf varieties. In summary, the dynamic pitch angle correction method significantly reduced MAE and maximum error in cutting width measurements across different rice varieties, with an overall improvement ranging from approximately 20% to 35%. This further demonstrates that the method effectively suppresses the cumulative error caused by attitude fluctuations in cutting width measurement, thereby enhancing the accuracy and stability of cutting width monitoring. The combined measurement results for various varieties show that real-time pitch angle correction technology effectively reduces cutting width measurement errors across different rice varieties, verifying the universality and effectiveness of this technology in precision agriculture harvesting operations.

[0173] Table 3 Comparison of measurement error assessment results for the same rice variety

[0174]

[0175] These errors primarily stem from the fact that pitch angle fluctuations directly alter the camera's imaging geometry, thus affecting the effective field of view and the distance measurement path between the camera and the rice canopy. This variation disrupts the correspondence between depth values ​​and true distances, introducing systematic bias into depth map-based estimations. As the amplitude of pitch angle variation increases, projection distortion and edge effects are further aggravated, leading to a decrease in the continuity and accuracy of the depth map. In summary, these results highlight the critical impact of pitch angle fluctuations on measurement accuracy and demonstrate that dynamic pitch correction can not only effectively improve the accuracy of rice plant height estimation but also significantly enhance the accuracy of cutting width detection, especially under complex conditions such as uneven ground.

[0176] In rice height measurement, plant structure characteristics significantly affect the measurement accuracy of depth cameras. Huiliangyou Yuehe Simiao, an indica two-line hybrid rice, has an average plant height of 105.03 cm. Its leaves are narrow and long, and it has a large number of tillers. Multiple tiller stems and leaves may overlap around a single plant, leading to measurement errors. Furthermore, the reflective properties of its smooth leaf surface may affect the quality of infrared texture matching, reducing the stability of depth calculation. Jinjing 818, a typical japonica rice, has an average plant height of 91.22 cm. Its leaves are upright, it has strong tillering ability, and its dense canopy with overlapping leaves increases shading and reduces measurement accuracy. Yangnong No. 1 has an average plant height of 85.03 cm, with broad, thick leaves, a compact plant type, good stem uprightness, and a moderate number of tillers. Its clear microtexture and suitable canopy density are beneficial for depth information extraction and measurement stability. Nanjing 9108 has a compact plant type with an average plant height of 88.04 cm. Its leaves are short and upright, and the canopy is uniform and dense. Although there is some leaf overlap, the upright leaves result in relatively light shading. Ningxiangjing 9 has an average plant height of 86.14 cm, relatively thick stems, good lodging resistance, a compact and uniform plant type, a moderate number of tillers, and clear leaf texture, providing a stable benchmark for depth measurement and facilitating accurate acquisition of rice height information.

[0177] The results of the cutting width detection showed that Huiliangyou Yuehe Simiao had the most significant error fluctuations, with a maximum positive error of +9cm and a minimum error of -4cm. It exhibited a significant error peak at the initial boundary position, indicating a large deviation in image recognition or boundary extraction, possibly related to the complexity of the plant structure. Jinjing 818 showed local error peaks (up to approximately +6cm) at the 6th and 10th boundary points, which may be affected by lodging or shading effects. Yangnong 1 exhibited a clear V-shaped error distribution: a large error at the beginning (approximately +5cm), a relatively stable middle section, and a further increase at the end, with a narrow overall error range (-2cm to +5cm), indicating that the model has strong adaptability and control over this variety. Nanjing 9108 showed a relatively stable error trend, with only slight fluctuations (up to approximately +5cm) at the 8th–12th boundary points, demonstrating good generalization performance. The error distribution of Ningxiangjing 9 is relatively dispersed, ranging from -3cm to +5cm. Although the error fluctuation frequency is high, the amplitude is moderate, indicating that the model has good robustness in image processing.

[0178] Comprehensive analysis shows that the morphological characteristics of different rice varieties have a differentiated impact on the performance of depth camera measurements by influencing plant geometry, optical properties, and texture distribution. Varieties with moderate plant height, compact plant type, and uniform canopy distribution (such as Yangnong No. 1, Nanjing 9108, and Ningxiangjing No. 9) exhibit superior performance in height and cutting width measurement stability. In contrast, tall, sparsely planted varieties (such as Huiliangyou Yuehe Simiao) have larger gaps between stems and leaves and are more prone to movement, resulting in more background information interference in the depth map. This leads to insufficient continuity of the point cloud at the top and boundary areas of the canopy, increasing the random error in height measurement and the maximum error in cutting width identification. Therefore, under the existing system configuration and algorithm model, short and compact varieties such as Yangnong No. 1, Nanjing 9108, and Ningxiangjing No. 9, due to their clearer geometric features and lower occlusion rate, achieve lower RMSE and MAPE values ​​in crop height and cutting width measurements, demonstrating higher accuracy and stability.

[0179] In measuring rice height and cutting width using an Intel RealSense D435i depth camera, different lighting and weather conditions significantly affected measurement accuracy and data quality. To evaluate the impact of different lighting conditions on the measurement of rice plant height using a depth camera, this example systematically compared and analyzed the measurement results of Huiliangyou Yuehe Simiao rice under conditions of direct sunlight, backlighting, overcast skies, and cloudy skies. The results are as follows: Figure 12 As shown.

[0180] In terms of rice height measurement performance, the best results were achieved under cloudy conditions, with an RMSE of 1.94 cm and a MAPE of 1.51%. This is mainly due to the natural diffusion effect of clouds providing a uniform and soft lighting environment. Under these conditions, the overall brightness distribution of the image is uniform, avoiding overexposure and underexposure, and the depth features of the rice canopy and the ground are clearly contrasted. Measurement performance under cloudy conditions is slightly lower than under cloudy conditions. The intermittent cloud cover causes dynamic changes in light intensity. Although near-overcast measurement results can be obtained during periods of cloud cover, the time-varying nature of light increases system instability, potentially leading to fluctuations in accuracy during continuous measurements. Under direct sunlight, the depth camera can obtain sufficient ambient light, but direct sunlight may cause local overexposure and the appearance of high-contrast areas in the image, affecting the feature extraction accuracy of the stereo matching algorithm in highlight areas. A significant brightness gradient forms between the high-reflectivity areas and shadow areas of the rice canopy, increasing the RMSE to 2.25 cm and the MAPE to 1.54%. The measurement error is most significant under backlighting conditions, with an RMSE of 2.70 cm and a MAPE of 2.07%. The main reason is that the light source is located in the direction of the camera's line of sight, resulting in uneven brightness distribution in the overall image and a significant reduction in the contrast between the rice paddies and the soil boundary. Furthermore, backlighting can easily cause lens flare and halo effects, reducing image quality and decreasing the reliability of depth calculations.

[0181] In terms of sectional width measurement, the boundary detection capability is also better under cloudy conditions, such as... Figure 12 As shown in (c), the maximum error is 4.62 cm, and the average absolute error is 2.17 cm. This is because the contrast between the rice canopy and the soil boundary is clear and the depth characteristics are stable under uniform light conditions. Figure 12 As shown in (d), the accuracy of the sectional width measurement decreased under cloudy conditions, with the maximum error increasing to 5.33 cm and the average absolute error being 2.53 cm, reflecting the interference effect of intermittent illumination changes on boundary continuity identification. Figure 12 As shown in (a), under direct sunlight, the strong directional light creates a complex light and shadow distribution in the boundary area of ​​the paddy field, increasing the complexity of boundary identification and thus introducing a certain degree of systematic error into the measurement of the cutting width, resulting in a maximum error of 5.84 cm and an average absolute error of 3.29 cm. Backlighting has the most severe negative impact on the measurement of the cutting width, such as... Figure 12 As shown in (b), the maximum error reached 6.63 cm, and the average absolute error was 3.96 cm, representing increases of 43.5% and 82.5% respectively compared to cloudy conditions. This is mainly due to the sharp decrease in contrast in the boundary region under backlighting conditions, blurred depth gradient changes, and difficulty in extracting boundary features, which reduces the accuracy of the segment boundary positioning. Comprehensive analysis indicates that a uniform diffuse lighting environment is beneficial for improving measurement accuracy and stability, while strong directional lighting and uneven lighting reduce the accuracy of depth calculation.

[0182] To evaluate the impact of ambient lighting conditions on the accuracy of cutting width measurement, this invention conducted a systematic comparative experiment on the cutting width measurement of five rice varieties—Huiliangyou Yuehe Simiao, Jinjing 818, Yangnong 1, Nanjing 9108, and Ningxiangjing 9—under two typical operating environments: direct sunlight and backlighting. The results are as follows: Figure 13 As shown.

[0183] The results show that lighting conditions have a significant impact on the framing measurement performance of depth cameras, with measurement accuracy generally lower under backlighting conditions than under frontlighting conditions. For example... Figure 13 As shown in (a) to (e), the average absolute error range for the five varieties under direct lighting conditions was 2.77–3.29 cm, while it increased to 3.24–3.96 cm under backlighting conditions. The maximum error ranged from 4.04 to 5.96 cm under direct lighting conditions, but rose to 5.11–6.63 cm under backlighting conditions. The combined measurement data for the five varieties show that both the average absolute error and the maximum error were lower under direct lighting conditions than under backlighting conditions, indicating that direct lighting is more conducive to accurate measurement of the cutting width.

[0184] To systematically evaluate the impact of camera installation position on the accuracy of rice plant height and cutting width measurements, this invention uses Huiliangyou Yuehe Simiao and Jinjing 818 rice varieties as research objects. The camera was fixed on a combine harvester at heights of 2.5m (installation height 1) and 2.8m (installation height 2), respectively, ensuring that both installation arrangements placed the rice plants within the effective measurement range of the equipment and allowed for a complete representation of their external contours within the field of view, thus meeting the requirements for comprehensive depth information acquisition. The results are as follows: Figure 14 As shown.

[0185] The results showed that the measured values ​​of rice height and cutting width under both installation height conditions were highly correlated with the manually measured results. Figure 14 As shown in (a), the RMSE of the plant height of Huiliangyou Yuehe Simiao seedlings at installation heights 1 and 2 were 2.20 cm and 2.27 cm, respectively; Figure 14 As shown in (b), the RMSE of plant height measurement for Jinjing 818 at installation height 1 and height 2 were 1.46 cm and 1.54 cm, respectively; the corresponding MAPE was less than 1.8%. In contrast, the RMSE under installation height 1 was slightly lower, showing a slight accuracy advantage. This is mainly attributed to the higher signal-to-noise ratio of the depth sensor at closer measurement distances and less affected by lens distortion and depth drift at long distances.

[0186] Regarding the accuracy of the cutting width measurement, the performance of both installation heights is within acceptable limits. For example... Figure 14 As shown in (a), the maximum error of the cutting width of Huiliangyou Yuehe Simiao at installation height 1 is approximately 6.44 cm, and the average absolute error is approximately 4.01 cm; the maximum error of the cutting width at installation height 2 is approximately 7.61 cm, and the average absolute error is approximately 4.20 cm. Figure 14 As shown in (b), the maximum error of the cutting width of Jinjing 818 at installation height 1 is about 6.19 cm, and the average absolute error is about 3.41 cm; the maximum error of the cutting width at installation height 2 is about 6.64 cm, and the average absolute error is about 3.65 cm.

[0187] It can be seen that installation height 1 is slightly better than installation height 2 in both maximum and average error. At the lower installation height, the camera achieves higher spatial resolution, and the actual lateral distance corresponding to a unit angle change is smaller, resulting in more accurate boundary positioning between the harvested and unharvested rice areas. Simultaneously, the point cloud density generated by close-range measurement is improved, providing richer geometric information for the boundary recognition algorithm and effectively reducing boundary blurring effects caused by data sparsity. However, the difference in average error between the two installation heights is small, and both meet the accuracy requirements for cut width measurement in combine harvester operations. Therefore, in practical applications, this difference has a limited impact on operation quality.

[0188] Compared to other major error sources such as pitch angle fluctuations, plant structure complexity, and changes in ambient light, the impact of installation height differences on overall measurement accuracy is relatively small. Therefore, in practical applications, provided that the rice target is within the camera's effective measurement range and a complete rice field of view can be obtained, a suitable installation height can be selected based on the mechanical structure layout requirements and operational convenience requirements, without the need for special optimization adjustments for minor accuracy differences.

[0189] This invention optimizes the lateral mounting position of a depth camera on a combine harvester. Using Nanjing 9108 and Ningxiangjing 9 rice varieties as examples, the measurement performance of two configurations—left-side offset mounting (mounting position 3) and center-axis mounting (mounting position 4)—was compared and analyzed. Both mounting positions ensured that the collected rice data remained within the camera's effective measurement range, not exceeding the reliable depth measurement range of the D435i under outdoor conditions. The results are as follows: Figure 15 As shown.

[0190] Experimental results show that both installation locations can effectively acquire rice height information, and the measurement results show a high correlation with manually measured values, verifying the accuracy and stability of the measurement method. Figure 15 As shown, the central axis mounting position exhibits superior accuracy in rice height measurement, with an RMSE reduction of approximately 5% compared to the left-off position. This is primarily attributed to the fact that the central axis mounting provides a more symmetrical field of view distribution, reducing left-right parallax inhomogeneity and spatial gradient variations in pixel resolution, thereby mitigating systematic errors in stereo vision introduced by perspective distortion and non-perpendicular viewing angles.

[0191] Regarding the accuracy of the cutting width measurement, such as Figure 15 As shown in (a), the maximum error of Nanjing 9108 at the left-side offset installation position 3 is 5.26 cm, and the average absolute error is 3.24 cm; the maximum error at the central axis installation position 4 is 4.19 cm, and the average absolute error is 2.86 cm. Figure 15 As shown in (b), the maximum error of Ningxiangjing 9 at installation position 3 was 5.64 cm, with an average absolute error of 3.08 cm; the maximum error at installation position 4 was 3.88 cm, with an average absolute error of 2.39 cm. This demonstrates that the centerline installation exhibits superior accuracy and stability in cutting width measurement, with better accuracy and consistency in boundary identification than the left-off offset installation. The left-off offset installation resulted in decreased boundary identification accuracy in the far right region due to the asymmetrical field of view distribution. Considering the comprehensive analysis of measurement accuracy, error distribution stability, and the consistency of the left and right coverage of the work path, the centerline installation position demonstrates superior performance in rice height and cutting width measurement.

[0192] Compared with existing literature, the advantages of this invention are reflected in the following aspects:

[0193] In crop height measurement, the average error obtained by this invention is 17.8 mm, which is significantly lower than the 28.3 mm reported by Guo et al. (Guo,XN, Zhou,HR, Zhang,GL, Ke,YB, Su, J., Zhao, ZM, 2018. Cropheight measurement system based on laser vision. Trans. CSAM. 49(2), 22-27. https: / / doi.org / 10.6041 / j.issn.10001298.2018.02.003) and the error reported by Blanquart et al. (Blanquart, JE, Sirignano, E., Lenaerts, B., Saeys, W., 2020. Online cropheight and density estimation in grain fields using LiDAR. Biosyst. Eng. 198,1-14. https: / / doi.org / The 82 mm reported in 10.1016 / j.biosystemseng.2020.06.014., and the 220 mm reported by Zhang and Grift (Zhang, L., Grift, TE, 2012. A LIDAR-based crop height measurement system for miscanthus giganteus. Comput. Electron. Agric. 85, 70-76. http: / / dx.doi.org / 10.1016 / j.compag.2012. 04.001) based on a LiDAR crop height measurement system. Kim et al. (Kim, WS, Lee, DH, Kim, YJ, Kim, T., Lee, WS, Choi, CH, 2021. Stereo-vision-based crop height estimation for agricultural robots. Comput. Electron. Agric. 181,105937. https: / / doi.org / 10.1016 / j.compag.2020.105937.) achieved the highest reported accuracy (15 mm), but their computation time was 50 ms, which is significantly higher than the method of this invention.The above results demonstrate that the method proposed in this invention achieves a better balance between measurement accuracy and calculation speed. In terms of slit width measurement, existing studies have reported errors mostly concentrated in the range of 32.6–120 mm. For example, Zhao et al. (Zhao, T., Noguchi, N., Yang, L., Ishii, K., Chen, J., 2016. Development of uncut crop edge detection system based on laser rangefinder for combine harvesters. Int. J. Agric. Biol. Eng. 9(2), 21-28. https: / / doi.org / 10.3965 / j.ijabe.20160902.195) proposed a method based on a laser rangefinder with an error of 25 mm; Wei et al. (Wei, LG, Zhang, XC, Wang, FZ, Che, Y., Sun, XW, Wang, ZW, 2017. Design and experiment of harvest boundary online recognition system for rice and wheat combine harvester based on laser detection. Trans.CSAE, 33(1): 30-35. https: / / doi.org / ) The laser boundary detection system designed in (10.11975 / i.issn.1002-6819.2017.z1.005.) has an error of 120 mm. Zhang et al. (Zhang, Z., Zhang, X., Cao, R., Zhang, M., Li, H., Yin, Y., Wu, S., 2022. Cut-edge detection method for wheat harvesting based on stereo vision. Comput. Electron. Agric. 197, 106910. https: / / doi.org / 10.1016 / j.compag.2022.106910) used the Otsu method combined with DBSCAN (density-based spatial clustering algorithm) to extract harvest boundary points, but the error was still 84.7 mm. In contrast, the cutting width measurement error of the method in this invention is only 36 mm, demonstrating a significant performance advantage.Although Luo et al. (Luo, YS, Wei, LL, Xu, LZ, Zhang, Q., Liu, JY, Cai, QB, Zhang WB, 2022. Stereo-vision-based multi-crop harvesting edge detection for precise automatic steering of combine harvester. Biosyst. Eng. 215, 115-128. https: / / doi.org / 10.1016 / j.biosystemseng.2021.12.016.) achieved the lowest error (32.6 mm), its single-frame processing time was as high as 300 ms, which limited its practical application in real-time operations.

[0194] In terms of computational efficiency, existing methods typically have a single-frame processing time between 65 and 60,000 ms. For example, Sun et al. (Sun, JW, Zhou, J., Wang, YD, He, YQ, Jia, HB, 2024. A cuttingwidth measurement method for the unmanned rice harvester based on RGB-Dimages. Measurement, 224, 113777. https: / / doi.org / 10.1016 / j.measurement.2023.113777) achieved 103.16 ms, and Zhang et al. (Zhang, Z., Zhang, X., Cao, R., Zhang, M., Li, H., Yin, Y., Wu, S., 2022. Cut-edge detection method for wheatharvesting based on stereo vision. Comput. Electron. Agric. 197, 106910. https: / / doi.org / ) The fastest processing time (10.1016 / j.compag.2022.106910) is 756.7ms; while the method of this invention only requires 28ms, making it the fastest solution in the comparative study.In terms of cost, the depth camera used in this invention costs approximately $200, while the stereo camera (ZED2, Stereo Labs Inc., San Francisco, CA, USA) used by Luo et al. (Luo, YS, Wei, LL, Xu, LZ, Zhang, Q., Liu, JY, Cai, QB, Zhang WB, 2022. Stereo-vision-based multi-crop harvesting edge detection for precise automatic steering of combine harvester. Biosyst. Eng. 215, 115-128. https: / / doi.org / 10.1016 / j.biosystemseng.2021.12.016.) costs approximately $750; in comparison, the stereo camera (Blanquart, JE, Sirignano, E., Lenaerts, B., Saeys, W., 2020. Online cropheight and density estimation in grain fields using LiDAR. Biosyst. Eng.) costs approximately $750. The LiDAR sensor (LMS 111, Sick, Waldkirch, Germany) used in 198,1-14 (https: / / doi.org / 10.1016 / j.biosystemseng.2020.06.014.) costs approximately $3,500. Therefore, the present invention offers a significant advantage in overall cost, which is particularly important given farmers' high sensitivity to the price of agricultural machinery.

[0195] Table 4 Comparison of Measurement Results

[0196]

[0197] In summary, the method of this invention has achieved clear and quantifiable improvements in terms of accuracy, real-time performance, and cost compared to existing research, fully demonstrating the engineering practical value and innovation of this work.

[0198] A system for measuring the height of crop plants to be harvested and the cutting width of a combine harvester, used to implement the method for measuring the height of crop plants to be harvested and the cutting width of a combine harvester, includes a depth camera, an inertial measurement unit, and an image processing module;

[0199] The depth camera is used to simultaneously acquire depth images and RGB images of the area to be harvested in front of the combine harvester;

[0200] The inertial measurement unit is integrated into the depth camera or set up separately, and is used to output triaxial angular velocity and triaxial acceleration;

[0201] The image processing module includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method for measuring the height of the crop to be harvested and the cutting width of the combine harvester according to any one of claims 1 to 7, including:

[0202] The depth image acquired by the depth camera is aligned with the RGB image. Based on the aligned image, the region of interest is extracted, and crop heads are identified through image processing. The plant height is calculated by combining the depth information. An instance segmentation model based on deep learning is used to identify the boundary line between the harvested and unharvested areas in the RGB image. Based on the pixel coordinates of the boundary line, the corresponding depth value, and the camera parameters, the real-time cutting width of the combine harvester is calculated through coordinate transformation. The inertial measurement unit data of the depth camera is acquired, and the camera attitude angle is dynamically estimated and corrected through a sensor fusion algorithm. The attitude angle is used to correct the calculation of the plant height and the cutting width.

[0203] Preferably, the depth camera is an Intel RealSense D435i depth camera 1, and the processor is a Jetson series embedded processor 2.

[0204] A harvester includes a system for measuring the height of the crop to be harvested and the cutting width of the combine harvester.

[0205] This invention achieves simultaneous acquisition of depth and color information using a single camera. Combined with an improved YOLO-SCC model, it enables simultaneous and accurate extraction of plant height and cutting width, resulting in low hardware cost and high system integration. The dynamic attitude correction algorithm effectively suppresses measurement noise caused by machine vibration and terrain undulations, significantly improving measurement robustness and reliability in unstructured field environments. The algorithm design fully considers the computing power of embedded platforms. By setting regions of interest, optimizing the sampling frame rate strategy (e.g., processing 1 frame every 5 frames), and using a lightweight model, the real-time performance of the entire system under limited resources is ensured. This invention can be used for harvesting various crops such as wheat, rice, millet, corn, and rapeseed.

[0206] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0207] The detailed descriptions listed above are merely specific illustrations of feasible embodiments of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for measuring the height of crop plants to be harvested and the cutting width of a combine harvester, characterized in that, Includes the following steps: Step S1: Simultaneously acquire depth images and RGB images of the area to be harvested in front of the combine harvester using a depth camera, and align the depth images and RGB images. Step S2: Based on the aligned image, extract the region of interest, identify the crop spike through image processing, and calculate the plant height by combining depth information; S3. Using an instance segmentation model improved based on deep learning, identify the boundary lines between harvested and unharvested regions in the RGB image; S4. Based on the pixel coordinates of the boundary line, the corresponding depth value, and the camera parameters, the real-time cutting width of the combine harvester is calculated through coordinate transformation. S5. Obtain the inertial measurement unit data of the depth camera, and dynamically estimate and correct the camera's attitude angle through a sensor fusion algorithm. The attitude angle is used to calculate and correct the plant height and the cutting width.

2. The method for measuring the height of the crop to be harvested and the cutting width of a combine harvester according to claim 1, characterized in that, The calculation of plant height in step S2 specifically involves: converting the RGB image of the region of interest to the HSV color space for threshold segmentation to obtain the crop spike region; mapping the pixel coordinates of the crop spike region to a depth image to obtain distance information; and calculating the plant height H according to the following formula based on the camera installation height, camera pitch angle, and distance from the spike to the camera: Where H1 is the camera installation height, and L i Let θ be the distance from the ear of grain corresponding to the i-th pixel to the camera, θ be the camera pitch angle, and n be the number of effective pixels.

3. The method for measuring the height of the crop to be harvested and the cutting width of a combine harvester according to claim 1, characterized in that, The calculation of the cutting width in step S4 is specifically as follows: The specific method for calculating the cut width is as follows: calculate the horizontal absolute distance w2 of the feature points on the boundary line relative to the depth camera, and combine it with the fixed distance w1 of the left divider relative to the camera to obtain the cut width w: 。 4. The method for measuring the height of the crop to be harvested and the cutting width of a combine harvester according to claim 1, characterized in that, The improved instance segmentation model described in step S3 is the YOLO11-SCC model. Its structural improvements include: introducing the Swin Transformer module to replace the original C3k2 module in the Neck part of YOLO11, which is used to model global context information through a self-attention mechanism; adding a Context Aggregation module to the Neck to fuse multi-scale context features to enhance boundary awareness; and replacing the original C3k2 module with the lightweight module CSPPC in the Backbone part to reduce redundant computation and improve feature representation capabilities.

5. The method for measuring the height of the crop to be harvested and the cutting width of a combine harvester according to claim 4, characterized in that, The training of the YOLO11-SCC model includes the following steps: S3.

1. Collect RGB images containing unharvested areas under different conditions, manually label the unharvested areas in the images, and convert the labeling data into a label file in YOLO instance segmentation format; S3.2 Perform data augmentation processing on the image and its corresponding label, including at least one of rotation, translation, scaling and horizontal flipping, and divide it into training set, validation set and test set according to a preset ratio; S3.

3. The YOLO11-SCC model is trained using a multi-task joint loss function, wherein the loss function is: Where, λ box =7.5, λ cls =0.5, λ df1 =1.5, λ seg =1.0; S3.

4. Stochastic gradient descent is used as the optimizer to tune the model hyperparameters, with an initial learning rate of 0.001, a batch size of 32, a momentum coefficient of 0.937, and a weight decay coefficient of 0.0005. S3.

5. Evaluate the performance of the trained model based on the test set. The evaluation metrics include at least one of the following: accuracy, recall, mean precision, mean intersection-union ratio, total floating-point operations, number of parameters, and inference speed.

6. The method for measuring the height of the crop to be harvested and the cutting width of a combine harvester according to claim 1, characterized in that, The coordinate transformation in step S4 specifically involves converting the pixel coordinates (u, v) and depth value d of the boundary point into a 3D point in the camera coordinate system using the camera intrinsic parameters; then, through the rigid body transformation matrix between the camera and vehicle coordinate systems, transforming it to the vehicle coordinate system to obtain the spatial position of the boundary point, and then calculating the cut width.

7. The method for measuring the height of the crop to be harvested and the cutting width of a combine harvester according to claim 1, characterized in that, Step S5 includes the following sub-steps: S5.1 Obtain triaxial angular velocity from the inertial measurement unit and triaxial acceleration ; S5.2, Based on angular velocity about the Y-axis The original pitch angle estimate is obtained by integration. ; S5.

3. The static pitch angle estimate is obtained by calculating the direction of gravity based on accelerometer data. ; S5.4 Construct an extended Kalman filter to define the system state as including the pitch angle. Gyroscope zero bias error The two-dimensional state vector is fused with gyroscope and accelerometer data, and the optimal pitch angle estimate is obtained through prediction and update steps. S5.5 Similarly, the optimal estimate of the roll angle φ is obtained.

8. A system for measuring the height of crop plants to be harvested and the cutting width of a combine harvester, used to implement the method for measuring the height of crop plants to be harvested and the cutting width of a combine harvester as described in any one of claims 1 to 7, characterized in that, It includes a depth camera, an inertial measurement unit, and an image processing module; The depth camera is used to simultaneously acquire depth images and RGB images of the area to be harvested in front of the combine harvester; The inertial measurement unit is integrated into the depth camera or set up separately, and is used to output triaxial angular velocity and triaxial acceleration; The image processing module includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method for measuring the height of the crop to be harvested and the cutting width of the combine harvester according to any one of claims 1 to 7, including: The depth image acquired by the depth camera is aligned with the RGB image. Based on the aligned image, the region of interest is extracted, and crop heads are identified through image processing. The plant height is calculated by combining the depth information. An instance segmentation model based on deep learning is used to identify the boundary line between the harvested and unharvested areas in the RGB image. Based on the pixel coordinates of the boundary line, the corresponding depth value, and the camera parameters, the real-time cutting width of the combine harvester is calculated through coordinate transformation. The inertial measurement unit data of the depth camera is acquired, and the camera attitude angle is dynamically estimated and corrected through a sensor fusion algorithm. The attitude angle is used to correct the calculation of the plant height and the cutting width.

9. The system for measuring the height of the crop to be harvested and the cutting width of the combine harvester according to claim 8, characterized in that, The depth camera is an Intel RealSense D435i depth camera (1), and the processor is a Jetson series embedded processor (2).

10. A harvester, characterized in that, Includes the harvester crop height and cutting width measurement system as described in any one of claims 8 or 9.

Images