A method and system for estimating ear density of a combine harvester to-be-harvested area based on airborne RGB-D vision, and a combine harvester

Abstract

The application provides a method and system for estimating ear density of a combine harvester based on an on-board RGB-D vision, and a combine harvester. The method collects RGB images and depth images of a front area to be harvested by an RGB-D camera installed on the front of the combine harvester and registers the images; determines a region of interest (ROI) in the registered images, inputs an image region corresponding to the ROI into a trained RT-DETR-based ear detection model, detects a mature ear target and outputs a detection frame and a confidence; screens effective detection frames according to a confidence threshold, and counts the number N of ears in the ROI ROI ; reads depth values of four corner points of the ROI, converts pixel coordinates of the corner points into spatial coordinates by using internal and external parameters of the camera, calculates an actual physical area S ROI corresponding to the ROI; and calculates an ear density per unit area ρ according to N ROI and S ROI . The application can convert ear detection results in an image space into an ear density in an actual physical space, and provide perception information of a dense degree of an ear layer for front feed control of the combine harvester.

Description

Technical Field

[0001] This invention belongs to the field of intelligent sensing technology for agricultural machinery and feedforward control technology for combine harvester operating parameters. Specifically, it relates to a method, system, and harvester for estimating ear density in the harvesting area of ​​a combine harvester based on airborne RGB-D vision. Background Technology

[0002] Feed rate is a crucial factor affecting the operational stability, harvest quality, and overall efficiency of combine harvesters. Current methods for monitoring combine harvester feed rate largely rely on torque and speed signals from key components such as the feeding auger, conveyor trough, and threshing drum, or indirectly reflect the feeding status through grain flow signals at the grain bin inlet. These methods typically only reflect load changes after the crop has entered the machine, constituting post-harvest feedback detection and failing to obtain crop status information in the harvestable area before the crop enters the machine.

[0003] Among the crop state parameters affecting the feed rate of combine harvesters, ear density reflects the number and spatial distribution characteristics of mature ears within the harvesting area, serving as a crucial visual parameter characterizing the density of the ear layer and the local potential feed intensity. Compared to relying solely on crop height or cutting width, the number and spatial distribution of ears more directly reflect changes in the biomass of the ear layer about to enter the combine harvester within a unit operating area. Therefore, real-time acquisition of ear density per unit area in the harvesting zone ahead of the combine harvester provides a feedforward sensing basis for the harvester.

[0004] Existing methods for obtaining the number of ears of grain mainly include manual quadrat surveys, UAV remote sensing image analysis, handheld device image detection, and static phenotyping platform detection. Manual quadrat counting is labor-intensive and highly subjective, making it difficult to achieve continuous, real-time, and regional detection. While UAVs or static phenotyping platforms can acquire ear distribution information within a certain range, their acquisition perspective, operating methods, and real-time requirements differ significantly from those of combine harvesters in field operations, making it difficult to directly meet the online sensing needs of forward-looking harvesters. Some ear detection methods based on two-dimensional images can output the number of ears of grain in an image, but they usually only count at the image space level and cannot convert the detection results into ear density per unit area in actual physical space.

[0005] Furthermore, mature grain ears typically exhibit characteristics such as small target size, dense distribution, bending and drooping, mutual occlusion, local overlap, and blurred boundaries. Under the conditions of forward-looking airborne acquisition by combine harvesters, factors such as machine vibration, changes in viewing angle, light fluctuations, and motion blur further increase the difficulty of detecting mature ears and estimating ear density. Existing methods still suffer from problems such as missed detection of small targets, adhesion of dense targets, false detection in complex backgrounds, and difficulty in mapping image counting results to physical density parameters in complex field environments.

[0006] Therefore, existing methods for monitoring the feed amount of combine harvesters have problems such as delayed feedback, difficulty in adapting existing ear count methods to the online scenario of combine harvesters looking ahead, and difficulty in converting two-dimensional image count results into actual physical spatial density parameters. There is an urgent need for an ear density estimation method suitable for the forward-looking operation scenario of combine harvesters, so as to realize the real-time perception of ear density per unit area in the area to be harvested ahead. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a method, system, and harvester for estimating ear density in the harvesting area of ​​a combine harvester based on airborne RGB-D vision. The method uses an airborne RGB-D vision sensor to obtain the target number of ears in the harvesting area ahead, and combines it with a depth image to calculate the actual physical area corresponding to the region of interest, thereby obtaining the ear density per unit area, providing feedforward sensing parameters for the combine harvester.

[0008] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0009] A method for estimating ear density in the harvesting area of ​​a combine harvester based on airborne RGB-D vision includes the following steps:

[0010] Step S1: Acquire RGB and depth images of the harvested area in front of the combine harvester using an RGB-D camera installed at the front of the combine harvester, and spatially register the RGB and depth images.

[0011] Step S2: Determine the region of interest (ROI) in the registered RGB image and obtain the corresponding position of the ROI in the RGB image and the depth image;

[0012] Step S3: Input the RGB image region corresponding to the ROI into the trained RT-DETR-based ear head detection model to detect mature ear head targets and obtain the ear head target detection box and its confidence score;

[0013] Step S4: Filter the detection frames of ear heads within the ROI according to a preset confidence threshold, and use the number of valid detection frames after filtering as the number of ear heads within the ROI. ;

[0014] Step S5: Based on the corner pixel coordinates, corresponding depth values, camera intrinsic parameters, and camera extrinsic parameters of the ROI, convert the ROI corner coordinates from pixel coordinates to spatial coordinates, and calculate the actual physical area corresponding to the ROI based on the spatial coordinates. ;

[0015] Step S6: Based on the number of ears within the ROI and ROI actual physical area Calculate the density of ears per unit area and the ear density As a feedforward sensing parameter output for the combine harvester feed rate.

[0016] In the above scheme, the ear head detection model based on RT-DETR includes Faster-EMA Backbone, DAttention Encoder, and P2-guided GCConvC3 Fusion Neck;

[0017] The Faster-EMA Backbone is used to enhance the feature extraction of shallow textures, edges, and local salient regions of the spikelet against complex backgrounds.

[0018] The DAttention Encoder is used to enhance long-range dependency modeling of high-level semantic features; the P2-guided GCConvC3 Fusion Neck is used for shallow detail-guided multi-scale context fusion.

[0019] Furthermore, the Faster-EMA Backbone is used to process the input feature map. Divide into branches to be convolved along the channel dimension. and retain branches :

[0020]

[0021] in, , ,

[0022] Perform a 3×3 convolution on the convolutional branch and preserve the branch. Using identity mapping:

[0023]

[0024]

[0025] Will and By concatenating the channels, we obtain partial convolutional output features:

[0026]

[0027] Will Divided by channel The feature group, denoted as the i-th feature group. The feature groups are Global average pooling is performed along both the horizontal and vertical directions:

[0028]

[0029]

[0030] Will and After concatenation, horizontal attention weights are generated by 1×1 convolution and the sigmoid function. and vertical attention weights :

[0031]

[0032] use and right Recalibration is performed to obtain attention-enhanced features:

[0033]

[0034] Here, ⊙ represents element-wise multiplication. The enhanced features from each group are concatenated and then fused through a 1×1 convolution to obtain the output features of the Faster-EMA Backbone. :

[0035]

[0036] The DAttention Encoder is used to input high-level semantic features. Query features are generated through linear mapping. Key features Sum value characteristics :

[0037]

[0038]

[0039]

[0040] Spatial sampling offset is generated through offset prediction convolution. :

[0041]

[0042] For the Query location Based on the reference sampling position and the learned offset The deformable sampling positions are obtained:

[0043]

[0044] At the deformable sampling location Bilinear interpolation sampling is performed on the key features and value features to obtain and And calculate attention weights:

[0045]

[0046] in, The feature dimension scaling factor is used to finally obtain the first... Output features at each position:

[0047]

[0048] The P2-guided GCConvC3 Fusion Neck is used to perform 1×1 convolution on shallow high-resolution features. Adjust the number of channels:

[0049]

[0050] Based on the spatial dimensions of the feature layers to be fused, Perform scale alignment to obtain the scale-aligned shallow guiding features. :

[0051]

[0052] Will Features fused with target layer Perform channel splicing:

[0053]

[0054] right Generate spatial context weights :

[0055]

[0056] based on Perform global context aggregation:

[0057]

[0058] global context Channel recalibration weights are generated after 1×1 convolution and nonlinear activation. :

[0059]

[0060] use right Enhancement:

[0061]

[0062] Enhanced The input C3 fusion structure is processed through two branches, then concatenated and output as fused features via a 1×1 convolution:

[0063]

[0064] in, represents a non-linear activation function, and ⊙ represents element-wise multiplication.

[0065] In the above scheme, in steps S3 and S4, the ROI image is obtained by cropping from the RGB image based on the coordinates of the upper left and lower right corners of the ROI; the ROI image is then resized and padded before being input into the RT-DETR-based ear detection model; the RT-DETR-based ear detection model outputs the ear detection boxes and their confidence scores, and filters valid detection boxes according to a preset confidence threshold, using the number of valid detection boxes as the number of ears within the ROI. .

[0066] In the above scheme, in steps S3 and S4, the entire RGB image containing the ROI is input into the RT-DETR-based ear detection model to obtain the ear target detection boxes and their confidence scores in the entire image; the coordinates of the center point of each detection box are calculated to determine whether the center point of the detection box is located within the ROI region; if the center point of the detection box falls within the ROI and the confidence score of the detection box is greater than a preset confidence threshold, then the detection box is determined to be a valid ear detection box; the number of valid detection boxes within the ROI is counted as the number of ears within the ROI. .

[0067] In the above scheme, step S5 includes the following sub-steps:

[0068] S5.1 Obtain the pixel coordinates of the four corner points of the ROI, including the top left corner point, the top right corner point, the bottom right corner point, and the bottom left corner point;

[0069] S5.2. Read the depth values ​​corresponding to the four corner points in the registered depth image; when the depth value corresponding to the corner point is greater than 0, take the depth value as the valid depth value; when the depth value corresponding to the corner point is less than or equal to 0 or cannot be read, search for pixels with a depth value greater than 0 in the neighborhood window centered on the corner point, and take the median of the valid depth values ​​in the neighborhood window as the alternative depth value of the corner point; if there are still no pixels with a depth value greater than 0 in the neighborhood window, retain the depth value of the corner point as an invalid value, and exclude or mark it in the subsequent area estimation;

[0070] S5.3. Convert the ROI corner points from pixel coordinates to three-dimensional coordinates in the camera coordinate system based on the camera intrinsic parameters;

[0071] S5.4. Based on the camera's extrinsic parameters, transform the three-dimensional coordinates to the world coordinate system or the vehicle coordinate system;

[0072] S5.5 Calculate the top width, bottom width, left length, and right length of the ROI based on the transformed spatial coordinates of the four corner points, and calculate the actual physical area corresponding to the ROI based on the top width, bottom width, left length, and right length. .

[0073] In the above scheme, in step S5.3, the first... The three-dimensional coordinates of each corner point in the camera coordinate system , , Calculate using the following formula:

[0074]

[0075] in, For the first The pixel coordinates of the corner points For the first Depth values ​​of each corner point and These are the focal length parameters of the camera in the horizontal and vertical directions, respectively. and The coordinates of the camera's principal point;

[0076] In step S5.4, the first The position of each corner point in the world coordinate system or vehicle coordinate system Calculate using the following formula:

[0077]

[0078] in, For the first The position of each corner point in the camera coordinate system Let be a rotation matrix. It is a translation vector.

[0079] In the above scheme, the actual physical area corresponding to the ROI. Calculate as follows:

[0080]

[0081]

[0082]

[0083] in, The width of the top edge of the ROI. This represents the width of the bottom edge of the ROI. This represents the actual average width of the ROI. The length of the left side of the ROI. The length of the right side of the ROI. This represents the actual average length of the ROI. The actual physical area corresponding to the ROI, in units of .

[0084] A system for estimating ear density in the harvesting area of ​​a combine harvester based on airborne RGB-D vision is used to implement the method for estimating ear density in the harvesting area of ​​a combine harvester based on airborne RGB-D vision. The system includes an RGB-D camera, an image processing module, an ear detection module, a ROI physical area estimation module, and an ear density calculation module.

[0085] The RGB-D camera is used to acquire RGB and depth images of the area to be harvested in front of the combine harvester;

[0086] The image processing module is used to register RGB images and depth images and determine the region of interest (ROI).

[0087] The ear-head detection module is used to detect mature ear targets within the ROI using the RT-DETR-based ear-head detection model, and outputs the ear-head detection box and its confidence level.

[0088] The ROI physical area estimation module is used to calculate the actual physical area of ​​the ROI based on the ROI corner depth value, camera intrinsic parameters, and camera extrinsic parameters.

[0089] The ear density calculation module is used to calculate the ear density per unit area based on the number of ears in the ROI and the actual physical area of ​​the ROI, and output the ear density per unit area to the display terminal, storage module or host computer.

[0090] A combine harvester includes the aforementioned ear density estimation system for the harvesting area of ​​the combine harvester based on onboard RGB-D vision.

[0091] Compared with the prior art, the beneficial effects of the present invention are:

[0092] This invention uses an RGB-D camera at the front of a combine harvester to acquire RGB and depth images of the harvested area in real time. It then uses a trained RT-DETR-based ear detection model to obtain the number of mature ear targets and combines depth information to calculate the actual physical area of ​​the ROI, thus realizing the conversion from the image space ear detection results to the actual physical space ear density per unit area.

[0093] The RT-DETR-based ear detection model of this invention improves the detection accuracy of the model in mature, dense, small-scale, occluded and complex background scenes by introducing structural designs such as Faster-EMA Backbone, DAttentionEncoder and P2-guided GCConvC3 Fusion Neck.

[0094] This invention utilizes an RGB-D camera to simultaneously acquire color and depth images. It features a simple hardware structure, high system integration, and adaptability to forward-looking airborne operation scenarios of combine harvesters. It also demonstrates good real-time performance and engineering application value.

[0095] The ear density information obtained by this invention can reflect the ear density and local crop biomass changes in the area to be harvested in advance, providing sensing information on ear density for the feedforward control of the combine harvester. Attached Figure Description

[0096] Figure 1 This is a schematic diagram illustrating the installation and coordinate transformation relationship of an RGB-D camera according to an embodiment of the present invention; wherein: 1, RGB-D camera; 2, embedded processor. The RGB-D camera is installed at the front of the combine harvester, and a camera coordinate system O is established. c -X c Y c Z c And the world coordinate system or vehicle coordinate system O w -X w Y w Z w Among them, h c θ represents the camera installation height, θ represents the installation tilt angle of the camera optical axis relative to the reference direction, and L represents the spatial distance between the area to be harvested, which is pointed to by the camera's field of view, and the camera.

[0097] Figure 2 This is a flowchart illustrating the overall process of estimating ear density according to an embodiment of the present invention;

[0098] Figure 3This is a structural diagram of an improved RT-DETR ear head detection model according to an embodiment of the present invention; wherein, RT-DETR represents the real-time detection Transformer, Faster-EMA Backbone represents the feature extraction backbone network that fuses partial channel convolution and grouped multi-scale attention, DAttention Encoder represents the deformable attention encoder, P2-guidedGCConvC3 Fusion Neck represents the P2 shallow feature-guided global context convolution and C3 multi-scale fusion neck, FPN / PAN Neck represents the feature pyramid network / path aggregation network fusion neck, Object Queries (N=500) represents the number of object queries as 500, RT-DETR Decoder represents the RT-DETR detection decoder, Detection Head represents the detection head, and P2 / 4, P3 / 8, P4 / 16 and P5 / 32 represent multi-scale feature layers that are downsampled by 4 times, 8 times, 16 times and 32 times relative to the input image, respectively.

[0099] Figure 4 This is a flowchart of ROI image cropping and ear detection and counting according to one embodiment of the present invention;

[0100] Figure 5 This is a flowchart of ROI corner depth acquisition and 3D coordinate transformation according to an embodiment of the present invention;

[0101] Figure 6 This is a flowchart illustrating the estimation of the average width and length area of ​​the ROI and the calculation of the ear density according to one embodiment of the present invention.

[0102] Figure 7 This is a flowchart showing the output of the number and density of panicles according to one embodiment of the present invention.

[0103] Figure 8 This is an online display interface diagram of a panicle density estimation system according to an embodiment of the present invention; wherein, RicePanicle Count and ROI Area Estimation System represents the panicle counting and ROI area estimation system, ROIPanicle Count represents the number of panicles in the ROI, ROI Physical Area represents the actual physical area of ​​the ROI, PanicleDensity represents the panicle density per unit area, Depth Image represents the depth image, and ROI and Panicle DetectionResult represents the detection results of the ROI region and panicles.

[0104] Figure 9This is a statistical chart showing the prediction results and errors of the number of rice panicles under different light intensities according to an embodiment of the present invention; where Strong light, Weak light, and Low light represent strong light, weak light, and low light conditions, respectively; True number of rice panicles represents the actual number of rice panicles; Predicted number of rice panicles represents the predicted number of rice panicles; V Channel Average represents the average value of the V channel in the HSV color space; MAE, RMSE, and MRE represent the mean absolute error, root mean square error, and mean relative error, respectively.

[0105] Figure 10 This is a statistical graph showing the prediction results and errors of the number of rice panicles under different panicle densities according to an embodiment of the present invention; where Low density, Medium density, and High density represent low density, medium density, and high density conditions, respectively; True number of rice panicles represents the actual number of panicles; Predicted number of ricepanicles represents the predicted number of panicles; Sample size represents the sample size; MAE, RMSE, and MRE represent the mean absolute error, root mean square error, and mean relative error, respectively.

[0106] Figure 11 This is a statistical distribution diagram of the error in predicting the number of ears of grain under different forward speeds according to an embodiment of the present invention. The left figure shows the statistical results of MAE, RMSE, and MRE of the combine harvester at forward speeds of 0.9 m / s, 1.29 m / s, and 1.4 m / s, and the right figure shows the distribution of absolute error at the corresponding forward speeds. Forward speed represents the forward speed, Absolute error represents the absolute error, and MAE, RMSE, and MRE represent the mean absolute error, root mean square error, and mean relative error, respectively.

[0107] Figure 12This is a statistical and cumulative distribution chart of the prediction error of the number of panicles in different rice varieties according to an embodiment of the present invention. The left chart shows the MAE, RMSE, and MRE statistical results of two rice varieties, Huiliangyou and Yangnong No. 1, and the right chart shows the cumulative distribution of absolute error for the two rice varieties. Rice variety represents rice variety, Variety 1 represents Huiliangyou, Variety 2 represents Yangnong No. 1, Absolute error represents absolute error, Cumulative proportion represents cumulative proportion, and Cumulative distribution of absolute error represents the cumulative distribution of absolute error.

[0108] Figure 13 This is a field trial and artificial quadrat counting diagram according to an embodiment of the present invention, wherein, Figure 13 (a) is a schematic diagram of the layout of the field test area and the sampling area. Figure 13 (b) is a field diagram of artificial quadrats for counting ears of grain.

[0109] Figure 14 This is a schematic diagram of the ear tip detection results under low light, medium light and strong light conditions according to an embodiment of the present invention.

[0110] Figure 15 This is a schematic diagram of the ear head detection results under low-density, medium-density, and high-density conditions according to an embodiment of the present invention.

[0111] Figure 16 This is a schematic diagram showing the ear head detection results when the forward speed of the combine harvester is 0.9 m / s, 1.29 m / s and 1.4 m / s, respectively, according to one embodiment of the present invention.

[0112] Figure 17 This is a schematic diagram showing the panicle head detection results of two rice varieties, Huiliangyou and Yangnong No. 1, according to an embodiment of the present invention. Detailed Implementation

[0113] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. The following embodiments are used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Equivalent substitutions or modifications made by those skilled in the art to the various embodiments without departing from the technical concept of the present invention should all fall within the scope of protection of the present invention.

[0114] like Figure 1As shown, exemplarily, an RGB-D camera 1 is mounted at the front of the combine harvester to acquire RGB and depth images of the area to be harvested in front of the harvester. In one embodiment, the RGB-D camera 1 is an Intel RealSense D435i depth camera. An embedded processor 2 is disposed inside the combine harvester body and is used to perform image registration, ROI region determination, ear target detection, ROI actual physical area estimation, and ear density calculation. Optionally, the embedded processor 2 is a Jetson series embedded processor. The system also includes a display terminal for real-time display of the RGB images, depth images, ROI regions, ear detection results, ROI actual physical area, and ear density per unit area.

[0115] like Figure 2 As shown, a method for estimating ear density in the harvesting area of ​​a combine harvester based on airborne RGB-D vision according to an embodiment of the present invention includes the following steps:

[0116] Step S1, RGB-D Image Acquisition and Registration: RGB and depth images of the area to be harvested in front of the combine harvester are acquired by an RGB-D camera installed at the front of the combine harvester, and the RGB and depth images are spatially registered.

[0117] Specifically, RGB and depth images of the harvested area are acquired using an RGB-D camera 1 mounted at the front of the combine harvester. The RGB images are used for target detection of mature ears of grain, and the depth images are used for estimating the actual physical area of ​​the Region of Interest (ROI). During acquisition, spatial registration is performed on the RGB and depth images to establish a mapping relationship between the position of each pixel in the RGB image and the corresponding depth value in the depth image.

[0118] In one implementation, the RGB image resolution is 1920×1080 pixels, and the model input size is adjusted to 640×640 pixels. Depth images are acquired synchronously with the RGB images and registered using camera calibration parameters. To improve system real-time performance, a multi-threaded architecture can be used for image acquisition, model inference, and result display, and a frame interval processing strategy can be set according to the computing power of the embedded platform.

[0119] Step S2, ROI region determination: Determine the region of interest (ROI) in the registered RGB image, and obtain the corresponding position of the ROI in the RGB image and the depth image;

[0120] Specifically, a region of interest (ROI) is determined in the registered RGB image. The ROI is selected as a rectangular window in the center of the image, representing the area to be harvested before the combine harvester enters the header. Since image edge areas and distant areas are easily affected by perspective distortion, target blurring, and background interference, selecting the image center as the ROI improves the stability of ear count.

[0121] In one embodiment, the Region of Interest (ROI) is set as a fixed rectangular window with a pixel size of 300 px × 226 px. During the installation and debugging phase, the camera's installation position and shooting angle can be adjusted so that the preset 300 px × 226 px fixed center ROI covers the target harvesting area in front of the combine harvester; during actual detection, the position and pixel size of the ROI remain unchanged. Since the distance between the camera and the crop canopy changes with the field operation conditions, the actual physical area corresponding to the fixed pixel size ROI is not a constant value. Therefore, it is necessary to dynamically estimate the actual physical area of ​​the ROI by combining depth images.

[0122] In one specific implementation, the width w and height h of the RGB image are read, and the coordinates of the center point of the RGB image are calculated as ( Using this center point as the center of the ROI, set the left boundary of the ROI to... The right boundary is set to The upper boundary is set to The lower boundary is set to This results in a central rectangle ROI with a width of 300 px and a height of 226 px. If the ROI boundary exceeds the valid range of the RGB image, the ROI boundary is then restricted to the valid pixel range of the RGB image.

[0123] Step S3, Detection of mature ear heads: Input the RGB image region corresponding to the ROI into the trained RT-DETR-based ear head detection model to detect mature ear heads and obtain the ear head target detection box and its confidence score;

[0124] Specifically, the RGB image region corresponding to the ROI is input into a trained RT-DETR-based ear head detection model to detect mature ear heads and obtain the ear head target detection box and its confidence score.

[0125] In one embodiment, the ROI image is first cropped from the registered RGB image based on the ROI region determined in step S2. The ROI image is then resized and padded to fill the model input size while maintaining the original aspect ratio. Subsequently, the ROI image is converted to the model input format and input into a trained RT-DETR-based ear detection model. The RT-DETR-based ear detection model outputs the detection bounding boxes and confidence scores of mature ear targets within the ROI image.

[0126] In one specific implementation, the ROI image is resized and padded to form a 640×640 pixel model input image, and after normalization, it is input into the RT-DETR ear detection model deployed on the TensorRT inference engine to improve the real-time detection performance in the combine harvester airborne scenario.

[0127] As shown in Figure 3, in one embodiment, the trained RT-DETR-based ear detection model is an improved RT-DETR ear detection model, which includes a Faster-EMA Backbone, a DAttention Encoder, a P2-guided GCConvC3 Fusion Neck, an RT-DETR Decoder, and a Detection Head. In Figure 3, the Faster-EMA Backbone is used to extract shallow texture, edge, and local salient region features from the ear image; the DAttention Encoder is used to enhance the long-range dependency modeling capability of high-level semantic features through learnable spatial offset sampling; the P2-guided GCConvC3 Fusion Neck is used to introduce P2 shallow high-resolution features into the FPN / PAN multi-scale fusion neck and enhance the boundary representation and localization capability of dense small-scale ear targets at maturity through global contextual convolution; Object Queries (N=500) are used to generate candidate target representations during the decoding stage; and the RT-DETR Decoder and Detection Head are used to output the ear target detection box and its confidence score.

[0128] The Faster-EMA Backbone is used to process the input feature map. Divide into branches to be convolved along the channel dimension. and retain branches :

[0129]

[0130] in, , ,

[0131] Perform a 3×3 convolution on the convolutional branch and preserve the branch. Using identity mapping:

[0132]

[0133]

[0134] Will and By concatenating the channels, we obtain partial convolutional output features:

[0135]

[0136] Will Divided by channel The feature group, denoted as the i-th feature group. The feature groups are Global average pooling is performed along both the horizontal and vertical directions:

[0137]

[0138]

[0139] Will and After concatenation, a 1×1 convolution is performed to obtain the spatial attention response, which is then processed by a sigmoid function to generate the horizontal attention weights. and vertical attention weights :

[0140]

[0141] use and right Recalibration is performed to obtain attention-enhanced features:

[0142]

[0143] Here, ⊙ represents element-wise multiplication. The enhanced features from each group are concatenated and then fused through a 1×1 convolution to obtain the output features of the Faster-EMA Backbone. :

[0144]

[0145] Unlike using only partial convolutional structures in FasterNet, the Faster-EMA Backbone in this invention does not simply perform convolution on a subset of channels and output directly. Instead, after partial channel convolution, it further performs grouped multi-scale attention recalibration on all concatenated channels. This allows channels that were not involved in the 3×3 convolution to participate in subsequent attention weight allocation, thus balancing inference speed, preservation of shallow details, and effective feature enhancement in complex field backgrounds. Unlike using EMA attention alone, this invention integrates partial channel convolution and grouped multi-scale attention into the same Backbone feature extraction block, enabling convolution redundancy reduction and attention enhancement to work together for the task of detecting small targets on mature spikelets.

[0146] The DAttention Encoder includes a query feature generation unit, a spatial offset prediction unit, a deformable sampling unit, and an attention weighted aggregation unit. Let's assume that based on the input high-level semantic features... Query features are generated through linear mapping. Key features Sum value characteristics :

[0147]

[0148]

[0149]

[0150] Based on input features Spatial sampling offset is generated through offset prediction convolution. :

[0151]

[0152] For the Query location Based on the reference sampling position and the learned offset The deformable sampling positions are obtained:

[0153]

[0154] At the deformable sampling location Bilinear interpolation sampling is performed on the key features and value features to obtain and And calculate attention weights:

[0155]

[0156] in, The feature dimension scaling factor is used to finally obtain the first... Output features at each position:

[0157]

[0158] In this way, the DAttention Encoder can adaptively adjust the sampling position of high-level semantic features based on the bending, drooping, occlusion, and local overlap of the ear-like targets in the image. Compared with fixed-window self-attention, this structure does not compute attention on a fixed regular grid, but obtains learnable sampling positions through offset prediction, thereby improving the expressive power of the spatial relationships of dense, small-scale ear-like targets.

[0159] The P2-guided GCConvC3 Fusion Neck includes a P2 shallow feature guidance unit, a scale alignment unit, a feature concatenation unit, a global context convolution unit, and a C3 fusion output unit. Let the shallow high-resolution features be... High-level semantic features are , and First, shallow high-resolution features are processed using 1×1 convolution. Adjust the number of channels:

[0160]

[0161] Based on the spatial dimensions of the feature layers to be fused, Perform upsampling or downsampling to integrate the features with the target fusion layer. With consistent spatial dimensions, scale alignment is performed to obtain the scale-aligned shallow guiding features. :

[0162]

[0163] Will Features fused with target layer Perform channel splicing:

[0164]

[0165] In the global context convolutional unit, first Generate spatial context weights :

[0166]

[0167] based on Perform global context aggregation:

[0168]

[0169] global context Channel recalibration weights are generated after 1×1 convolution and nonlinear activation. and to Enhancement:

[0170]

[0171] use right Enhancement:

[0172]

[0173] Enhanced The input is a C3 fusion structure, which includes two branches: one branch is processed by several bottleneck units containing global context convolutions, and the other branch retains shallow guiding information. Finally, the two branches are concatenated and the fused features are output through a 1×1 convolution.

[0174]

[0175] in, represents a non-linear activation function, and ⊙ represents element-wise multiplication.

[0176] Compared to the ordinary C3 structure, the GCConvC3 in this invention is not composed solely of conventional convolutional bottlenecks, but rather introduces global context weights during the bottleneck feature fusion process. This enables the model to establish a connection between local edge texture features and global spikelet distribution features. Unlike conventional FPN or PAN multi-scale fusion, this invention further introduces shallow P2 high-resolution features as guiding information to supplement the edge, texture, and position information that is easily lost during multiple downsampling processes of small-scale spikelet targets at maturity.

[0177] Furthermore, considering the characteristics of numerous, small-scale, densely distributed, and significantly partially occluded ear heads in mature field front-view images, in a preferred embodiment, the number of target queries for the RT-DETR ear head detection model is set to 500 to improve the coverage and detection capabilities of candidate targets in dense, small-target scenarios. The target queries are used to generate candidate target representations during the decoding stage. The 500 target queries do not imply that 500 ear heads must be detected within the ROI, but rather enhance the model's ability to represent dense ear head candidate targets. In actual counting, only valid detection boxes filtered by the confidence threshold are counted.

[0178] Compared to a target query count of 300, setting the target query count to 500 provides more candidate target representation locations in images with a large number of dense, small-scale ear-like targets, thereby reducing the probability of missed detections of dense ear-like targets. Furthermore, compared to further increasing the target query count, 500 target queries offer a better trade-off between detection accuracy and inference speed, meeting the real-time requirements of the combine harvester's onboard platform. The ablation test results (see Table 4) show that setting the target query count to 500 enhances the model's candidate coverage of small, dense ear-like targets at maturity, reduces missed detections, and achieves superior overall performance in metrics such as Precision, Recall, F1-score, mAP@0.5, and AP_small@0.5:0.95. This indicates that this setting can improve the detection capability of dense, small targets while maintaining good real-time inference performance.

[0179] Step S4: Count the number of ears within the ROI: Filter the ear detection frames within the ROI according to a preset confidence threshold, and use the number of valid detection frames after filtering as the number of ears within the ROI. ;

[0180] Specifically: such as Figure 4 As shown, in one embodiment, the ROI image is obtained by cropping the ROI region determined in step S2, and the cropped ROI image is input into the trained RT-DETR-based ear detection model to obtain the target detection boxes of mature ears within the ROI image and their confidence scores. Subsequently, the detection boxes output by the model are filtered by a confidence threshold, retaining ear detection boxes with a confidence score greater than a preset threshold, and the number of valid detection boxes after filtering is taken as the number of ears within the ROI. .

[0181] In one specific implementation, the preset confidence threshold can be set based on field lighting conditions, ear density, and model detection results. When the confidence level of the detection frame is greater than the preset confidence threshold, the detection frame is determined to be a valid ear count frame; when the confidence level of the detection frame is less than or equal to the preset confidence threshold, it is not included in the ear count within the ROI. This method reduces the impact of complex field backgrounds, leaf shading, and local overlap on ear count results.

[0182] In another embodiment, the entire RGB image containing the ROI can be input into the trained RT-DETR-based ear detection model to obtain ear target detection boxes in the entire image. Then, the coordinates of the center point of each detection box are calculated, and it is determined whether the center point of the detection box is located within the ROI. If the center point of the detection box falls within the ROI and the confidence score of the detection box is greater than a preset confidence threshold, the ear target corresponding to that detection box is considered a valid ear target. If the center point of the detection box is outside the ROI, or the confidence score of the detection box is less than or equal to the preset confidence threshold, it is not counted in the number of ear targets within the ROI. Finally, the number of valid ear targets within the ROI is counted and recorded as follows: .

[0183] In a preferred embodiment, the ROI image is cropped first and then the ear head detection is performed to reduce the interference of background information in non-ROI areas on the ear head detection results and reduce the proportion of irrelevant background in the model inference input area, thereby improving the stability and real-time performance of ear head number statistics within the ROI.

[0184] Step S5: ROI Corner Depth Acquisition, 3D Coordinate Transformation, and Actual Physical Area Estimation: Based on the ROI's corner pixel coordinates, corresponding depth values, camera intrinsic parameters, and camera extrinsic parameters, the ROI corner coordinates are converted from pixel coordinates to spatial coordinates, and the actual physical area corresponding to the ROI is calculated based on the spatial coordinates. ;

[0185] As shown in Figure 5, let the pixel coordinates of the top-left and bottom-right corners of the ROI be (x0, y0) and (x1, y1) respectively. Then the four corner points of the ROI can be represented as:

[0186] P1(x0, y0);

[0187] P2(x1-1, y0);

[0188] P3(x1-1, y1-1);

[0189] P4(x0, y1-1).

[0190] Where P1, P2, P3, and P4 represent the top left, top right, bottom right, and bottom left corners of the ROI, respectively.

[0191] For each ROI corner, the corresponding depth value is obtained from the registered depth image. If the depth value corresponding to the corner is greater than 0, it is taken as a valid depth value and used directly for 3D coordinate transformation. If the depth value corresponding to the corner is less than or equal to 0 or cannot be read, the corner depth value is determined to be invalid. When the corner depth value is invalid, pixels with a depth value greater than 0 are searched within a neighborhood window centered on the corner, and the median of the valid depth values ​​within the neighborhood window is taken as the alternative depth value for the corner to reduce the impact of depth holes, noise points, and local outliers on area estimation.

[0192] In one specific implementation, the neighborhood window is a window centered on the corner point of the ROI with a radius of 6 pixels.

[0193] Let the pixel coordinates of the i-th corner point be (u i v i ), corresponding to a depth value of d i Then the three-dimensional coordinates of the corner point in the camera coordinate system can be expressed as:

[0194]

[0195] Among them, (X) i,c Y i,c Z i,c Let F be the three-dimensional coordinates of the i-th corner point in the camera coordinate system. x and F y C represents the focal length parameters of the camera in the horizontal and vertical directions, respectively. x and C y These are the coordinates of the camera's principal point.

[0196] Considering the mounting position and orientation of the RGB-D camera on the combine harvester, the corner coordinates are further transformed from the camera coordinate system to the world coordinate system or the vehicle coordinate system using camera extrinsic parameters. Let the rotation matrix be R and the translation vector be T, then the position of the i-th corner point in the world coordinate system can be expressed as:

[0197] P i,w = RP i,c + T;

[0198] Among them, P i,w Let P be the position of the i-th corner point in the world coordinate system or vehicle coordinate system. i,c Let R be the position of the i-th corner point in the camera coordinate system, and let T be determined by the camera mounting attitude and mounting position, respectively.

[0199] Furthermore, such as Figure 6As shown, based on the spatial coordinates of the four corner points of the ROI, the top width, bottom width, left length, and right length of the ROI are calculated, and the actual physical area corresponding to the ROI is estimated by multiplying the average width and average length. Let the spatial coordinates of the four corner points of the ROI be P1, P2, P3, and P4, respectively, then the top width W of the ROI is... top The planar distance between P1 and P2, and the width W of the lower edge of the ROI. bottom Let L be the planar distance between P4 and P3, and the length L of the left side of the ROI. left Let L be the planar distance between P1 and P4, and the right-hand length of the ROI. right Let P2 be the planar distance between P3 and P3.

[0200] ROI actual average width W ROI and actual average length L ROI Calculate according to the following formulas:

[0201]

[0202]

[0203] The actual physical area S corresponding to ROI ROI Calculate using the following formula:

[0204] S ROI = W ROI × L ROI ;

[0205] Among them, W top W represents the width of the top edge of the ROI. bottom W represents the width of the bottom edge of the ROI. ROI L represents the actual average width of the ROI. left L represents the length of the left side of the ROI. right L represents the length of the right side of the ROI. ROI S represents the actual average length of the ROI. ROI This represents the actual physical area corresponding to the ROI, in m².

[0206] The above method can convert a ROI with a fixed pixel size into an actual physical area, thereby avoiding area estimation errors caused by changes in the distance between the camera and the crop canopy.

[0207] Step S6, Calculation and Result Output of Ear Density per Unit Area: Based on the number of ears per unit area within the ROI. and ROI actual physical area Calculate the density of ears per unit area and the ear density As a feedforward sensing parameter output for the combine harvester feed rate.

[0208] After obtaining the number of ears N within the ROI, ROI And the actual physical area S of ROI ROI Then, calculate the ear density per unit area ρ:

[0209]

[0210] Where ρ is the spike density per unit area, with units of spikes / m²; N ROI S represents the target number of mature ears detected within the ROI, in ears; ROI This represents the actual physical area corresponding to the ROI, in m².

[0211] Through the above calculations, the ear detection results in the RGB image are converted into ear density per unit area in actual physical space. This ear density can reflect the ear layer density and local crop biomass changes in the harvesting area in front of the combine harvester, and can be used as a feedforward visual input for feed rate prediction, forward speed adjustment, header parameter optimization, and anti-blocking control.

[0212] Results output and display:

[0213] like Figure 7 As shown, the number of ears N within the ROI is obtained. ROI ROI (Actual Physical Area S) ROI After calculating the ear density per unit area ρ, the system outputs the coordinates of the ear detection frame, the detection confidence level, the location information of the ROI region, and the number of ears N within the ROI. ROI ROI (Actual Physical Area S) ROI And the density of ears per unit area ρ.

[0214] like Figure 8 As shown, in one embodiment, the output results are displayed in real time via a display terminal. The displayed content includes a depth image, the Region of Interest (ROI), the mature ear detection frame, and the number N of valid ears within the ROI. ROI ROI (Actual Physical Area S) ROI At least one of the following: the density of ears per unit area ρ and the current sample information.

[0215] Furthermore, the status bar of the display terminal can show the current sample number, RGB image file name, depth image file name, ROI pixel size, and the number of ears N within the ROI. ROI ROI (Actual Physical Area S) ROIThe calculation results include the ear count density ρ per unit area. To reduce the impact of single-frame image detection fluctuations on result display and statistical analysis, the ear count or ear density results of several consecutive frames can be processed by moving average or median filtering to obtain smoothed ear count or ear density output values.

[0216] Model training sample construction:

[0217] In one embodiment, images of mature grains, such as mature rice, are collected from a field. The collected images include mature panicle images under different rice varieties, panicle densities, lighting conditions, occlusion levels, and field speeds. Visible mature panicle targets in the images are manually labeled with bounding boxes. Targets that are severely occluded but still have clear panicle outlines are labeled independently; targets whose outlines are not reliably distinguishable or whose visible area is too small are not labeled to reduce the impact of noisy labels on model training.

[0218] In one specific implementation, the constructed mature ear-head dataset contains 3147 images with 531234 labeled mature ear-head targets, and is divided into training, validation, and test sets. The training, validation, and test sets contain 2496, 338, and 313 images respectively, with corresponding labeled targets of 414405, 60251, and 56578, used for model training, parameter tuning, and performance evaluation.

[0219] To further enhance the model's adaptability to complex field environments, data augmentation is performed on the training set images during the training phase. This data augmentation includes at least one of the following: brightness enhancement, brightness reduction, image blurring, and JPEG compression distortion. Brightness enhancement and reduction are used to simulate different lighting conditions, image blurring is used to simulate slight vibrations and motion blur during combine harvester operation, and JPEG compression distortion is used to simulate quality degradation during image transmission and storage. This data augmentation is applied only to the training set; the validation and test sets retain their original images.

[0220] In one implementation, the model input size is set to 640×640, the number of training epochs is set to 300, the optimizer is AdamW, and the initial learning rate is set to 1×10⁻⁶. -4 After training, the model's detection performance is evaluated based on the test set. The evaluation metrics include at least one of the following: Precision, Recall, F1-score, mAP@0.5, mAP@0.5:0.95, AP_small, number of parameters, computational cost, and inference speed.

[0221] Field verification method:

[0222] To verify the effectiveness of the ear density estimation method described in this invention, RGB-D images of the harvestable area were continuously acquired during combine harvester field operations. The ear detection results within the ROI were then output using the trained RT-DETR-based ear detection model deployed on an embedded processor. The system estimates the density based on the number of ears N within the ROI. ROI The physical area S of the ROI estimated from the RGB-D image. ROI Obtained through joint calculation.

[0223] In field reference measurements, fixed-area quadrats can be set up along the direction of the combine harvester's movement on one side of the data collection area, and the visible mature ears within the quadrats can be manually counted in the field. Let N be the number of manually counted ears within the quadrats. m The actual area of ​​the quadrat is S. m The reference ear density ρ obtained by manual measurement m It can be represented as:

[0224]

[0225] System estimation of density ρ s Compared with manually measured density ρ m The difference can be expressed as absolute error E. a and relative error E r express:

[0226] E a = |ρ s - ρ m |;

[0227] E r = × 100%.

[0228] For multiple sets of field validation samples, the mean absolute error (MAE) can be further calculated. ρ and mean relative error (MRE) ρ It is used to evaluate the overall density estimation performance of the system.

[0229] In one embodiment, the robustness of the model can be verified by factors such as different light intensities, different panicle densities, different advancing speeds, and different rice varieties. Verification results show that the present invention can stably output panicle density information for the harvested area under complex field scenarios.

[0230] Validation results of field ear density estimation:

[0231] In one specific embodiment, the field test platform is a tracked self-propelled combine harvester. An RGB-D camera and an embedded processor are mounted at the front of the combine harvester to continuously acquire forward-looking RGB images and depth images of the area to be harvested. Figure 13 As shown, rectangular experimental areas were divided in the experimental field. A combine harvester operated in a straight line along the centerline of each area. The front section served as an adjustment zone to gradually stabilize the harvester's forward speed, header height, and operating posture. The rear section served as a data acquisition zone to collect RGB-D images under stable operating conditions. To obtain a reference ear density in the field, fixed-area quadrats were set up along the harvester's forward direction on one side of the data acquisition zone, and visible mature ears within each quadrat were manually counted. The results of the manual quadrat counting were used to calculate the reference ear density, which was then compared with the system-estimated density calculated based on the number of ears in the ROI and the actual physical area of ​​the ROI.

[0232] In one embodiment, mature rice samples from two rice varieties, Yangnong No. 1 and Huiliangyou, were selected for field verification. These samples exhibited certain differences in panicle number, panicle density, plant structure, and degree of local shading, which can be used to verify the panicle density estimation capability of this invention under different rice varieties and panicle structure conditions.

[0233] During field validation, the system estimated the density based on the number of ears N within the ROI. ROI The physical area S of the ROI estimated from the RGB-D image. ROI The density was obtained through joint calculation; the density was manually measured by counting the number of ears N within a fixed-area quadrat. m and sample area S m The calculated density was obtained. The comparison results between the system-estimated density and the manually measured density are shown in Table 1.

[0234] Table 1. Validation results of field ear density estimation

[0235]

[0236] Table 1 shows that the overall trend of the density estimated by the system is consistent with that of the density measured manually. The average absolute error of the 10 field samples was 9.7 panicles / m², and the average relative error was 4.26%. Among them, the average absolute error of Yangnong No. 1 sample was 9.0 panicles / m², and the average relative error was 4.78%; the average absolute error of Huiliangyou sample was 10.4 panicles / m², and the average relative error was 3.75%. The results indicate that under different rice varieties and different panicle density conditions, the density estimated by the system maintains good consistency with the density measured manually. This shows that the present invention can effectively convert the panicle detection results in image space into panicle density per unit area in actual physical space, and can provide reliable input for characterizing the panicle density in the harvesting area in front of the combine harvester and for adjusting subsequent operation parameters.

[0237] like Figure 14-17 As shown, in a typical field scenario, the trained RT-DETR-based ear detection model can detect mature ear targets in the cropped ROI image and output detection boxes and their confidence scores. Valid detection boxes are filtered by setting a pre-set confidence threshold, and the number of valid detection boxes after filtering is counted to obtain the number of ears N within the ROI. ROI .

[0238] Multifactor field robustness verification:

[0239] To further verify the adaptability of the present invention under complex field operating conditions, in one embodiment, the test set and field-collected samples were grouped and evaluated according to light intensity, panicle density, combine harvester forward speed, and rice variety. This grouping evaluation was used to analyze the stability of the trained RT-DETR-based panicle detection model described in this invention in predicting the number of panicles under different field environments and crop morphological conditions.

[0240] Under different light intensity conditions, the acquired images were divided into three categories: strong light, weak light, and low light. The error between the model-predicted number of ears and the manually labeled reference number of ears was statistically analyzed for each category. For example... Figure 9 and Figure 14 As shown, the method of this invention can maintain relatively stable results in predicting the number of panicles under different light conditions, indicating that the model has a certain adaptability to fluctuations in field light. Under strong light conditions, local overexposure and high-contrast shadows may occur, while under low light conditions, target texture weakening and boundary blurring may occur. However, this invention can mitigate the impact of light changes on panicle detection results through shallow detail guidance, multi-scale feature fusion, and long-range dependency modeling.

[0241] Under different ear density conditions, the images were divided into three categories—low density, medium density, and high density—based on the number of reference ears, and the prediction error of the model under different ear density levels was evaluated. For example... Figure 10 and Figure 15 As shown, the method of this invention can effectively detect ear-like targets under different density conditions. In low-density scenes, the ear-like targets are relatively sparsely distributed, resulting in a lower risk of false detection by the model; in high-density scenes, the mutual occlusion and local overlap of ear-like targets are more obvious, increasing the risk of missed detection by the model. This invention improves the candidate target coverage and spatial relationship modeling capabilities in high-density scenes by increasing the number of object queries and introducing a DAttention Encoder, thereby enhancing the detection stability of dense ear-like targets.

[0242] Under different forward speed conditions, the field operation samples of combine harvesters were grouped according to their forward speed to evaluate the prediction error of the number of ears of grain under onboard dynamic data collection conditions. For example... Figure 11 and Figure 16 As shown, the method of this invention can output relatively stable ear count prediction results at different forward speeds. When the forward speed increases, the image may be affected by motion blur, vibration disturbance, and viewpoint changes, resulting in blurring of some small-scale ear boundaries. This invention improves the robustness of the model under dynamic acquisition conditions through image enhancement training, shallow detail feature fusion, and ROI region constraint statistics.

[0243] Rice samples with different plant type, panicle type, and canopy structure characteristics were selected and grouped for evaluation under different rice variety conditions. For example... Figure 12 and Figure 17 As shown, the method of this invention exhibits good stability in predicting the number of panicles under different rice varieties. Different varieties may differ in panicle size, panicle erectness, degree of bending or drooping, shading intensity, and background texture. This invention enhances the model's effective feature extraction, long-range relationship modeling, and small target detail preservation capabilities by using Faster-EMA Backbone, DAttention Encoder, and P2-guidedGCConvC3 Fusion Neck, thereby improving the model's adaptability to differences in panicle morphology among different varieties.

[0244] In the above multi-factor robustness verification, the error between the model's predicted number of panicles and the human reference number of panicles can be evaluated using the mean absolute error (MAE), root mean square error (RMSE), and mean relative error (MRE). MAE characterizes the average deviation of the prediction results, RMSE reflects the impact of larger errors, and MRE characterizes the relative deviation of the prediction results relative to the human reference value. Group verification under different light conditions, different panicle densities, different advance speeds, and different rice varieties further demonstrates that the method of this invention can adapt to the complex field operation environment of combine harvesters and stably output panicle density information for the harvested area.

[0245] Table 2. Grouping and Evaluation Content of Multifactor Robustness Validation

[0246]

[0247] Comparison and validation of different detection models:

[0248] To verify the effectiveness of the RT-DETR-based panicle detection model used in this invention for detecting panicles in mature rice, in one embodiment, the trained RT-DETR-based panicle detection model described in this embodiment is compared with YOLOv5, YOLOv8, YOLO11, YOLO12, YOLO26, and the original RT-DETR-1 model on the same mature rice panicle dataset. All models use the same dataset partitioning, input image size, and evaluation metrics to ensure the comparability of experimental results. Evaluation metrics include Precision, Recall, F1-score, mAP@0.5, mAP@0.5:0.95, AP_small@0.5:0.95, model parameter count, and inference speed.

[0249] In one specific implementation, the performance comparison of different detection models is shown in Table 3.

[0250] Table 3 Performance Comparison of Different Detection Models

[0251]

[0252] As shown in Table 3, although the YOLO series models have high inference speed and small parameter count, they lag behind RT-DETR-1 and the model of this invention in metrics such as mAP@0.5:0.95 and AP_small@0.5:0.95. This indicates that their fine-grained localization and small target detection capabilities are still insufficient in scenarios with densely distributed mature ears, numerous small-scale targets, and significant occlusion and overlap. The original RT-DETR-1 outperforms most YOLO series models in detection accuracy, but has a large number of parameters. Compared with the original RT-DETR-1, the model of this invention shows improvements in Precision, Recall, F1-score, mAP@0.5, mAP@0.5:0.95, and AP_small@0.5:0.95, while reducing the number of parameters from 28.60 M to 18.72 M. This indicates that the model of this invention improves the detection capability of densely distributed small-scale ears at maturity while maintaining real-time inference capability.

[0253] Ablation experiments based on the RT-DETR model validated the following:

[0254] To verify the contribution of each improved module to the model performance, in one implementation, the original RT-DETR-1 was used as the baseline model, and Faster-EMA Backbone, DAttention Encoder, GCConvC3 Fusion Neck, P2 Fusion, and the number of object queries was set to 500 in sequence, and ablation experiments were conducted. The role of each improved module and the verification results are shown in Table 4.

[0255] Table 4. Ablation test results of the improved RT-DETR spikelet detection model

[0256]

[0257] As shown in Table 4, compared to the original RT-DETR-1 model, the introduction of the Faster-EMA Backbone reduced the number of model parameters from 28.60 M to 16.90 M, and increased the FPS from 28.20 to 34.87. Simultaneously, Precision, Recall, F1-score, and mAP@0.5 all improved, indicating that the Faster-EMA Backbone can enhance effective feature extraction capabilities while reducing model complexity. Further introduction of the DAttention Encoder increased AP_small@0.5:0.95 to 80.31%, demonstrating its ability to enhance the model's representation of spatial relationships in small-scale ear-head targets and densely occluded regions. Introducing the GCConvC3 Fusion Neck further reduced the number of model parameters to 16.76 M and increased the FPS to 37.68, indicating that this structure can improve feature fusion efficiency and inference efficiency. After introducing P2 Fusion, metrics such as Recall and mAP@0.5:0.95 were improved, indicating that shallow high-resolution features can supplement the edge, texture, and spatial location information that is easily lost during the downsampling process of small-scale spikelets.

[0258] After further increasing the number of object queries to 500, the Precision, Recall, F1-score, mAP@0.5, and AP_small@0.5:0.95 of the model in this invention reached 92.00%, 91.53%, 91.76%, 96.65%, and 80.48%, respectively, demonstrating superior overall detection performance. These results indicate that the model in this invention is not a simple superposition of network modules, but rather a targeted structural design addressing the problems of small size, dense distribution, severe shading, and complex background interference in mature rice panicles, thereby improving the accuracy of panicle detection and the stability of panicle density characterization.

[0259] Comparison with existing methods for counting ears of grain and estimating density:

[0260] In one embodiment, the method of the present invention is compared with manual quadrat counting, UAV remote sensing ear count, handheld device image detection, static phenotypic platform detection, and ear detection method based solely on RGB images. The results are shown in Table 5.

[0261] Table 5 Comparison of the method of the present invention with existing methods for counting and estimating ear counts and densities

[0262]

[0263] As shown in Table 5, the method of this invention differs from traditional methods such as manual counting, UAV remote sensing, handheld device detection, static phenotypic platform detection, and simple two-dimensional image detection. Its key lies in combining the target detection results of the ear heads in the forward-looking RGB image of the combine harvester with the estimation of the actual physical area of ​​the ROI in the depth image, thereby outputting the ear head density per unit area. This output has clear physical meaning and can be directly used as visual input for predicting the combine harvester's feed rate and adjusting operating parameters.

[0264] In summary, this invention acquires RGB and depth images of the harvested area using an RGB-D camera at the front of a combine harvester. It then utilizes an improved RT-DETR model to detect mature ears of grain, counts the number of ears within the Region of Interest (ROI), and calculates the actual physical area of ​​the ROI by combining the depth image, ultimately obtaining the ear density per unit area. Through model comparison, ablation experiments, multi-factor robustness verification, and artificial quadrat density comparison, it is evident that this invention can stably output ear density information for the harvested area, providing a forward-looking perception basis for combine harvester feed prediction, forward speed adjustment, header parameter optimization, and anti-clogging control.

[0265] This invention also provides a system for estimating ear density in the harvesting area of ​​a combine harvester based on airborne RGB-D vision, including an RGB-D camera, an image processing module, an ear detection module, a ROI physical area estimation module, and an ear density calculation module. The RGB-D camera is used to acquire RGB images and depth images of the harvesting area in front of the combine harvester.

[0266] The image processing module is used to register RGB images and depth images and determine the region of interest (ROI).

[0267] The panicle detection module is used to detect mature rice panicle targets within the ROI using the RT-DETR-based panicle detection model, and outputs the panicle detection box and its confidence level.

[0268] The ROI physical area estimation module is used to calculate the actual physical area of ​​the ROI based on the ROI corner depth value, camera intrinsic parameters, and camera extrinsic parameters.

[0269] The ear density calculation module is used to calculate the ear density per unit area based on the number of ears in the ROI and the actual physical area of ​​the ROI, and output the ear density per unit area to the display terminal, storage module or host computer.

[0270] The system can be integrated into the front of a combine harvester to output the ear density of the harvested area in real time.

[0271] The present invention also provides a combine harvester, including the combine harvester ear density estimation system based on airborne RGB-D vision, wherein the combine harvester can acquire the ear density of the area to be harvested ahead during operation and use the ear density as input information for feed rate prediction and operation parameter feedforward adjustment.

[0272] The above embodiments use rice as an example to describe the method of the present invention in detail. It should be noted that the term "ear tip" in the present invention refers to the panicle at the maturity stage of grains, including but not limited to rice and wheat ear tips.

[0273] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0274] The detailed descriptions listed above are merely specific illustrations of feasible embodiments of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the technical concept of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for estimating ear density in the harvesting area of ​​a combine harvester based on airborne RGB-D vision, characterized in that, Includes the following steps: Step S1: Acquire RGB and depth images of the harvested area in front of the combine harvester using an RGB-D camera installed at the front of the combine harvester, and spatially register the RGB and depth images. Step S2: Determine the region of interest (ROI) in the registered RGB image and obtain the corresponding position of the ROI in the RGB image and the depth image; Step S3: Input the RGB image region corresponding to the ROI into the trained RT-DETR-based ear of grain. The detection model detects targets at the mature ear stage and obtains the target detection box and its confidence level. Step S4: Filter the detection frames of ear heads within the ROI according to a preset confidence threshold, and use the number of valid detection frames after filtering as the number of ear heads within the ROI. ; Step S5: Based on the corner pixel coordinates, corresponding depth values, camera intrinsic parameters, and camera extrinsic parameters of the ROI, convert the ROI corner coordinates from pixel coordinates to spatial coordinates, and calculate the actual physical area corresponding to the ROI based on the spatial coordinates. ; Step S6: Based on the number of ears within the ROI and ROI actual physical area Calculate the density of ears per unit area and the ear density As a feedforward sensing parameter output for the combine harvester feed rate.

2. The method for estimating ear density in the harvesting area of ​​a combine harvester according to claim 1, characterized in that, The RT-DETR-based ear detection model includes Faster-EMA Backbone, DAttention Encoder, and P2-guided GCConvC3 Fusion Neck; The Faster-EMA Backbone is used to enhance the feature extraction of shallow textures, edges, and local salient regions of the spikelet against complex backgrounds. The DAttention Encoder is used to enhance long-range dependency modeling of high-level semantic features; The P2-guidedGCConvC3 Fusion Neck is used for shallow detail-guided multi-scale context fusion.

3. The method for estimating ear density in the harvesting area of ​​a combine harvester according to claim 2, characterized in that, The Faster-EMA Backbone is used to process the input feature map. Divide into branches to be convolved along the channel dimension. and retain branches : in, , , Perform a 3×3 convolution on the convolutional branch and preserve the branch. Using identity mapping: , , Will and By concatenating the channels, we obtain partial convolutional output features: ,Will Divided by channel The feature group, denoted as the i-th feature group. The feature groups are Global average pooling is performed along both the horizontal and vertical directions: ,, ,Will and After concatenation, horizontal attention weights are generated by 1×1 convolution and the sigmoid function. and vertical attention weights : ,use and right Recalibration is performed to obtain attention-enhanced features: In this context, ⊙ represents element-wise multiplication. The enhanced features from each group are concatenated and then fused using a 1×1 convolution to obtain the output features of the Faster-EMA Backbone. : The DAttention Encoder is used to determine the high-level semantic features input. Query features are generated through linear mapping. Key features Sum value characteristics : , ,, Spatial sampling offset is generated through offset prediction convolution. : For the first Query location Based on the reference sampling position and the learned offset The deformable sampling positions are obtained: At the deformable sampling location Bilinear interpolation sampling is performed on the key features and value features to obtain and And calculate attention weights: ,in, The feature dimension scaling factor is used to finally obtain the first... Output features at each position: The P2-guided GCConvC3 Fusion Neck is used to perform 1×1 convolution on shallow high-resolution features. Adjust the number of channels: Based on the spatial dimensions of the feature layers to be fused, Perform scale alignment to obtain the scale-aligned shallow guiding features. : ,Will Features fused with target layer Perform channel splicing: ,right Generate spatial context weights : ,based on Perform global context aggregation: global context Channel recalibration weights are generated after 1×1 convolution and nonlinear activation. : ,use right Enhancement: The enhanced features are input into the C3 fusion structure, processed through two branches, concatenated, and then output as fused features via a 1×1 convolution: ,in, represents a non-linear activation function, and ⊙ represents element-wise multiplication.

4. The method for estimating ear density in the harvesting area of ​​a combine harvester according to claim 1, characterized in that, In steps S3 and S4, the ROI image is cropped from the RGB image based on the coordinates of the top left and bottom right corners of the ROI; the ROI image is then resized and padded before being input into the RT-DETR-based ear detection model; the RT-DETR-based ear detection model outputs ear detection boxes and their confidence scores, and filters valid detection boxes according to a preset confidence threshold, using the number of valid detection boxes as the number of ears within the ROI. .

5. The method for estimating ear density in the harvesting area of ​​a combine harvester according to claim 1, characterized in that, In steps S3 and S4, the RGB image containing the ROI is input into the RT-DETR-based ear detection model to obtain the ear target detection box and its confidence level in the entire image; the coordinates of the center point of each detection box are calculated to determine whether the center point of the detection box is located within the ROI region; if the center point of the detection box falls within the ROI and the confidence level of the detection box is greater than the preset confidence threshold, then the detection box is determined to be a valid ear detection box. The number of valid detection frames within the ROI is counted as the number of ear heads within the ROI. .

6. The method for estimating ear density in the harvesting area of ​​a combine harvester according to claim 1, characterized in that, Step S5 includes the following sub-steps: S5.1 Obtain the pixel coordinates of the four corner points of the ROI, including the top left corner point, the top right corner point, the bottom right corner point, and the bottom left corner point; S5.2 Read the depth values ​​corresponding to the four corner points in the registered depth image; when the depth value corresponding to the corner point is greater than 0, the depth value is taken as the valid depth value; when the depth value corresponding to the corner point is less than or equal to 0 or cannot be read, search for pixels with a depth value greater than 0 in the neighborhood window centered on the corner point, and take the median of the valid depth values ​​in the neighborhood window as the alternative depth value of the corner point. If there are still no pixels with a depth value greater than 0 in the neighborhood window, the depth value of the corner point is retained as an invalid value and excluded or marked in the subsequent area estimation. S5.

3. Convert the ROI corner points from pixel coordinates to three-dimensional coordinates in the camera coordinate system based on the camera intrinsic parameters; S5.

4. Based on the camera's extrinsic parameters, transform the three-dimensional coordinates to the world coordinate system or the vehicle coordinate system; S5.5 Calculate the top width, bottom width, left length, and right length of the ROI based on the transformed spatial coordinates of the four corner points, and calculate the actual physical area corresponding to the ROI based on the top width, bottom width, left length, and right length. .

7. The method for estimating ear density in the harvesting area of ​​a combine harvester according to claim 6, characterized in that, In step S5.3, the first The three-dimensional coordinates of each corner point in the camera coordinate system , , Calculate using the following formula: ,in, For the first The pixel coordinates of the corner points For the first Depth values ​​of each corner point and These are the focal length parameters of the camera in the horizontal and vertical directions, respectively. and The coordinates of the camera's principal point; In step S5.4, the first The position of each corner point in the world coordinate system or vehicle coordinate system Calculate using the following formula: ,in, For the first The position of each corner point in the camera coordinate system Let be a rotation matrix. It is a translation vector.

8. The method for estimating ear density in the harvesting area of ​​a combine harvester according to claim 6, characterized in that, The actual physical area corresponding to the ROI Calculate as follows: , ,, ,in, The width of the top edge of the ROI. This represents the width of the bottom edge of the ROI. This represents the actual average width of the ROI. The length of the left side of the ROI. The length of the right side of the ROI. This represents the actual average length of the ROI. The actual physical area corresponding to the ROI, in units of .

9. A system for estimating ear density in the harvesting area of ​​a combine harvester based on airborne RGB-D vision, used to implement the method described in any one of claims 1 to 8, characterized in that, It includes an RGB-D camera, an image processing module, a spikelet detection module, a ROI physical area estimation module, and a spikelet density calculation module; The RGB-D camera is used to acquire RGB and depth images of the area to be harvested in front of the combine harvester; The image processing module is used to register RGB images and depth images and determine the region of interest (ROI). The panicle detection module is used to detect mature rice panicle targets within the ROI using the RT-DETR-based panicle detection model, and outputs the panicle detection box and its confidence level. The ROI physical area estimation module is used to calculate the actual physical area of ​​the ROI based on the ROI corner depth value, camera intrinsic parameters, and camera extrinsic parameters. The ear density calculation module is used to calculate the ear density per unit area based on the number of ears in the ROI and the actual physical area of ​​the ROI, and output the ear density per unit area to the display terminal, storage module or host computer.

10. A combine harvester, characterized in that, Includes the ear density estimation system for the harvesting area of ​​a combine harvester based on airborne RGB-D vision as described in claim 9.

Images