Single-view rgb and imu fusion-based soybean leaf angle prediction method

Abstract

The present application relates to the technical field of crop phenotype measurement, and discloses a soybean leaf inclination angle prediction method based on monocular RGB and IMU fusion. A single-frame soybean RGB image and a camera IMU posture are obtained, and an instance segmentation model is used to obtain a mask of each visible leaf. A zero-sample normal field and a low-rank adaptive normal field are respectively generated by using a latent space diffusion method, and normalized fusion is performed on pixels inside the eroded mask and pixels whose angle difference does not exceed a threshold. Adaptive erosion, outlier pixel removal based on quartile distance and inverse distance weighted aggregation are sequentially performed on each leaf to obtain a leaf normal in a camera coordinate system. A rotation matrix from the camera coordinate system to the world coordinate system is constructed according to the camera IMU posture, the leaf normal is converted to the world coordinate system, and the leaf inclination angle is calculated according to the angle between the leaf normal and the vertical direction. The method does not require a laser radar, a multi-camera array or a leaf IMU in the reasoning stage, and can realize non-contact prediction of the absolute inclination angle of a single leaf.

Description

Technical Field

[0001] This invention relates to the field of crop phenotyping technology, specifically to a method for predicting soybean leaf tilt angle by fusing monocular RGB and IMU. Background Technology

[0002] Leaf tilt angle refers to the angle between the leaf surface and the horizontal plane. In densely planted soybeans, the tilt of the upper leaves affects the transmission of solar radiation into the canopy, further influencing the light exposure of different leaf layers. Therefore, rapidly obtaining single-leaf tilt angle data in the field is of practical significance for plant architecture evaluation, canopy monitoring, and breeding material screening. Existing measurement methods mainly include protractor or inclinometer measurement, attaching inertial measurement units to the leaf surface, hemispherical photography or gap rate inversion, lidar point cloud processing, and multi-view 3D reconstruction.

[0003] Contact measurements require leaf-by-leaf operation, which is inefficient and may disturb leaf attitude. Hemispherical photography and gap rate inversion typically yield canopy averages, making it difficult to determine the tilt angle of a single leaf. While lidar and multi-view reconstruction can obtain 3D geometric information, the equipment and calibration processes are complex. Monocular RGB image acquisition is convenient, but recovering the 3D leaf attitude from a single frame image presents ambiguities regarding depth and normal. Field leaf occlusion, low boundary contrast, illumination variations, and cluttered backgrounds can amplify prediction errors. Therefore, a prediction method is needed that can obtain the absolute tilt angle of a single leaf from a single frame RGB image and camera attitude without relying on lidar, multi-camera arrays, or leaf-attached sensors during the inference stage. Summary of the Invention

[0004] The technical problem to be solved by this invention is: to address the difficulty of directly recovering the three-dimensional pose of leaves from monocular images, the instability of segmentation boundary and self-shadow region normal prediction, and the tendency of pixel-level abnormal normals to propagate to the final tilt angle result, this invention provides a soybean leaf tilt angle prediction method that fuses monocular RGB with camera IMU, and outputs the absolute tilt angle of each visible leaf under field acquisition conditions.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] S1. Acquire a single-frame RGB image of a soybean plant and the camera IMU attitude data corresponding to the RGB image, establish a world coordinate system aligned with the direction of gravity, and determine the camera coordinate system.

[0007] S2. Perform leaf instance segmentation on the RGB image to obtain a binary instance mask corresponding to each visible leaf;

[0008] S3. Input the RGB image into the latent space diffusion normal estimation model to obtain the zero-sample normal prediction and the normal prediction after soybean domain LoRA adaptation. In the internal region retained after instance mask erosion, select pixels whose angular distance between the two normal predictions does not exceed a preset threshold, and perform normalized convex combination on the corresponding normals to obtain a fused dense normal field.

[0009] S4. For each valid instance mask, determine the erosion radius according to the area of ​​the valid instance mask, and perform adaptive morphological erosion on the valid instance mask; perform outlier pixel culling based on interquartile range and inverse distance weighted normalization aggregation on the pixel normals in the fused dense normal field in the preserved area after erosion to obtain the single-leaf unit normal in the camera coordinate system.

[0010] S5. Based on the camera IMU attitude data, construct the rotation matrix from the camera coordinate system to the world coordinate system in the order of ZYX Euler angles, transform the single-blade unit normal to the world coordinate system, and determine the blade tilt angle based on the angle between the transformed blade normal and the vertical axis of the world coordinate system.

[0011] In this invention, the world coordinate system Aligned with the direction of gravity, its The axis points vertically upwards. The plane coincides with the horizontal plane; camera coordinate system Using the camera optical axis The axis, the right direction of the image is The axis and the downward direction of the image are Coordinate conventions for axes.

[0012] blade tilt angle Defined as the angle between the leaf surface and the horizontal plane. Let... The blade unit normal after transformation to the world coordinate system and L2 normalization, If the vertical unit vector is the world coordinate system, then the blade tilt angle is calculated according to the following formula:

[0013]

[0014] The absolute value is used to eliminate the ambiguity of signs caused by the inconsistency between the outer and inner normal orientations of the leaf surface; when the leaf is in a horizontal position... When the blade is in a vertical position, .

[0015] Compared with existing technologies, this invention provides a method for predicting soybean leaf tilt angle by fusing monocular RGB and IMU, which has the following beneficial effects:

[0016] 1. In the prediction stage, this invention only requires a regular monocular RGB camera and an IMU rigidly mounted on the camera. It does not require attaching sensors to each leaf, nor does it rely on lidar or multi-camera synchronous arrays. The acquisition method is suitable for mobile or handheld field operations.

[0017] 2. This invention first performs instance-level segmentation on the blades, and then completes normal estimation and geometric solution within the mask of each blade. The output object is a single visible blade in the image, rather than the average tilt angle of the canopy, which can preserve the pose differences between blades.

[0018] 3. This invention utilizes the synthesized RGB-normal pixel pair to perform LoRA adaptation on the diffusion normal estimation model and retains zero-sample prediction; restricted fusion is performed only inside the mask and at the location where the consistency of the two predictions meets the threshold, reducing the interference of blade edges, self-shadow boundaries and background residue on the normal field.

[0019] 4. This invention employs a continuous processing method that combines adaptive erosion, outlier pixel removal based on interquartile range, and inverse distance-weighted aggregation. The erosion scale varies with the blade mask area, thus accommodating both large and small blades; outlier removal and weighted aggregation further weaken the impact of abnormal pixels on the principal normal of a single blade.

[0020] 5. This invention uses the attitude provided by the camera IMU for the transformation between the camera coordinate system and the gravity-aligned world coordinate system, so that the blade normal obtained under different shooting pitch, roll and yaw conditions can be unified into the world coordinate system, thereby obtaining an absolute tilt angle with clear physical meaning. Attached Figure Description

[0021] Figure 1 This is a flowchart of the process for selecting blade instances using a mask, robust aggregation of single-blade normal direction, and solving for blade tilt angle in this invention. Detailed Implementation

[0022] The following is combined with Figure 1 The present invention will be further described in conjunction with specific embodiments.

[0023] Example 1: Construction of Training and Supervised Data

[0024] This embodiment uses a standard monocular RGB camera to acquire images of soybean plants, and a camera IMU is rigidly mounted on the camera body, denoted as [IMU name missing]. The camera IMU is used to record the roll, pitch, and yaw angles during image acquisition. To obtain the end-to-end evaluation truth value, the blade IMU is attached to the front of the blade under test during the data construction phase, denoted as... The two IMUs share a microcontroller for synchronization. Image filenames are written with a six-digit HHMMSS timestamp and aligned with the sensor logs. IMU readings are averaged within a 1-second window centered on the image acquisition time. The blade IMU is only used to provide the true blade tilt angle during end-to-end evaluation and is not involved in actual prediction.

[0025] The world coordinate system W is aligned with the direction of gravity. The axes are vertically upward; the camera coordinate system C uses OpenCV coordinate conventions. The camera IMU pose is constructed using the ZYX Euler angles to create a rotation matrix from the camera coordinate system to the world coordinate system.

[0026]

[0027] To train the leaf instance segmentation model, images were collected during the seedling stage in a soybean experimental field at the West Campus of China Agricultural University. The collection period was from 10:00 to 11:00 AM on a sunny day. Each plant was photographed from three perspectives: top-down, approximately 45° oblique, and horizontal side view, resulting in 105 RGB field images. A small number of public domain soybean images were also supplemented, and 110 images were retained after cleaning. LabelMe was used to annotate each visible leaf surface with polygon instances, using 15-30 vertices per polygon. The JSON annotations were converted to YOLO normalized mask format using a script, and the images were divided into 80 training images, 20 validation images, and 10 test images. After quality control, 42 sets of image-camera pose-leaf ground truth angle triplets were retained for end-to-end evaluation.

[0028] To achieve pixel-level surface normal supervision, a 3D soybean plant model was imported into 3ds Max, leaf textures were redesigned, HDR ambient lighting and directional sunlight were applied, and the Physical Camera renderer was used to simultaneously output an RGB rendering image and a camera coordinate system unit normal image for each virtual camera pose. Normal components Image channels are encoded using the following formula:

[0029]

[0030] By changing the camera azimuth, pitch, field of view, plant configuration, and lighting parameters, 549 pairs of pixel-aligned RGB-normal data were obtained to form the SoybeanNorm dataset, and a LoRA fine-tuning training subset and a virtual test subset were pre-divided.

[0031] Example 2: Training and Inference of Blade Instance Segmentation Model

[0032] Offline unsharp masking is performed on the training images. Let the original image be... The image is sharpened to g, using a 5×5 Gaussian kernel, with a sharpening factor c of 0.8.

[0033]

[0034] Sharpening does not change the geometric position of the image, so the original polygon annotations are retained, and the training images are expanded from 80 to 160. Online enhancement includes random rotation ±15°, scaling range [0.5, 1.5], cropping 2°, perspective distortion 0.0005, and Mosaic, random erasure, Copy-Paste, Mixup, and HSV perturbations; among them, the adoption probabilities of Mosaic, random erasure, Copy-Paste, and Mixup are 1.0, 0.4, 0.3, and 0.1, respectively; the HSV perturbation parameters are as follows. , and .

[0035] The instance segmentation network uses YOLOv11m-seg, with CBAM attention modules injected via forward hooks at layers 16, 19, and 22 of the neck layer, corresponding to channel widths of 256, 512, and 1024, respectively. CBAM performs channel recalibration and spatial recalibration sequentially, improving the network's response to leaf region and leaf margin features in complex field backgrounds. Training is divided into two phases: the first phase freezes the first 10 layers of the backbone network, training only the detection and segmentation heads for 50 epochs using the AdamW optimizer with an initial learning rate of 0.005 and cosine annealing; the second phase unfreezes the entire network starting from the checkpoint of the first phase, adjusting the initial learning rate to 0.001, training for a maximum of 150 epochs, with an early stopping tolerance of 40 epochs.

[0036] During prediction, the current RGB image is input into the segmentation model, and a set of binary instance masks is obtained with a confidence threshold of 0.2, an IoU threshold of 0.3, and a high-resolution mask mode. Delete masks with fewer than 2000 pixels; calculate the centroid using the mask image moments. and according to the second-level space bond The sorting process makes the leaf numbers in adjacent images more dependent on spatial location than on detection confidence.

[0037] Example 3: Diffusion Normal Estimation and Constrained Fusion

[0038] A monocular RGB image is used as input to the normal estimation module. The diffusion normal estimation model employs a latent space diffusion structure: a frozen VAE encodes the RGB image as latent variables, the UNet denoiser receives the channel stack of the RGB latent variables and the noise latent variables, and zero-sample normal field is obtained using DDIM sampling. Using 549 pairs of SoybeanNorm data from Example 1, LoRA fine-tuning was performed on higher-level modules of UNet to obtain a normal field adapted to the soybean leaf domain. LoRA training employs a smaller learning rate, gradient pruning, and early stopping to reduce the disruption of the original geometric prior by small-scale synthetic data.

[0039] The mask for each blade is first etched to obtain the internal region far from the blade edge. Only within this internal region, and with zero sample normals... With Adaptive Normal The angular distance is not greater than the threshold. Perform Safe-Hybrid fusion at the pixel location:

[0040]

[0041] For pixels that do not meet the regional or consistency conditions, zero-sample normal prediction is used. As the surface normal of the corresponding pixel, the above-mentioned normalized convex combination is not performed. This processing preserves the general normal prior of the zero-sample diffusion model, while introducing soybean domain adaptation results only at locations where the two predictions support each other.

[0042] During inference, the cropped image can be zero-filled and scaled to a fixed resolution. After inference, it can be pasted back onto the original canvas. Enhancements are made when testing with horizontally or vertically flipped copies, and the corresponding normal components are changed synchronously. or The symbols are used to average the results of multiple inferences and perform... Normalization. Within the blade mask, edge weights are constructed based on the Sobel amplitude of the RGB image, followed by bilateral filtering and mild residual sharpening; the area outside the mask is filled with (0,0,1). The metric derived data used for tilt angle calculation undergoes signed reflection of the y-axis component according to the synthetic normal coordinate convention, followed by background-aware cleaning.

[0043] Example 4: Robust Polymerization in a Single Leaf Direction

[0044] For each effective blade mask First, determine the adaptive erosion radius based on the number of mask pixels:

[0045]

[0046] in, For the number of mask pixels, This indicates that the radius is limited to the range of 3-15 pixels. Larger blades use a larger erosion radius to remove more edge contamination, while smaller blades use a smaller radius to preserve internal pixels; if less than 50 pixels remain after erosion, the erosion is canceled.

[0047] Outlier pixel removal based on interquartile range is then performed. Using the component median ñ of the normal to the eroded pixel as a reference, the 1st... Each pixel normal Calculate distance .set up and These are the 25th and 75th quantiles of the distance set, respectively. Only pixels that satisfy the following formula are retained:

[0048]

[0049] The retained pixels are aggregated using an inverse distance-weighted method. Pixels closer to the median normal have a higher weight.

[0050]

[0051] This yields the unit normal representing the overall attitude of the blade in the camera coordinate system. .

[0052] Example 5: Coordinate Transformation and Blade Tilt Angle Output

[0053] Read the camera IMU roll angle corresponding to the current RGB image. Pitch angle and yaw angle Construct the rotation matrix from the camera coordinate system to the world coordinate system according to the ZYX Euler angle order:

[0054]

[0055] The single-leaf unit normal obtained through robust aggregation in the camera coordinate system. Transform to world coordinate system:

[0056]

[0057] And on Perform L2 normalization. To eliminate the sign difference caused by the opposite directions of the outer and inner normals on the leaf surface, take... Vertical unit vector of the world coordinate system The absolute value of the dot product is used to calculate the blade tilt angle using the following formula:

[0058]

[0059] The obtained blade tilt angles range from 0° to 90°, with 0° corresponding to horizontal blades and 90° corresponding to vertical blades. The blade tilt angles corresponding to each instance mask are associated with the spatially sorted blade numbers to form a list of tilt angles for all valid visible blades in the current RGB image.

[0060] Example 6: Effect Verification

[0061] The results were validated using 42 sets of field images, camera pose, and leaf IMU ground truth data. When directly averaging the normals of pixels inside the mask, the mean absolute error (MAO) was 6.71° and the root mean square error (RMSE) was 12.86°. After adding fixed erosion, the MAO and RMS were 5.32° and 9.47°, respectively. After adaptive erosion, the MAO and RMS were 4.53° and 7.82°, respectively. After adding outlier pixel removal based on interquartile range, the MAO and RMS were 3.47° and 6.15°, respectively. Further inverse distance-weighted aggregation resulted in a complete processing pipeline with an MAO of 2.88° and an RMS of 4.88°. These results illustrate the role of each processing step in suppressing edge contamination and anomalous normals.

Claims

1. A method for predicting soybean leaf tilt angle by fusing monocular RGB and IMU, characterized in that, include: Acquire a single-frame RGB image of a soybean plant and the camera IMU attitude data corresponding to the single-frame RGB image, establish a world coordinate system aligned with the direction of gravity, and determine the camera coordinate system; The single-frame RGB image is segmented into leaf instances to obtain instance masks corresponding to each visible leaf; The single-frame RGB image is input into the latent space diffusion normal estimation model to obtain the zero-sample normal prediction and the normal prediction after low-rank adaptation of the soybean leaf domain. In the internal region retained after the instance mask is eroded, pixels whose angular distance between the zero-sample normal prediction and the normal prediction after low-rank adaptation of the soybean leaf domain does not exceed a preset angular difference threshold are selected, and the two normal predictions at the selected pixels are normalized convexly combined to obtain a fused dense normal field. For each effective instance mask, the erosion radius is determined according to the area of ​​the effective instance mask, and adaptive morphological erosion is performed on the effective instance mask. The pixel normals in the fused dense normal field within the eroded area are subjected to outlier pixel culling based on interquartile range and inverse distance weighted normalization aggregation to obtain the single-leaf unit normal in the camera coordinate system. Based on the camera IMU attitude data, a rotation matrix from the camera coordinate system to the world coordinate system is constructed in ZYX Euler angle order to transform the single-leaf unit normal to the world coordinate system. The tilt angle of the corresponding blade is determined based on the angle between the transformed single-leaf unit normal and the vertical direction of the world coordinate system.

2. The method for predicting soybean leaf tilt angle by fusion of monocular RGB and IMU according to claim 1, characterized in that, The leaf instance segmentation uses the YOLOv11m-seg network, and CBAM attention modules, which are sequentially connected by channel attention and spatial attention, are injected into the three neck feature layers of the YOLOv11m-seg network. The segmentation training images were sharpened using Unsharp Masking before training.

3. The method for predicting soybean leaf tilt angle by fusion of monocular RGB and IMU according to claim 2, characterized in that, The three neck feature layers are layers 16, 19, and 22, with corresponding channel widths of 256, 512, and 1024, respectively. The YOLOv11m-seg network adopts a two-stage transfer learning approach. In the first stage, the first 10 layers of the backbone network are frozen and the detection head and segmentation head are trained. In the second stage, the entire network is unfrozen and fine-tuned.

4. The method for predicting soybean leaf tilt angle by fusion of monocular RGB and IMU according to claim 1, characterized in that, The soybean leaf domain low-rank adaptation is trained using pixel-aligned synthetic RGB-normal data. The synthetic RGB-normal data is physically rendered using a 3D model of a soybean plant under different camera orientations, pitches, fields of view, plant configurations, and lighting parameters, simultaneously obtaining RGB rendering images and camera coordinate system unit normal images. The low-rank adaptation only applies to the UNet high-level modules of the latent space diffusion normal estimation model.

5. The method for predicting soybean leaf tilt angle by fusion of monocular RGB and IMU according to claim 4, characterized in that, Let the zero-sample normal prediction be The normal direction prediction after low-rank adaptation of soybean leaf domain is: The normalized convex combination is according to In progress, among which , This is a numerically stable term; it applies only if the pixel is located within the internal region after the instance mask erosion and and When the angular distance between them is not greater than the preset angular difference threshold, use The fused surface normal is used as the pixel's normal; for pixels that do not meet the internal region condition or angular distance condition, the zero-sample normal is used for prediction. This serves as the surface normal of the pixel.

6. The method for predicting soybean leaf tilt angle by fusion of monocular RGB and IMU according to claim 5, characterized in that, In the process of generating the zero-sample normal prediction and the normal prediction after low-rank adaptation of the soybean leaf domain, the cropped image is zero-filled and scaled to a fixed inference resolution, and the inference result is pasted back to the original canvas. When testing horizontal or vertical flipping, the sign of the corresponding normal component of the enhanced image is changed synchronously, and the average of multiple inference results is calculated and L2 normalized. Edge-aware filtering is performed within the instance mask based on the Sobel magnitude of the RGB image, and the background normal (0,0,1) is filled outside the instance mask.

7. The method for predicting soybean leaf tilt angle by fusion of monocular RGB and IMU according to claim 1, characterized in that, The instance masks are filtered according to the confidence threshold and the cross-union ratio threshold. Instance masks with fewer than the preset area threshold are deleted. The instance masks are sorted according to the centroid coordinates of each instance mask by the secondary spatial key to determine the leaf number.

8. The method for predicting soybean leaf tilt angle by fusion of monocular RGB and IMU according to claim 1, characterized in that, The corrosion radius of the adaptive morphological corrosion satisfy: in For the first The number of pixels in an instance mask. ( , ) indicates that The adaptive morphological erosion is cancelled when the number of remaining pixels after erosion is less than 50, and the number is limited to 3 to 15.

9. The method for predicting soybean leaf tilt angle by fusion of monocular RGB and IMU according to claim 8, characterized in that, The outlier pixel removal based on interquartile range includes: calculating the distance from the normal of each pixel to the median normal of each component. , retain satisfaction Pixels, of which and These are the 25th and 75th quantiles of the distance, respectively; the pixel weights in the inverse distance weighted normalization aggregation are... ,in, For numerically stable terms, and for L2 normalization is performed to obtain the unit normal of the single leaf.

10. The method for predicting soybean leaf tilt angle by fusion of monocular RGB and IMU according to claim 1, characterized in that, The rotation matrix from the camera coordinate system to the world coordinate system is: in , and These are the roll, pitch, and yaw angles output by the camera's IMU; the single-leaf unit normal in the camera coordinate system. Convert to a single-leaf unit normal in world coordinates: And calculate the blade tilt angle according to the following formula: in, The blade tilt angle is the vertical unit vector in the world coordinate system. The value range is 0° to 90°.

Images