A monitoring system for the degree of mulberry tree lodging

Abstract

This invention discloses a mulberry tree lodging monitoring system. It includes: an attitude assessment unit, a viewing angle initialization unit, a trajectory planning unit, an image acquisition and panoramic synthesis unit, a lodging level judgment unit, and a data processing module. The attitude assessment unit, viewing angle initialization unit, trajectory planning unit, image acquisition and panoramic synthesis unit, and lodging level judgment unit are all connected to the data processing module. During the flight path and observation position generation stage, the system simultaneously measures the occlusion ratio, temperature bias caused by the background, and thermal image coverage redundancy, prioritizing observation positions that meet the thermal threshold conditions. This mechanism can significantly reduce temperature reading deviations caused by self-shading, foliage obstruction, or interference from neighboring trees during surround acquisition, improve the visibility of abnormal hot spots and thermal asymmetry, and reduce missed and false detections from the source.

Description

Technical Field

[0001] This invention relates to the field of forest tree monitoring technology, specifically a mulberry tree lodging monitoring system. Background Technology

[0002] Currently, orchard lodging identification relies heavily on manual inspections or single-sensor methods (such as visible light cameras or laser point clouds). Manual methods are labor-intensive, inconsistent, and prone to missed detections in mulberry orchards with dense foliage, undulating ground, or irregular tree spacing. While visible light-based aerial photography is low-cost, it is sensitive to lighting conditions, shadows, and background complexity, making it difficult to reliably represent tree posture and structural integrity. Methods relying solely on 3D point clouds can obtain tree geometry, but in situations with dense foliage and severe self-occlusion, the point clouds are incomplete, contain many holes, and there is uncertainty in tilt and displacement estimations.

[0003] Thermal infrared imaging can reflect the temperature distribution on the tree surface, which is valuable for identifying abnormalities such as damaged branches, obstructed root systems, or damage caused by wind and snow. However, existing methods generally suffer from the following problems:

[0004] The selection of observation sites lacks thermal visibility constraints: most flight path planning only considers geometric visibility and obstacle avoidance, ignoring the combined effects of thermal radiation field of view, self-shading and background emission, resulting in insufficient thermal image coverage or significant temperature bias.

[0005] Thermal panoramic imaging is prone to omissions and inconsistencies: when capturing images around the tree, such as on the shaded side or under the canopy, there are often gaps or misalignments in the stitching, and there is a lack of closed-loop quality control and re-capture mechanisms.

[0006] Loose multi-domain (thermal infrared / visible light / 3D) registration: Errors caused by viewing angle, distortion and time difference between different sensors are not systematically corrected, and the estimates of the fall angle and lateral displacement are prone to drift.

[0007] Features and interpretation are not coupled in a closed loop: information such as geometric tilt, thermal asymmetry and central displacement are often processed separately, and changes in training data quality and imaging conditions are not dynamically fed back to the acquisition and registration stages, affecting the stability and verifiability of the classification. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a mulberry tree lodging degree monitoring system, including: a posture assessment unit, a viewing angle initial adjustment unit, a trajectory planning unit, an image acquisition and panoramic synthesis unit, a lodging level judgment unit, and a data processing module;

[0009] The posture evaluation unit, initial viewpoint adjustment unit, trajectory planning unit, image acquisition and panoramic synthesis unit, and fall level judgment unit are all connected to the data processing module.

[0010] The attitude assessment unit is used to conduct a circumferential aerial survey of the target mulberry tree by an imaging device carried by an airborne remote sensing carrier, generate a three-dimensional morphological model of the target plant, and determine whether the plant's attitude is upright or tilted based on the three-dimensional morphological model, as well as determine whether the working airspace conditions allow for circumferential observation or whether there are circumferential obstacles.

[0011] The aforementioned perspective adjustment unit is used to trigger trunk angle analysis based on the posture evaluation results, determine the tilt amount based on the trunk spatial information in the three-dimensional morphology model, and adjust the viewing angle of the infrared imager and the visible light camera according to the tilt amount.

[0012] The trajectory planning unit is used to pre-set several observation positions around the outer perimeter of the tree trunk, select the benchmark observation positions based on the working space conditions, and generate supplementary observation positions according to the viewing angle interval limit and the distance limit from the trunk to form a surrounding observation trajectory.

[0013] An image acquisition and panoramic synthesis unit is used to acquire infrared images along the trajectory and stitch them together into a panoramic thermal image.

[0014] The lodging level judgment unit is used to determine the abnormal signs and degree of lodging of mulberry trees based on the differences in surface heat distribution and morphological information in the panoramic thermal image, and output the corresponding location range.

[0015] Furthermore, the image acquisition and panoramic synthesis unit also includes a panoramic enhancement module. The panoramic enhancement module performs pixel confidence assessment on the stitched panoramic thermal image to identify missing areas, obtains boundary coordinates through contour extraction, and extracts effective pixels within a set range outside the missing areas. When the number of effective pixels is insufficient, the sampling radius is expanded until a threshold is met. Based on the geometric structure of the tree trunk and crown, geostatistical interpolation is used to infer the infrared values ​​of the effective pixels to enhance the missing areas.

[0016] Furthermore, the trajectory determination unit includes an observation position improvement module, which is used to set up at least one supplementary observation position between any two reference observation positions, such that the circumferential angle between the supplementary observation position and the adjacent reference position is not greater than a preset upper limit, and the straight-line distance from the supplementary observation position to the outer surface of the trunk is within the allowable range; a polar coordinate reference frame is established around the central axis of the trunk, and the position of the supplementary observation position is determined based on the parameters of the reference observation position under the reference frame and the trunk size information.

[0017] Furthermore, it also includes an obstacle avoidance verification sub-process: a three-dimensional detection volume is constructed at each supplementary observation position, and it is determined whether the detection volume and the obstacle have spatial overlap, and the occlusion ratio of the infrared image obtained at that position and the obstacle is simulated; when there is spatial overlap and / or the occlusion ratio reaches the threshold, the observation position is determined to be invalid and the positioning calculation sub-process is re-executed.

[0018] Furthermore, it also includes a viewpoint adjustment unit: used to identify the pixel position of the same reference mark in images obtained from different observation positions, convert the pixel position into physical coordinates of the image through imaging parameters, retrieve the theoretical position of the reference mark in the three-dimensional morphological model, compare the two to obtain the angle correction amount, so as to correct the viewing angle of the imaging device.

[0019] Furthermore, the posture evaluation unit includes a trunk tilt estimation scheme: extracting the spatial information of the two ends of the trunk's main axis from the three-dimensional morphological model, and using the angle between the connecting line and the ground reference plane as the basis for the tilt amount.

[0020] Furthermore, the posture evaluation unit also includes a set of tree trunk boundary points extracted from the three-dimensional morphological model for straight line fitting, and determines whether the plant is in an upright or tilted state based on the consistency between the fitted straight line and the vertical reference line passing through the boundary points.

[0021] Furthermore, the attitude assessment unit also includes a work airspace assessment module, which is used to fit the target area based on the plant morphology point set, segment the spatial data of surrounding obstacles, calculate the minimum clearance and compare it with safety rules, and output information on whether it can be observed around or whether there are surrounding obstacles.

[0022] The beneficial effects of this invention are: during the route and observation position generation stage, it simultaneously measures the occlusion ratio, temperature bias caused by the background, and thermal image coverage redundancy, prioritizing the selection of observation positions that meet the thermal threshold conditions. This mechanism can significantly reduce temperature reading deviations caused by self-shading, foliage occlusion, or interference from neighboring plants during surround acquisition, improve the visibility of abnormal hot spots and thermal asymmetry, and reduce missed detections and false detections from the source.

[0023] The stitched results are improved by identifying and interpolating missing regions, and the cross-validation error and missing percentage are used as quality thresholds to trigger orderly supplementary acquisition and verification. This closed loop ensures that the panoramic thermal imagery reaches a stable threshold in terms of coverage integrity and geometric consistency, preventing low-quality images from flowing into subsequent grading models and improving the overall reliability of interpretation.

[0024] Using the consistency of multi-domain reference markers as the registration convergence condition, fine-tuning ends only when all three domains simultaneously meet the acceptance threshold; otherwise, the framing pose and extrinsic parameters are automatically reverted. This strategy effectively suppresses drift caused by cross-modal and cross-time, significantly improving the stability and repeatability of geometric quantities such as tilt angle and central axis lateral displacement. Attached Figure Description

[0025] Figure 1 A schematic diagram illustrating the principle of a mulberry tree lodging monitoring system; Detailed Implementation

[0026] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0027] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0028] like Figure 1 As shown, a mulberry tree lodging degree monitoring system includes: a posture assessment unit, a perspective initialization unit, a trajectory planning unit, an image acquisition and panoramic synthesis unit, a lodging level judgment unit, and a data processing module;

[0029] The posture evaluation unit, initial viewpoint adjustment unit, trajectory planning unit, image acquisition and panoramic synthesis unit, and fall level judgment unit are all connected to the data processing module.

[0030] The attitude assessment unit is used to conduct a circumferential aerial survey of the target mulberry tree by an imaging device carried by an airborne remote sensing carrier, generate a three-dimensional morphological model of the target plant, and determine whether the plant's attitude is upright or tilted based on the three-dimensional morphological model, as well as determine whether the working airspace conditions allow for circumferential observation or whether there are circumferential obstacles.

[0031] The aforementioned perspective adjustment unit is used to trigger trunk angle analysis based on the posture evaluation results, determine the tilt amount based on the trunk spatial information in the three-dimensional morphology model, and adjust the viewing angle of the infrared imager and the visible light camera according to the tilt amount.

[0032] The trajectory planning unit is used to pre-set several observation positions around the outer perimeter of the tree trunk, select the benchmark observation positions based on the working space conditions, and generate supplementary observation positions according to the viewing angle interval limit and the distance limit from the trunk to form a surrounding observation trajectory.

[0033] An image acquisition and panoramic synthesis unit is used to acquire infrared images along the trajectory and stitch them together into a panoramic thermal image.

[0034] The lodging level judgment unit is used to determine the abnormal signs and degree of lodging of mulberry trees based on the differences in surface heat distribution and morphological information in the panoramic thermal image, and output the corresponding location range.

[0035] The target mulberry tree is imaged from multiple angles by an aerial vehicle, generating a three-dimensional morphological model of the target mulberry tree, and determining whether there are any surrounding observation obstacles in the operational airspace;

[0036] Candidate observation positions are generated around the main axis of the target mulberry tree. A thermally visible field of view (TLP) detector is established for each candidate observation position. The TLP detector is determined based on the imager's field of view, the expected measurement distance, and the spatial relationship between the target and scene obstacles. It is used to estimate the occlusion ratio of the infrared radiation line of sight and the temperature bias caused by the background. The observation position is accepted only when the occlusion ratio is not higher than a first occlusion threshold and the temperature bias is not higher than a first temperature bias threshold, and a surrounding observation trajectory is planned accordingly. Infrared and visible light images are acquired simultaneously along the observation trajectory and then stitched together to obtain a panoramic thermal image. (The panoramic thermal image is then subjected to missing region identification and filling: missing regions are calibrated based on pixel confidence, and supplementary observation positions are generated along the normal direction of the missing boundary. Geostatistical interpolation is used to fill in the missing regions, and the panoramic quality is judged based on the cross-validation error and the missing proportion. When the cross-validation error is higher than a first error threshold or the missing proportion is higher than a first proportion threshold, a re-sampling process is triggered, and the supplementary observation positions are re-sampled first until the quality requirements are met.)

[0037] Joint registration and viewpoint refinement are performed on infrared images, visible light images and three-dimensional morphological models: based on the positional error of the same reference mark in the three, the registration is confirmed to be effective only when the corresponding errors in the infrared domain, visible domain and three-dimensional domain are not higher than their respective preset acceptance thresholds; otherwise, the framing posture is adjusted and the refinement is repeated.

[0038] Based on the three-dimensional morphological model and the panoramic thermal image, fusion features for lodging determination are extracted, including at least: geometric tilt degree, thermal asymmetry along the tree circumference and lateral displacement features of the trunk axis.

[0039] The fused features are input into the lodging grading model trained with historical orchard data, and the abnormal areas and lodging levels are output. When the level reaches the alarm threshold and the grading reliability is not lower than the second reliability threshold, the location marking and alarm are executed.

[0040] The thermal visibility detector simultaneously considers the combined effects of tree self-shading and surrounding obstacle occlusion, and calculates the thermal image coverage redundancy for candidate observation positions; the observation position is only accepted when the thermal image coverage redundancy is not lower than the first coverage threshold.

[0041] Candidate observation positions are generated to satisfy the joint constraints of the viewing angle interval and the distance from the interference. The parameter range of the joint constraints is automatically adjusted according to the working radius and the desired spatial resolution.

[0042] The supplementary acquisition process schedules supplementary observation positions in descending order of missing percentage and descending order of boundary complexity, and automatically verifies the panoramic quality indicators after the supplementary acquisition is completed.

[0043] Joint registration is based on the consistency of the multi-domain reference markers. Fine-tuning ends only when the infrared domain, visible domain, and three-dimensional domain all meet the acceptance threshold. If they do not meet the threshold, closed-loop corrections are performed on the camera pitch, roll, and yaw angles respectively.

[0044] The thermal asymmetry is obtained by dividing the periphery of the tree into multiple sectors, calculating the average surface temperature of each sector, and comparing the differences between opposing sectors.

[0045] The lodging classification model employs an online sample screening strategy during operation, adding only data samples that pass the consistency acceptance and whose classification residuals are not higher than the second error threshold to the incremental learning set to update the model parameters.

[0046] When the lodging level reaches the alarm threshold, the abnormal area is marked with a polygon on the orchard base map, and the observation trajectory, panoramic quality index and grading reliability are archived together for review.

[0047] Example 1: Single-tree surround monitoring of a leveled orchard plot

[0048] Scene: A mulberry orchard on flat land, with a plant spacing of approximately 3.0 m × 1.2 m. The target is a single mature mulberry tree.

[0049] Platform and sensors: Quadcopter UAV, three-axis gimbal; 640×512 infrared camera with a lens equivalent of 13 mm; 20 MP global shutter visible light camera; airborne RTK; onboard edge computing unit.

[0050] Ground-based: Handheld control terminal (tablet), including task planning and quality monitoring interface; back-end server for data archiving and model management.

[0051] The system performs circumferential scanning and short stops within a safety zone 4–6 m from the outer edge of the tree trunk to generate a three-dimensional morphological model in real time; the system allows 360° circumference based on the clearance distance.

[0052] Eighteen candidate observation sites are initially generated circumferentially (two layers high, nine of which are equiangularly distributed in each layer), and a thermally visible detection volume is established for each observation site.

[0053] Occlusion threshold: ≤ 15%; Temperature offset threshold (background / self-shading combined): ≤ 0.8 ℃; Thermal image coverage redundancy: ≥ 25%. Observation positions that do not meet the conditions are automatically removed, and finally 12 observation positions are accepted to form a surround trajectory; the viewing angle interval is controlled at 20°–40°, and the distance from the dryness is maintained at 3.5–5.0 m.

[0054] The drone stops capturing images sequentially along its trajectory, stabilizing each point for 2 seconds to suppress gimbal jitter; infrared and visible light images are acquired simultaneously to complete the initial panoramic thermal image stitching. The system then performs missing region identification and interpolation to improve the stitched results: cross-validation RMSE threshold: ≤ 0.6 ℃; missing percentage threshold: ≤ 3%.

[0055] If the threshold is not met, 2–3 supplementary observation positions will be automatically generated along the direction of the missing boundary normal and a supplementary sampling command will be issued. After completion, the quality indicators will be checked until they meet the standards.

[0056] Using high-contrast markers (or natural feature points) bound to the tree trunk as a reference, simultaneous acceptance testing is conducted in the infrared, visible, and 3D domains: Infrared domain position error: ≤ 6 pixels; Visible domain position error: ≤ 2 pixels; 3D domain geometric error: ≤ 15 mm. If any domain fails to meet the standard, gimbal pitch / heading fine-tuning and short-range reshooting are triggered until the consistency of the three domains meets the threshold.

[0057] The tilt of the trunk's main axis and the lateral displacement curve of the central axis are obtained from the 3D model; the panoramic thermal image is divided into several sectors in the circumferential direction, and the thermal asymmetry between the sectors is calculated; the three types of features are input into the trained lodging classification model to give the moderate judgment and the polygon of the abnormal area, and the ground alarm and map annotation are triggered when the reliability is ≥ 0.85.

[0058] Example 2: Partial Encirclement and Multi-Height Replenishment Harvesting in Sloping Orchards with Netting

[0059] Scene: A mulberry orchard on a sloping hillside with north-south contour lines, with bird nets and guy wires between the rows, and irrigation pipes and pillars providing some cover in some areas.

[0060] First, a short arc is circled to complete the rapid modeling; the system determines that there is a clear net and string line in the southeast quadrant, and a complete 360° circle is not feasible, only an open arc segment of 220° from northwest to northeast is allowed to fly.

[0061] Sixteen candidate observation points are generated in the flyable arc segment and deployed at two different heights (e.g., the upper edge of the tree canopy and near the middle of the trunk). A thermally visible detection body is established for each point, using the following thresholds: occlusion ratio: ≤ 12%; temperature offset: ≤ 0.7 ℃; thermal image coverage redundancy: ≥ 30%.

[0062] Simultaneously, a 3D obstacle avoidance check was introduced: if a site spatially overlaps with a mesh cover, guy wire, or pillar, or if the simulated occlusion ratio exceeds a threshold, it is considered an invalid site. Eleven sites were retained after screening.

[0063] After the initial assembly, the system determined that the missing percentage was 6.5% and the RMSE was 0.72℃, which did not meet the threshold (≤ 5%, ≤ 0.6℃). Four supplementary observation sites were automatically generated along the normal direction of the missing boundary, one of which was eliminated because it overlapped with the pull line. The remaining three supplementary sites were sampled again, and two additional scouting sites were added at a lower height to observe the shadow area under the canopy. After review, the missing percentage decreased to 2.1% and the RMSE decreased to 0.48℃, passing the quality threshold.

[0064] Due to the change in viewing angle and uneven lighting caused by the slope, the initial registration error was 8 pixels in the infrared domain, 2 pixels in the visible domain, and 17 mm in the three-dimensional domain. The system automatically issued a retake command for gimbal pitch fine adjustment and lateral displacement of 0.4 m, and the second registration reached the threshold (infrared ≤ 6 pixels, visible ≤ 2 pixels, three-dimensional ≤ 15 mm).

[0065] The 3D model reveals clear characteristics of principal axis tilt and central axis bow-shaped displacement; circumferential thermal asymmetry shows significant differences in the northeast-southwest opposing sectors. The grading model outputs severe lodging and provides anomaly polygons from the crown margin to one side of the principal axis; the grading reliability is 0.91, meeting the reporting criteria; because this sample meets the three-domain consistency acceptance and has low classification residuals, the system includes it in the incremental learning set to adapt to the domain bias of the slope and netted environment.

Claims

1. A monitoring system for the degree of lodging of mulberry trees, characterized in that, include: The unit includes a posture assessment unit, a preliminary viewpoint adjustment unit, a trajectory planning unit, an image acquisition and panoramic synthesis unit, a lodging level determination unit, and a data processing module. The posture evaluation unit, the initial viewpoint adjustment unit, the trajectory planning unit, the image acquisition and panoramic synthesis unit, and the fall level judgment unit are all connected to the data processing module. The attitude assessment unit is used to conduct a circumferential aerial survey of the target mulberry tree by an imaging device carried by an airborne remote sensing carrier, generate a three-dimensional morphological model of the target plant, and determine whether the plant's attitude is upright or tilted based on the three-dimensional morphological model, as well as determine whether the working airspace conditions allow for circumferential observation or whether there are circumferential obstacles. The aforementioned perspective adjustment unit is used to trigger trunk angle analysis based on the posture evaluation results, determine the tilt amount based on the trunk spatial information in the three-dimensional morphology model, and adjust the viewing angle of the infrared imager and the visible light camera according to the tilt amount. The trajectory planning unit is used to pre-set several observation positions around the outer perimeter of the tree trunk, select the benchmark observation positions based on the working space conditions, and generate supplementary observation positions according to the viewing angle interval limit and the distance limit from the trunk to form a surrounding observation trajectory. The image acquisition and panoramic synthesis unit is used to simultaneously acquire infrared images and visible light images along the observation trajectory, and perform panoramic stitching to obtain a panoramic thermal image. The lodging level judgment unit is used to determine the abnormal signs and degree of lodging of mulberry trees based on the differences in surface heat distribution and morphological information in the panoramic thermal image, and output the corresponding location range.

2. The mulberry tree lodging monitoring system according to claim 1, characterized in that, The image acquisition and panoramic synthesis unit also includes a panoramic enhancement module. The panoramic enhancement module performs pixel confidence assessment on the stitched panoramic thermal image to identify missing areas, obtains boundary coordinates through contour extraction, and extracts effective pixels within a set range outside the missing areas. When the number of effective pixels is insufficient, the sampling radius is expanded until a threshold is met. Based on the geometric structure of the tree trunk and crown, the infrared values ​​of the effective pixels are inferred using geostatistical interpolation to enhance the missing areas.

3. The mulberry tree lodging monitoring system according to claim 1, characterized in that, The trajectory determination unit includes an observation position improvement module, which is used to set up at least one supplementary observation position between any two reference observation positions, such that the circumferential angle between the supplementary observation position and the adjacent reference position is not greater than a preset upper limit, and the straight-line distance from the supplementary observation position to the outer surface of the trunk is within the allowable range; a polar coordinate reference frame is established around the central axis of the trunk, and the position of the supplementary observation position is determined based on the parameters of the reference observation position under the reference frame and the trunk size information.

4. The mulberry tree lodging monitoring system according to claim 3, characterized in that, It also includes an obstacle avoidance verification sub-process: a three-dimensional detection volume is constructed at each supplementary observation position, and it is determined whether the detection volume and the obstacle have spatial overlap, and the occlusion ratio of the infrared image obtained at that position and the obstacle is simulated; when there is spatial overlap and / or the occlusion ratio reaches the threshold, the observation position is determined to be invalid and the positioning calculation sub-process is re-executed.

5. The mulberry tree lodging monitoring system according to claim 1, characterized in that, It also includes a viewpoint adjustment unit: used to identify the pixel position of the same reference mark in images obtained from different observation positions, convert the pixel position into physical coordinates of the image through imaging parameters, retrieve the theoretical position of the reference mark in the three-dimensional morphological model, compare the two to obtain the angle correction amount, so as to correct the viewing angle of the imaging device.

6. The mulberry tree lodging monitoring system according to claim 1, characterized in that, The attitude assessment unit includes a trunk tilt estimation scheme: extracting the spatial information of the two ends of the trunk main axis in the three-dimensional morphological model, and using the angle between the line connecting the two ends of the trunk main axis and the ground reference plane as the basis for the tilt amount.

7. A mulberry tree lodging monitoring system according to claim 6, characterized in that, The posture evaluation unit also includes a set of tree trunk boundary points extracted from the three-dimensional morphological model for straight line fitting. Based on the consistency between the fitted straight line and the vertical reference line passing through the boundary points, the plant is determined to be in an upright or tilted state.

8. A mulberry tree lodging monitoring system according to claim 1, characterized in that, The attitude assessment unit also includes a work airspace assessment module, which is used to fit the target area based on the plant morphology point set, segment the spatial data of surrounding obstacles, calculate the minimum clearance and compare it with the safety rules, and output information on whether it can be observed around or whether there are surrounding obstacles.

Images