Early intelligent inspection method for pine wood nematode disease based on land-air multi-modal fusion

Abstract

The application discloses a kind of pine wood nematode disease early intelligent inspection methods based on land-air multi-modal fusion, comprising: S1, unmanned aerial vehicle carries hyperspectral spectrometer and obtains spectral data, carries out suspected disease pixel preliminary screening identification and generates suspected target list;S2, generate the optimal inspection path of ground unmanned vehicle based on suspected target list;S3, ground unmanned vehicle carries acquisition equipment and collects suspected target full-band multi-angle spectrum, three-dimensional point cloud and stereo vision data;S4, the data collected in step S3 is input into spectrum-structure-vision three-modal fusion network, with spectral features as query vector, the structure and visual features are weighted fusion, and the final diagnosis result is output;S5, generate hierarchical disposal report.The efficiency and accuracy of the two-stage progressive detection mechanism of preliminary screening by unmanned aerial vehicle and precise screening by ground unmanned vehicle are improved, the detection period is advanced by 10-20 days through complementary verification from two dimensions of internal cause and external cause through modal complementary fusion, and the diagnosis accuracy is improved through cross-modal fusion reasoning.

Description

Technical Field

[0001] This invention relates to the field of early intelligent inspection technology for pine wilt disease, specifically to an early intelligent inspection method for pine wilt disease based on land-air multimodal fusion. Background Technology

[0002] Pine wilt disease, often referred to as the cancer of pine trees, is the number one forestry quarantine disease in my country. The pathogenic nematode is transmitted by the pine sawyer beetle, and can kill pine trees within 40 to 60 days after invading them, posing a serious threat to the carbon sequestration function and biodiversity of forest ecosystems.

[0003] The key to controlling pine wilt disease lies in early detection. In the early stages of infection, approximately weeks 1-3, pine needles remain green on the outside, making any abnormalities difficult to detect with the naked eye; however, physiological and biochemical changes are already occurring inside the needles, such as impaired water transport and degradation of photosynthetic pigments. When the needles show a visible yellowish-brown discoloration, the disease has entered the middle to late stages, and infected pine trees have become oviposition sites for the pine sawyer beetle, drastically increasing the risk of disease spread. Therefore, capturing changes in the internal biochemical components of needles using spectral methods before the external discoloration is observed is the core approach to achieving early warning.

[0004] Currently, the monitoring methods for pine wilt disease in forest areas are mainly divided into the following categories:

[0005] Manual inspection: Relies on forest rangers to visually interpret information on foot or by vehicle. The daily coverage area is limited, and it is severely constrained by terrain, weather and other conditions. It also relies on subjective experience, resulting in a high rate of missed inspections. Furthermore, it has poor accessibility and is highly dangerous in complex mountainous forest areas.

[0006] Satellite remote sensing monitoring: using multispectral satellites to conduct large-scale forest health assessments. However, satellite imagery typically has a spatial resolution of 10-30 meters, making it impossible to distinguish spectral differences at the individual tree canopy level; its limited spectral resolution makes it difficult to capture fine spectral changes in early biochemical components of needles; and it is also limited by cloud cover and revisit cycles of approximately 5-16 days, making real-time high-frequency monitoring impossible.

[0007] Single-UAV multispectral inspection: using a drone equipped with a visible light or multispectral camera to image the canopy. However, visible light can only reflect changes in canopy color, at which point the disease has already entered the middle to late stages; conventional multispectral methods have a limited number of bands and the band settings are generalized, without optimization for the early spectral characteristics of pine wilt disease, resulting in low early detection accuracy, alarm lead time is usually less than 10 days, and the false alarm rate is high.

[0008] Ground-based hyperspectral static measurement. Existing plant stress diagnosis methods based on ground-based hyperspectral measurement are mostly static single-plant measurement scenarios, suitable for farmland or experimental plots, but not suitable for large-area mobile inspection scenarios in forest areas, and the daily coverage area is extremely limited.

[0009] Therefore, the main drawback of the current technology is:

[0010] 1. Insufficient early detection capability: Existing methods lack modeling and detection techniques for the specific spectral characteristics of early-stage pine wilt disease. Current spectral analyses mostly use general vegetation indices and have not established a disease-specific spectral index system targeting the specific physiological and biochemical changes caused by pine wilt infection, such as impaired water transport, accelerated chlorophyll degradation, and accumulation of defensive secondary metabolites. This makes effective detection difficult before visible symptoms appear.

[0011] 2. Platform efficiency and accuracy are mutually exclusive. While drone inspections are highly efficient, their accuracy is limited, with low signal-to-noise ratios and insufficient spatial resolution to distinguish details in the canopy of individual plants. Ground-based equipment offers high accuracy but low efficiency, with limited daily coverage. Existing technologies are all based on a single platform and lack a two-tiered, collaborative detection mechanism that combines wide-area initial screening at high altitudes with detailed diagnosis at ground levels. Consequently, they cannot simultaneously achieve both the efficiency of large-area coverage and the diagnostic accuracy at the individual plant level.

[0012] 3. Diagnostic information is limited in scope; current solutions rely solely on single-modal information, either pure spectral or pure image data for diagnosis. Hyperspectral imaging reflects intrinsic changes in the biochemical components within needles, while LiDAR 3D point clouds reflect extrinsic changes in the external morphological structure of the tree canopy; the two are complementary. However, current technologies lack a cross-modal correlation model between spectral and structural features, limiting the accuracy and window period of early detection. Summary of the Invention

[0013] This invention provides an intelligent early inspection method for pine wilt disease based on land-air multimodal fusion. It models and detects specific spectral characteristics of pine wilt disease in the early stage, namely 1-3 weeks after infection and before discoloration. It improves the efficiency and accuracy of identification through a two-stage progressive detection mechanism of UAV initial screening and ground unmanned vehicle fine screening. Modal complementary fusion verifies the detection period by 10-20 days from both intrinsic and extrinsic dimensions, and improves the diagnostic accuracy through cross-modal fusion reasoning.

[0014] This invention provides the following technical solution: an early intelligent inspection method for pine wilt disease based on land-air multimodal fusion, comprising:

[0015] S1. The UAV is equipped with a hyperspectral instrument to acquire spectral data of pine forest area with a spectral range of 400-1000nm. Based on the spectral index of disease, the suspected disease pixels are initially screened and identified. The location and spectral summary characteristics of the suspected disease pixels are recorded. Based on the spatial distance of the suspected disease pixels, clustering and aggregation are performed to generate a list of suspected targets.

[0016] S2. Calculate the severity of the disease and the initial screening confidence level based on the spectral summary features of the suspected target, and generate the optimal inspection path for the ground unmanned vehicle in combination with the target location.

[0017] S3. Ground unmanned vehicles equipped with data acquisition devices collect full-band multi-angle spectral data, 3D point cloud data, and stereo vision data of suspected targets;

[0018] S4. Input the data collected in step S3 into the spectral-structural-visual trimodal fusion network. The spectral-structural-visual trimodal fusion network includes an encoder that extracts spectral, structural and visual features and a cross-modal interactive fusion module. Using spectral features as query vectors, the structural and visual features are weighted and fused to output the final diagnostic result.

[0019] S5. Generate a graded treatment report containing location, disease level, and treatment recommendations based on the confidence level of the diagnostic results.

[0020] This intelligent early detection method for pine wilt disease based on land-air multimodal fusion addresses three major pain points in existing pine wilt disease detection technologies: difficulty in early detection, low accuracy of single platforms, and fragmented modal information. It provides an intelligent early detection method for pine wilt disease based on air-space collaboration and spectral-structural dual-modal fusion, realizing full-process automation from large-area initial screening to refined diagnosis of individual trees, advancing the detection window to 15-20 days after infection, and providing key technical support for the early prevention and control of pine wilt disease.

[0021] As an optional solution to the early intelligent inspection method for pine wilt disease based on land-air multimodal fusion described in this invention, steps S1 and S4 both perform disease discrimination based on spectral data using a disease-specific early spectral index system. This system includes three core disease spectral indicators: chlorophyll absorption valley blue shift, water absorption depth ratio, and lignin-cellulose absorption characteristic shift. The spectral summary feature includes these three core disease spectral indicators. The chlorophyll absorption valley blue shift is the difference between the absorption valley wavelength of the sample and the healthy reference value of the same variety. When the difference is less than... The threshold is used to identify suspected early infection indicators; the moisture absorption depth ratio is the ratio of the moisture absorption depth of the sample to the healthy reference value. When the ratio is less than the threshold, it is identified as a suspected early infection indicator; the lignin-cellulose absorption characteristic offset is the difference between the average spectral slope of the sample and the healthy reference value. When the absolute value of the difference is greater than the threshold, it is identified as a suspected early infection indicator. After collecting full-band multi-angle spectral data of suspected targets in step S4, the cellulose absorption characteristic offset of the corresponding spectra in steps S1 and S4 is cross-validated, and the initial screening identification in step S1 is corrected based on the validation results. When generating the optimal inspection path for the ground unmanned vehicle in conjunction with the target location, the digital elevation model is called for accessibility assessment. An improved ant colony algorithm is adopted, with the shortest total travel distance as the optimization objective. The constraints include the ground vehicle's range and terrain passability. The improvement strategy of the improved ant colony algorithm includes: introducing a terrain slope factor and a DEM elevation difference penalty term into the heuristic function. , As a heuristic information factor, The path segment length, The slope angle, For the elevation difference, The elevation normalization constant is used; pheromone updates employ a hybrid mechanism combining an elite strategy and a maximum-minimum ant system, with pheromone concentration limited to... Within the scope, during the path search process, unreachable nodes are removed from the node set in advance, and the joint constraint conditions of slope and reachability are set as double slope constraints.

[0022] As an optional scheme of the early intelligent inspection method for pine wilt disease based on land-air multimodal fusion described in this invention, the method includes: collecting full-band multi-angle spectra of suspected targets, including:

[0023] S3.1 Whiteboard calibration under natural lighting conditions;

[0024] S3.2. Based on the characteristics of the pine tree crown, multi-angle data collection was carried out. After collecting 3-5 frames continuously from each angle, the median value was taken to obtain the multi-angle crown spectrum.

[0025] S3.3. The multi-angle canopy spectrum was weighted according to the biochemical response of mature leaves after pine wilt nematode infection and then weighted to obtain the full-band multi-angle spectrum.

[0026] As an optional scheme of the early intelligent inspection method for pine wilt disease based on land-air multimodal fusion described in this invention, the following steps are included: Collecting 3D point cloud data of suspected targets includes: acquiring 3D point clouds of pine trees through lidar scanning; after denoising, ground separation, and individual tree segmentation, extracting the canopy point cloud density index and branch exposure index as inputs to the spectral-structural-visual three-modal fusion network. Collecting stereoscopic visual data of suspected targets includes: acquiring stereoscopic image pairs of suspected diseased branches using a binocular camera; accurately extracting the needle region using a lightweight semantic segmentation network; using the SGBM semi-global stereo matching algorithm to obtain a disparity map; and generating a 3D depth map by combining camera calibration parameters and a depth formula. The spectral feature encoder preprocesses the spectral data and its first derivative through smoothing filtering, then extracts spectral feature vectors representing the internal biochemical state of the trees. The structural feature encoder extracts multi-scale 3D geometric features from the cleaned canopy 3D point cloud using the PointNet++ architecture and a SetAbstraction hierarchical structure, outputting a structural feature vector. The visual feature encoder inputs the disparity map into the ResNet-50 backbone network, extracts hierarchical visual features, and outputs a visual feature vector. The cross-modal interactive fusion module performs cross-attention calculations and concatenates the spectral feature vectors, structural feature vectors, and visual feature vectors, generating a four-level disease classification result including healthy, suspected, highly suspected, and confirmed diseases based on the concatenated vector.

[0027] Cross-attention calculation includes:

[0028] S4.1 Feature Space Mapping: Select any two feature vectors and map them to query vector, key vector and value vector respectively through a learnable linear transformation matrix;

[0029] S4.2 Attention Weight Calculation: By scaling the dot product operation and combining it with the Softmax function, the attention weight matrix at each position is calculated to measure the correlation between the two.

[0030] S4.3 Cross-modal fusion representation generation: The calculated attention weights are used to weight and aggregate the value vectors to output the cross-modal fusion representation vector.

[0031] The present invention has the following beneficial effects:

[0032] 1. This intelligent early inspection method for pine wilt disease based on land-air multimodal fusion balances efficiency and accuracy through a two-stage progressive detection paradigm. It establishes a two-stage progressive detection mechanism: wide-area hyperspectral initial screening by UAVs (with recall priority) combined with multi-sensor fine diagnosis by ground-based unmanned vehicles (UAVs) (with accuracy priority). UAVs perform large-area rapid scanning at a rate of 40-60 acres / minute, with the initial screening model identifying suspected diseased trees in real time; ground-based UAVs only conduct close-range fine-tuning of suspected targets, achieving a diagnostic accuracy of no less than 90%. This fundamentally solves the contradiction between efficiency and accuracy that cannot be achieved with a single platform.

[0033] 2. This intelligent early detection method for pine wilt disease based on land-air multimodal fusion employs complementary fusion analysis of spectral and structural modes. Hyperspectral imaging (400-2500nm) captures biochemical pathological signals such as chlorophyll degradation, water transport obstruction, and accumulation of defensive secondary metabolites within the needles; LiDAR acquires three-dimensional point clouds of the canopy, quantifying structural abnormalities such as reduced canopy density and increased branch and trunk exposure caused by needle shedding. The two modes complementarily validate each other from both intrinsic and extrinsic dimensions, advancing the detectable window for pine wilt disease from the visual discoloration stage (approximately 4-6 weeks post-infection) to 15-20 days post-infection, approximately 10-20 days earlier than existing technologies.

[0034] 3. This method for early intelligent inspection of pine wilt disease based on land-air multimodal fusion constructs a disease-specific early spectral index system. Based on measured hyperspectral data at different infection stages, specific spectral indicators for pine wilt disease—chlorophyll absorption trough-blue shift, water absorption depth ratio, and lignin-cellulose absorption characteristic shift—are established, forming a quantifiable and physically interpretable ternary criterion for early disease diagnosis.

[0035] 4. This intelligent early inspection method for pine wilt disease based on land-air multimodal fusion improves diagnostic accuracy through cross-modal fusion reasoning. It employs a Cross-Attention Transformer to achieve trimodal cross-modal fusion of hyperspectral features, LiDAR 3D structural features, and binocular visual features. Utilizing a cross-attention mechanism, different modal features interact and complement each other in the semantic space, outputting a four-level fine-grained diagnostic result (healthy / suspected / highly suspected / confirmed), with diagnostic accuracy significantly superior to single-modal solutions.

[0036] 5. This intelligent early inspection method for pine wilt disease based on land-air multimodal fusion achieves a fully automated closed-loop process. From initial screening by drones to cloud-based task scheduling, autonomous navigation and diagnosis by ground vehicles, and generation of tiered reports, the entire process requires no manual intervention. It can be deployed in remote mountainous areas and large forest areas, with a detection efficiency more than 100 times that of manual inspection.

[0037] 6. This intelligent early detection method for pine wilt disease based on land-air multimodal fusion has strong versatility and scalability. The disease-specific early spectral index system and preliminary screening model can be transferred to other forest pests and diseases (such as red turpentine beetle and pine needle brown spot disease), only requiring the addition of specific spectral feature data for the corresponding diseases and fine-tuning of the model for its applicability. Attached Figure Description

[0038] Figure 1 is a schematic diagram of the steps of the intelligent early inspection method for pine wilt disease based on land-air multimodal fusion in Embodiment 1 of the present invention.

[0039] Figure 2 is a diagram of the spectral-structural-visual trimodal Cross-Attention Transformer fusion network structure of Embodiment 1 of the present invention.

[0040] Figure 3 is a schematic diagram of the early spectral characteristics and indicators of pine wilt disease in Example 1 of the present invention.

[0041] Figure 4 is a flowchart of the calculation of early-stage specific spectral indicators of pine wilt disease in Example 1 of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Example 1

[0044] Please refer to Figures 1 to 4. One of the methods for early intelligent inspection of pine wilt disease based on land-air multimodal fusion includes:

[0045] Step S1: Wide-area hyperspectral initial screening by UAV

[0046] The drone, equipped with a lightweight pushbroom hyperspectral imager, conducted large-area canopy spectral data acquisition in the target forest area. The acquired data was used to obtain spectral summary feature vectors through an early-stage specific spectral index system for pine wilt disease. These vectors were then used to initially screen suspected targets using a lightweight 1D-CNN model optimized with TensorRT and quantized with INT8. After clustering and aggregation using the DBSCAN algorithm, GPS coordinates were recorded to form a list of suspected targets.

[0047] The imager has a spectral range of 400-1000nm, a spectral resolution of no more than 5nm, and no less than 120 bands. The drone's flight parameters are set as follows: relative tree canopy height of 30-50m, terrain-following flight, flight speed of about 5m / s, and a single flight coverage area of ​​about 400-600 acres.

[0048] The lightweight 1D-CNN initial screening model, optimized with TensorRT and quantized with INT8, has approximately 0.5M parameters, a single-pixel inference time of no more than 5ms, performs real-time pixel-by-pixel inference, and outputs normal or suspected binary classification results.

[0049] During clustering and aggregation, the DBSCAN algorithm is used to cluster and aggregate multiple suspected pixels with a spatial distance of no more than 3m. Multiple pixels on the same tree canopy are merged into a single suspected target, and their GPS coordinates are recorded. The GNSS / RTK positioning accuracy is no more than 3cm. The DBSCAN clustering parameters are set as follows: neighborhood radius. The corresponding spatial resolution of UAV imagery is set to a minimum sample size of MinPts=5. Considering that a single tree canopy contains approximately 10-30 pixels, the distance metric used is Euclidean distance.

[0050] The spectral summary feature vector contains calculated values ​​for three core spectral indices: chlorophyll absorption trough blue shift. Water absorption depth ratio Lignin-cellulose absorption characteristic offset Among them, the lignin-cellulose absorption characteristic offset Because the recorded camera's spectral range only covers 400-1000 nm, VNIR is temporarily used as a substitute index, with the spectral slope shift in the 750-1000 nm range. As a proxy variable for the lignin-cellulose index, and After experimental calibration, the correlation coefficient r was determined. 2 >0.82, p<0.01; 680nm absorption valley depth; 550nm green peak reflectance value; red edge position (REP, wavelength of red edge inflection point); Normalized Difference Vegetation Index (NDVI). A total of 7 feature vectors are used as inputs to the 1D-CNN initial screening model. The 1D-CNN input layer has a dimension of 7 (corresponding to 7-dimensional features), the output layer has a dimension of 2 (normal / suspected), and the hidden layer configuration is: Conv1D(7,32,kernel=5)+ReLU+MaxPool(2)+Conv1D(32,64,kernel=3)+ReLU+GlobalAvgPool+FC(64,2).

[0051] Step S2: Cloud Task Scheduling

[0052] The drone uploads a list of suspected targets to a cloud-based analysis platform via its onboard data link communication module. This module supports dual-mode operation of 4G / 5G cellular mobile communication and LoRa spread spectrum communication. The list of suspected targets includes GPS coordinates, spectral summary feature vectors, and confidence scores. The cloud-based collaborative task management module performs accessibility assessments based on the spatial distribution of the targets and utilizes a digital elevation model. An improved ant colony algorithm is used to generate the optimal inspection path for the ground vehicle, with the shortest total travel distance as the optimization objective. Constraints include the ground vehicle's range and terrain passability. A JSON-formatted inspection task list is generated and automatically distributed to the unmanned ground vehicle.

[0053] Specific improvement strategies for the ant colony algorithm include:

[0054] (i) Introduce a terrain slope factor and a DEM elevation difference penalty term into the heuristic function: , As a heuristic information factor, The path segment length, The slope angle, For the elevation difference, The elevation normalization constant is used; pheromone updates employ a hybrid mechanism combining an elite strategy and a maximum-minimum ant system, with pheromone concentration limited to... Within the range, the number of elite ants is 5; during the path search process, unreachable nodes (steep cliffs, water bodies, bare rocks, etc.) are removed from the node set beforehand. The DEM data uses ASTER GDEM v3 with a resolution of 30m. The joint constraints of slope and accessibility are: the slope of any segment of the path does not exceed 35 degrees (matching the maximum climbing ability of the ground vehicle), and the path segment does not cross grids with a slope exceeding 45 degrees.

[0055] Step S3: Fine-grained re-inspection of ground-based unmanned vehicles using multiple sensors

[0056] The ground-based unmanned vehicle is equipped with the following sensor payloads and uses autonomous navigation based on GNSS / RTK + IMU + visual SLAM multi-source fusion positioning to sequentially approach each suspected target (stopping 3-5m away from the target pine tree) and perform multi-sensor fine re-inspection:

[0057] (a) A full-band hyperspectral imager with a spectral range of 400-2500nm, a VNIR band resolution of no more than 3nm, a SWIR band resolution of no more than 8nm, and a number of bands of no less than 300. It is equipped with an electric rotating gimbal and canopy spectra are acquired at three angles of 0 degrees, 30 degrees and 60 degrees. 3-5 frames are acquired continuously at each angle. The median spectrum is taken to reduce random noise. Whiteboard correction is automatically performed before acquisition.

[0058] The whiteboard calibration process includes: before data acquisition, using a standard Spectralon diffuse whiteboard (reflectivity ≥50%) as a reference, acquiring the whiteboard spectrum under natural lighting conditions. and dark current The target reflectivity is calculated as follows: The three pitch angles were set as follows: 0 degrees for collecting the spectrum from the lower and middle parts of the canopy (corresponding to older leaves), 30 degrees for collecting the spectrum from the middle part of the canopy (corresponding to mature leaves), and 60 degrees for collecting the spectrum from the top of the canopy (corresponding to new shoots). The weighted average weights of the three angle spectra were 0.3, 0.4, and 0.3, respectively (the middle part had the highest weight because the biochemical response of mature leaves after pine wilt nematode infection was the most significant). The positioning accuracy of the motorized rotating gimbal was + / -0.5 degrees, and the acquisition time for a single angle was approximately 2-3 seconds (including the gimbal rotation and positioning time). The total acquisition time for the three rounds was approximately 10-12 seconds, which matched the LiDAR scanning time (10-15 seconds), enabling synchronous parallel acquisition.

[0059] Multi-source fusion localization employs an extended Kalman filter (EKF) framework: state vector It includes 16 dimensions: 3D position, 3D velocity, 4D attitude quaternion, 3D accelerometer bias, and 3D gyroscope bias. The IMU provides acceleration and angular velocity measurements at 200Hz for state prediction, GNSS / RTK provides position observation at 10Hz, and visual SLAM provides 6-DOF pose observation at 30Hz. GNSS signal quality is monitored by the HDOP value; when HDOP > 2.0 or the number of locked satellites < 4, signal obstruction is identified, and the covariance weight of the visual SLAM observations is automatically increased to 0.8 (VSLAM):0.2 (GNSS). The fusion positioning accuracy in canopy environments has been verified to be better than 0.3m (RMSE), meeting navigation requirements at a target proximity of 3-5m. The ground vehicle has a range of at least 30km (48V / 200Ah battery, Panasonic 21700 cells), and a single charge can complete detailed re-inspection of approximately 80-120 suspected targets.

[0060] (b) A LiDAR laser radar (16 lines, ranging accuracy not exceeding ±2cm, horizontal field of view 360 degrees, vertical field of view not less than 30 degrees) is used to perform a 360-degree all-round three-dimensional scan of the target pine tree at a rotation speed of 10Hz (lasting 10-15 seconds) to obtain approximately 150,000-200,000 points of three-dimensional point cloud data of the tree crown.

[0061] CSF cloth simulation filtering parameters: cloth resolution 0.3m (matching tree canopy scale), stiffness coefficient 1 (default is rigid cloth), time step 0.65, maximum number of iterations 500. Statistical filtering denoising uses the K-nearest neighbor method, with a neighborhood k=30 and a standard deviation threshold. Outside points are removed if their average distance from their neighbors exceeds a threshold. After ground point segmentation, a region growing algorithm is used to segment the canopy point cloud into individual trees, with a growth radius of 0.5-1.2m (adaptively adjusted according to species), and a minimum of 50 points are retained per tree. LiDAR structural quantification indicators include:

[0062] (i) Canopy point cloud density index = Upper layer point cloud count / Total point cloud count (upper layer is defined as the area above 1 / 3 of the treetop height), after needles fall off. Significantly reduced;

[0063] (ii) Branch Exposure Index (BEI) = Number of point clouds of branches without leaf cover / Total number of point clouds. It is segmented by point cloud reflectance intensity threshold (reflectance intensity I < 15 is classified as branch, I ≥ 15 is classified as needle). The above structural index serves as auxiliary input features for the LiDAR branch in the Cross-Attention Transformer.

[0064] (c) A binocular stereo vision camera (resolution not less than 1920x1080, baseline distance 10-15cm, frame rate not less than 30fps) is used to collect needle-scale stereo image pairs in the suspected lesion twig area. The disparity map is generated by the SGBM semi-global stereo matching algorithm and the three-dimensional depth map of the needle area is restored by combining the camera calibration parameters.

[0065] SGBM semi-global stereo matching parameters: disparity search range [0, 128], SAD window size 7x7, smoothing penalty P1=8x3x7 2 =1176, Discontinuous penalty P2=32x3x7 2 =4704, uniqueness ratio 12, sub-pixel precision enabled, left-right consistency check enabled. The depth recovery formula is Z=(f*b) / d, where f is the camera focal length, b is the baseline distance (10-15cm), and d is the disparity value. The ROI of the coniferous region is automatically extracted using a lightweight semantic segmentation network (U-Net, approximately 3M parameters), and depth recovery is performed only on the coniferous region to reduce computation. The U-Net segmentation model achieves an IoU > 0.85 in the coniferous region. The spatial resolution of the 3D depth map reaches approximately 0.5mm / pixel at a distance of 3-5m, which can distinguish the morphological features of a single coniferous leaf.

[0066] The ground vehicle uses a tracked or six-wheel independent suspension all-terrain chassis (with a climbing ability of no less than 35 degrees and an obstacle crossing height of no less than 20cm), and is equipped with an onboard high-performance computing unit (NVIDIA Jetson AGX Orin, computing power of 275 TOPS) for real-time data processing. When the GNSS signal is attenuated to an accuracy of less than 2m due to canopy obstruction, it automatically switches to a fusion positioning mode based on visual SLAM and IMU.

[0067] Step S4: Spectral-structural-visual trimodal fusion diagnosis

[0068] The multi-source data collected in step S3 is fed into the trained cross-modal fusion network, which is based on the Cross-Attention Transformer architecture and contains three parallel encoder branches and a cross-modal interaction fusion module.

[0069] The system is designed with the following fault-tolerance strategies for the complex operating environment in forest areas:

[0070] (1) Severe weather: When the wind speed is >10m / s or the rainfall intensity is >2mm / h, the UAV will automatically trigger the return-to-home procedure, the ground vehicle will activate the sensor rain cover and switch to standby mode, and after the weather conditions recover (queried every 15 minutes by the integrated meteorological API), it will automatically resume the unfinished task from the breakpoint. Wind speed and rainfall thresholds are monitored in real time by the airborne micro weather station.

[0071] (2) Low light conditions: When the ambient irradiance is <200W / m² (early morning / dusk / dense cloud cover), the hyperspectral imager automatically switches to active light source supplement mode (built-in halogen tungsten lamp, color temperature 3200K, power 50W), and at the same time adjusts the integration time from the default 10ms to 30-50ms to ensure that the spectral signal-to-noise ratio is >=30dB.

[0072] (3) Communication interruption: When the 4G / 5G signal is interrupted and LoRa cannot connect, the drone and ground vehicle will cache the task data locally (onboard SSD storage capacity 256GB, which can cache the full data of about 72 hours of continuous operation). After the communication is restored, the data will be automatically synchronized to the cloud through the breakpoint resume transmission protocol (supporting MD5 checksum and deduplication).

[0073] (4) Multiple canopy overlap: When adjacent pine canopy layers overlap, spectral unmixing (linear mixing model + abundance inversion) is used to separate mixed pixels. Endmembers with an abundance >60% after separation are analyzed separately to ensure diagnostic accuracy at the single tree level.

[0074] (5) Seasonal phenological changes: The feature library stores reference spectra and indicator baselines of healthy pine trees in different seasons (spring / summer / autumn / winter). During detection, the reference baselines of the corresponding season and region are automatically matched according to GPS date and latitude and longitude to eliminate the systematic influence of phenological differences and latitudinal gradients on the diagnostic results.

[0075] (a) Hyperspectral encoder with full-band reflectance spectrum and its first derivative spectrum The input is a combined spectral feature vector (smoothed by SG filtering, with a window width of 15 and a second-order polynomial), processed through 3 layers of one-dimensional convolution, global average pooling, and fully connected layers, outputting a 256-dimensional spectral feature vector. .

[0076] (b) The LiDAR point cloud encoder takes as input a canopy 3D point cloud (approximately 150,000-200,000 points) after statistical filtering denoising and CSF cloth simulation filtering ground point segmentation. It uses the PointNet++ architecture to extract multi-scale 3D structural features through a Set Abstraction hierarchical structure and outputs a 256-dimensional structural feature vector. .

[0077] (c) Binocular visual encoder: Using the disparity map generated by SGBM stereo matching as input, a ResNet-50 backbone network (pre-trained weight initialization) is used to extract hierarchical visual features, and a 256-dimensional visual feature vector is output. .

[0078] The three feature vectors are fed into the Cross-Attention Transformer module, where three sets of cross-modal cross-attention are calculated: spectral → LiDAR cross-attention (with...). For Query, The three sets of fused representation vectors (each 256-dimensional) are concatenated into a 768-dimensional vector, which is then processed by a fully connected layer (128-dimensional, ReLU+Dropout(0.3)) and Softmax to output four-level classification probabilities: Level I - Healthy, Level II - Suspected (only spectral indicators are abnormal), Level III - Highly Suspected (spectral and structural bimodal abnormalities), and Level IV - Confirmed (all three modalities are abnormal, and the spectral abnormality pattern matches the known pine wilt disease patterns in the feature library with a degree of not less than 85%).

[0079] Step S5: Tiered Reporting and Handling Recommendations

[0080] The system automatically generates a graded detection report for each detection target (including GPS coordinates, detection level, confidence score, spectral index values, quantitative indicators of canopy 3D structure, and visual images), and automatically generates differentiated treatment recommendations based on the confirmed diagnosis level: Level I - Healthy → Routine monitoring; Level II - Suspected → Manual retesting or intensified monitoring within 7 days; Level III - Highly Suspected → Sampling and sending samples to the laboratory for PCR / nematode isolation and testing within 3 days; Level IV - Confirmed → Immediately mark the infected trees, initiate the infected tree removal process (felling, fumigation, on-site burning or shredding), and administer preventative pesticide injections to pine trees within a 100m radius.

[0081] It should be noted that the construction rules for the early-stage specific spectral index system for pine wilt disease are as follows: Through systematic analysis of the hyperspectral reflectance data of needles from different pine species (Masson pine, black pine, slash pine, Chinese pine, etc.) at different infection stages (healthy, incubation period approximately 7-14 days, initial manifestation period approximately 15-30 days, development period approximately 30-45 days), three core spectral indices and their calculation formulas were established:

[0082] (1) Blue shift of chlorophyll absorption valley. Chlorophyll has a strong absorption valley around 680 nm, and pine wood nematode infection causes this absorption valley to shift towards shorter wavelengths. Definition The wavelength of the absorption valley of the sample to be tested Health reference values ​​for the same variety The difference: .when At that time, it was judged as a suspected indicator of early infection.

[0083] (2) Moisture absorption depth ratio. A characteristic absorption valley for liquid water exists at 970 nm; infection by pine wood nematode causes this absorption valley to become shallower. Definition Moisture absorption depth of the sample to be tested Compared with health reference values The ratio of, where Linear interpolation was performed using reflectance values ​​at 900nm and 1050nm as baselines: , , ,when At that time, it was judged as a suspected indicator of early infection. The abnormal judgment threshold of each indicator was determined by ROC analysis: ROC curves were constructed using the indicator calculation values ​​of approximately 1000 known healthy / infected pine tree samples (confirmed by PCR), and the point with the maximum Youden index (sensitivity + specificity - 1) was selected as the optimal threshold.

[0084] Experimental calibration:

[0085] (i) The threshold is -3nm, that is, when Chlorophyll levels <-3nm were considered abnormal (AUC=0.87, sensitivity 82%, specificity 79%).

[0086] (ii) The threshold is 0.85, that is, when A value <0.85 is considered an abnormality in moisture index (AUC=0.91, sensitivity 86%, specificity 83%).

[0087] (iii) The threshold is 0.005%*nm^{-1}, that is, when The results indicated an abnormality in the structural component index (AUC=0.84, sensitivity 78%, specificity 81%). The three-choice-two-option voting rule of the ternary criterion was experimentally verified to control the initial screening false alarm rate to within 8% under the condition of 95% recall.

[0088] Since the airborne hyperspectral imager in step S1 has a spectral range of 400-1000 nm (VNIR), it only covers spectral parameters. and indicators The wavelength range. Indicator The SWIR band around 1200nm is involved, and cannot be directly calculated in stage S1. Therefore, the three surrogate metrics actually input to the 1D-CNN in stage S1 are: , and (750-1000nm slope offset, as) (VNIR agent).

[0089] After the ground vehicle's full-band spectrometer (covering 400-2500nm) collects data in step S4, the indicators are... Perform precise calculations and compare them with proxy indicators. Perform cross-validation: If , The conversion coefficient, calibrated experimentally (approximately 1.24), is used to correct the initial screening judgment based on the full-band calculation results. This two-layer mechanism of proxy + verification ensures the real-time performance of the initial screening in S1 while avoiding errors introduced by proxy indicators through the precise verification in S4.

[0090] (3) Lignin-cellulose absorption characteristic shift. An absorption shoulder exists around 1200 nm for both lignin and cellulose; the degradation of cell wall components in the later stages of pine wood nematode infection leads to changes in scattering characteristics. Definition The average spectral slope of the sample in the 1150-1250 nm range Compared with health reference values The difference: , ,when At that time, it was judged as a suspected indicator of early infection.

[0091] The above three indicators constitute a ternary criterion system. In step S1, the airborne 1D-CNN initial screening model performs binary classification based on the joint analysis of the above three spectral indicators; in specific implementation, when at least two of the three indicators are abnormal, it is judged as spectral abnormality, triggering the subsequent multimodal fusion diagnosis process.

[0092] Regarding cross-modal fusion inference methods, this application proposes a trimodal cross-modal fusion architecture based on Cross-Attention Transformer. Unlike the traditional approach of simply concatenating multi-source data and feeding it into a single classifier, this invention uses a cross-attention mechanism to allow feature vectors from different modalities to interactively fuse in the semantic space. This enables each modality to actively query complementary information from other modalities, thereby generating a richer and more robust fused representation than a single modality. The following details the encoder design, cross-modal attention mechanism, and fusion classification.

[0093] (1) Hyperspectral encoder. The input is a full-band reflectance spectrum. and its first derivative spectrum The joint vector, after SG smoothing filtering (window width 15, second-order polynomial), has a dimension of approximately 300. The encoder employs a three-layer one-dimensional convolutional neural network (1D-CNN): the first layer has a kernel size of 7 and 32 channels; the second layer has a kernel size of 5 and 64 channels; and the third layer has a kernel size of 3 and 128 channels. Each convolutional layer is followed by batch normalization and ReLU activation. After the convolutional layers, global average pooling is used to compress the spatial dimension into a single vector, which is then mapped to a 256-dimensional feature space through a fully connected layer, finally outputting a hyperspectral feature vector. This encoding process can be abstractly represented as: .

[0094] (2) LiDAR point cloud encoder. The input is a 3D point cloud of the canopy after statistical filtering for noise reduction and CSF cloth simulation filtering for ground point segmentation, approximately 150,000-200,000 points, each point containing 3D spatial coordinates. The encoder employs a PointNet++ architecture, extracting multi-scale local geometric features from an unordered point cloud through hierarchical Set Abstraction operations: First, keypoints are selected by sampling from the farthest point. Then, a spherical neighborhood is constructed centered on each keypoint, and PointNet is applied to the point set within the neighborhood to extract local features. This operation is repeated twice to capture the fine-grained structure and overall morphological features of the canopy at different scales. Finally, after global feature aggregation and a fully connected layer, a 256-dimensional structural feature vector is output. .

[0095] (3) Binocular vision encoder. The input is a pair of needle-scale stereo images acquired by a binocular stereo vision camera, with a resolution of [missing information]. The disparity map is generated using the SGBM semi-global stereo matching algorithm. The encoder employs a ResNet-50 backbone network, pre-trained on ImageNet and then transferred to this task. It extracts hierarchical visual features from low-level texture to high-level semantics through four residual stages. Finally, after global average pooling and fully connected layers, a 256-dimensional visual feature vector is output. .

[0096] (4) Cross-modal cross-attention fusion mechanism. The feature vectors output by the three encoders are fed into the Cross-Attention Transformer module, which is the core innovative component of this invention. Traditional self-attention mechanisms calculate the association weights between elements within the same sequence, while the cross-attention mechanism allows a feature vector from one modality to act as a query to retrieve complementary information from the feature vectors of another modality. For any two feature vectors from different modalities... and Cross-attention calculation consists of the following three steps.

[0097] The first step is to and Mapped to the query, key, and value spaces respectively using learnable linear transformation matrices: ,in For learnable parameter matrix, Dimensions for each attention head.

[0098] The second step involves calculating the attention weight matrix between the query vector and the key vector using a scaled dot product operation, which measures the difference between the two vectors. right The degree of relevance of each position: The scaling factor This is used to prevent the softmax gradient from vanishing due to excessively large dot product values. Attention weight matrix. Each line represents Corresponding position in the middle Attention distribution across all locations.

[0099] The third step involves using attention weights to weighted aggregate the value vectors, generating a cross-modal fusion representation: .

[0100] The process of constructing a known feature database for pine wilt disease:

[0101] (1) Collect full-band spectral, LiDAR point cloud and binocular vision data of infected pine trees (no less than 200 trees) confirmed by PCR / nematode isolation and detection, and record the number of days of infection (the infection start time is determined by regular PCR detection).

[0102] (2) The central values ​​and distribution ranges of each modality feature were statistically analyzed according to the infection stage (incubation period, initial manifestation period, and development period). , a reference template specific to the construction phase;

[0103] (3) The spectral anomaly mode matching degree is defined as the spectral vector to be measured. Reference vectors of the same variety and stage in the feature library Cosine similarity between them: A pattern match is determined when Sim ≥ 0.85. The feature library is updated annually based on newly confirmed samples, and an exponential moving average (EMA, decay coefficient 0.9) is used to fuse the old and new templates. To adapt to the annual evolution of disease characteristics and the variation of different geographical populations, the feature database is organized with a three-level index structure of variety-stage-region (provincial administrative region), supporting rapid retrieval.

[0104] The training of the cross-modal fusion network employs a three-stage strategy: Stage 1 (Independent pre-training of each encoder): The hyperspectral encoder undergoes self-supervised pre-training on a public hyperspectral vegetation dataset (such as GHISACONUS) with spectral reconstruction as the surrogate task; the LiDAR point cloud encoder is pre-trained for shape classification on ModelNet40 and forestry point cloud datasets; the binocular vision encoder uses ImageNet pre-trained weights. Stage 2 (Joint fine-tuning): An annotated pine wilt disease multimodal dataset (recommended to include at least 500 pine tree samples, covering healthy and various disease levels, with ground truth annotations by forestry experts) is used, employing the cross-entropy loss function... End-to-end training was performed using AdamW optimizer (learning rate 1e-4, weight decay 0.01), batch size 16, and a total of 150 epochs, with cosine annealing as the learning rate scheduling method. Data augmentation strategies included: random spectral noise injection (signal-to-noise ratio 30-45dB), random point cloud rotation (+ / -15 degrees) / scaling (0.9-1.1 times), and random brightness / contrast perturbation of the image. The third stage (domain adaptive fine-tuning): For different forest regions (differences in species, climate, and altitude), an adversarial domain adaptation method (domain classifier gradient reversal layer) was used to reduce the distribution differences between the source and target domains. Class imbalance was addressed using... This alternative to standard cross-entropy loss alleviates the class imbalance problem where healthy samples far outnumber diseased samples. Recommended training hardware configuration: 4 x NVIDIA A100 80GB GPUs; training time approximately 36 hours (including data loading).

[0105] This method has a complete detection-feedback-optimization closed-loop mechanism:

[0106] (1) The diagnostic results of step S5 (especially the laboratory PCR retest confirmation results of level III and IV) are fed back to the model training database through the cloud platform as incremental training samples. Every 200 newly confirmed samples are accumulated, the model fine-tuning process is automatically triggered, and the update cycle does not exceed 30 days.

[0107] (2) The initial screening model in step S1 and the confirmed model in step S4 adopt a differentiated update strategy: the model in step S1 uses recall as the optimization target (penalizing missed detections) and adopts online hard example mining. The loss weight of the missed detection samples confirmed in step S4 is increased by 10 times to participate in the next round of fine-tuning. The model in step S4 uses accuracy as the optimization target (penalizing false positives) and adopts an active learning strategy to screen low confidence samples (maximum classification probability < 0.7) and add them to the training set after priority manual annotation.

[0108] (3) Establish a false detection / missed detection log database in the cloud, and analyze the regularity of false alarms and missed detections regularly (every quarter) (such as high false alarm rate under specific terrain conditions, light angle, and pine tree species scenarios), and optimize feature extraction parameters and judgment thresholds accordingly.

[0109] (4) Model version management: A new version of the model is generated after each fine-tuning. The most recent 5 versions of the model are retained in the cloud. The effectiveness of the new model is verified on new data using A / B testing (recall rate does not decrease, precision rate is improved). Only after the verification is passed can the current production model deployed on drones and ground vehicles be replaced.

[0110] The fusion vector It contains an enhanced representation of modality A after retrieval with modality B as the context, while preserving the semantic structure of modality A itself.

[0111] This method calculates three sets of cross-attention to achieve fully connected interactions between the three modalities:

[0112] (i) Spectroscopic-LiDAR cross-attention: using hyperspectral features as queries and LiDAR features as keys and values ​​to generate a spectral-structure fusion representation. ;

[0113] (ii) LiDAR-visual cross-attention: using LiDAR features as queries and visual features as keys and values ​​to generate structure-visual fusion representations. ;

[0114] (iii) Spectral-visual cross-attention: using hyperspectral features as the query and visual features as the key and value, a spectral-visual fusion representation is generated. Each cross-attention group is calculated independently and has its own learnable parameter matrix.

[0115] (5) Fusion Classification Layer. The fusion representation vectors of the three sets of cross-attention outputs are concatenated to obtain the comprehensive multimodal fusion features: The concatenated features are then processed through two fully connected network layers for classification. The first layer maps the 768 dimensions to 128 dimensions, followed by a ReLU activation function and Dropout regularization with a dropout rate of 0.3 to prevent overfitting. The second layer maps the 128 dimensions to 4 dimensions, followed by a Softmax function, outputting a four-level classification probability distribution. .

[0116] The four-level classification means: Level I (Healthy), corresponding to the highest probability of being healthy; Level II (Suspected), only spectral indicators (such as...) , , Level 1 (highly suspected): Abnormalities are found in both spectral and LiDAR structural indicators (such as canopy density index and branch exposure), forming dual evidence of internal and external causes; Level 2 (confirmed): Abnormalities are found in all three modalities, and the spectral abnormality pattern matches the known feature library of pine wilt disease with a degree of not less than 85%, constituting a triple confirmation of spectral, structural, and visual factors.

[0117] The total number of parameters in this cross-modal fusion network is approximately 25M, and the single inference time on an NVIDIA Jetson AGX Orin (275 TOPS computing power) is less than 200ms. Compared with traditional feature splicing or simple voting fusion methods, the cross-attention fusion mechanism can adaptively learn the correlation weights between features of different modalities. Especially when dealing with the fine distinction between adjacent disease levels (such as Level II suspected and Level III highly suspected), cross-modal interaction can effectively capture complex diagnostic patterns that cannot be expressed independently by a single modality: for example, when the spectral index of conifers shows only slight abnormalities (weak evidence from a single spectral source) but the LiDAR canopy point cloud shows a significant decrease in density (clear structural evidence), the spectral-LiDAR cross-attention mechanism automatically increases the attention weight of structural features on spectral features, strengthening the complementary information of the two in the fused representation, thereby pushing the diagnostic confidence from Level II to Level III. This adaptive modal weighting capability is not possessed by traditional fixed-weight fusion methods, and is the core reason why this method is superior to existing technologies in terms of early detection window and diagnostic accuracy.

[0118] Regarding the two-stage progressive detection paradigm: This invention proposes a two-stage progressive strategy of wide-area initial screening → refined diagnosis:

[0119] Level 1 (S1) focuses on recall (no less than 95%) and detection efficiency (40-60 acres / minute), and uses a lightweight model to run in real time on edge devices;

[0120] The second stage (S3-S4) focuses on accuracy (no less than 90%), performing full-band fine re-examination and cross-modal fusion diagnosis only on suspected targets output from the first stage (usually less than one-thousandth of the total scans). This paradigm decouples detection efficiency and diagnostic accuracy into two independent optimization stages, resolving the inherent efficiency-accuracy contradiction of a single platform at the architectural level.

[0121] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0122] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for early intelligent inspection of pine wilt disease based on land-air multimodal fusion, characterized in that, include: S1. The UAV is equipped with a hyperspectral instrument to acquire spectral data of pine forest area with a spectral range of 400-1000nm. Based on the spectral index of disease, the suspected disease pixels are initially screened and identified. The location and spectral summary characteristics of the suspected disease pixels are recorded. Based on the spatial distance of the suspected disease pixels, clustering and aggregation are performed to generate a list of suspected targets. S2. The list of suspected targets includes GPS coordinates, spectral summary feature vectors and confidence scores. The cloud-based collaborative task management module performs accessibility assessment based on the spatial distribution of the targets and calls the digital elevation model. An improved ant colony algorithm is used to generate the optimal inspection path for the ground vehicle, with the shortest total driving distance as the optimization objective. Constraints include the ground vehicle's driving range and terrain passability. A JSON-formatted inspection task list is generated and automatically sent to the ground unmanned vehicle. S3. Ground unmanned vehicles equipped with data acquisition devices collect full-band multi-angle spectral data, 3D point cloud data, and stereo vision data of suspected targets; S4. Input the data collected in step S3 into the spectral-structural-visual trimodal fusion network. The spectral-structural-visual trimodal fusion network includes an encoder that extracts spectral, structural and visual features and a cross-modal interactive fusion module. Using spectral features as query vectors, the structural and visual features are weighted and fused to output the final diagnostic result. S5. Generate a graded treatment report containing location, disease level, and treatment recommendations based on the confidence level of the diagnosis results; Both steps S1 and S4 are based on a disease-specific early spectral index system to identify diseases from spectral data. This system includes three core disease spectral indicators: chlorophyll absorption trough blue shift, water absorption depth ratio, and lignin-cellulose absorption characteristic shift. The spectral summary features include these three core disease spectral indicators. After collecting full-band multi-angle spectral data of suspected targets in step S3, the lignin-cellulose absorption characteristic shifts of the corresponding spectra in steps S1 and S3 are cross-validated. Specifically, the lignin-cellulose absorption characteristic shift of the spectrum corresponding to step S1 uses the 750-1000nm slope shift as a proxy indicator, while the lignin-cellulose absorption characteristic shift of the spectrum corresponding to step S3 uses the full-band calculation results. The initial screening identification in step S1 is corrected based on the validation results.

2. The method for early intelligent inspection of pine wilt disease based on land-air multimodal fusion according to claim 1, characterized in that: Improvement strategies for the ant colony algorithm include: introducing a terrain slope factor and a DEM elevation difference penalty term into the heuristic function. , As a heuristic information factor, The path segment length, The slope angle, For the elevation difference, The elevation normalization constant is used; pheromone updates employ a hybrid mechanism combining an elite strategy and a maximum-minimum ant system, with pheromone concentration limited to... Within the scope, during the path search process, unreachable nodes are removed from the node set in advance, and the joint constraint conditions of slope and reachability are set as double slope constraints.

3. The method for early intelligent inspection of pine wilt disease based on land-air multimodal fusion according to claim 1, characterized in that: The acquisition of full-band, multi-angle spectra of suspected targets includes: S3.1 Whiteboard calibration under natural lighting conditions; S3.

2. Based on the characteristics of the pine tree crown, multi-angle data collection was carried out. After collecting 3-5 frames continuously from each angle, the median value was taken to obtain the multi-angle crown spectrum. S3.

3. The multi-angle canopy spectrum was weighted according to the biochemical response of mature leaves after pine wilt nematode infection and then weighted to obtain the full-band multi-angle spectrum.

4. The method for early intelligent inspection of pine wilt disease based on land-air multimodal fusion according to claim 1, characterized in that: The collection of 3D point cloud data of suspected targets includes: obtaining 3D point clouds of pine trees through LiDAR scanning, and after denoising, ground separation and single tree segmentation, extracting the canopy point cloud density index and branch exposure index as inputs to the spectral-structural-visual three-modal fusion network.

5. The method for early intelligent inspection of pine wilt disease based on land-air multimodal fusion according to claim 1, characterized in that: The acquisition of stereo vision data of suspected targets includes: acquiring stereo image pairs of suspected diseased branches using a binocular camera, accurately extracting the needle region through a lightweight semantic segmentation network, calculating the disparity map using the SGBM semi-global stereo matching algorithm, and generating a three-dimensional depth map by combining camera calibration parameters and depth formula.

6. The method for early intelligent inspection of pine wilt disease based on land-air multimodal fusion according to claim 5, characterized in that: The spectral feature encoder extracts spectral feature vectors that characterize the internal biochemical state of trees after preprocessing the spectral data and its first derivative through smoothing filtering. Structural Feature Encoder: The cleaned canopy 3D point cloud is processed using the PointNet++ architecture and a Set Abstraction hierarchical structure to extract multi-scale 3D geometric features and output structural feature vectors. The visual feature encoder inputs the disparity map into the ResNet-50 backbone network, extracts hierarchical visual features, and outputs a visual feature vector.

7. The method for early intelligent inspection of pine wilt disease based on land-air multimodal fusion according to claim 6, characterized in that: The cross-modal interactive fusion module performs cross-attention calculations and concatenates spectral feature vectors, structural feature vectors, and visual feature vectors. Based on the concatenated vectors, it generates a four-level disease classification result, including healthy, suspected, highly suspected, and confirmed diseases. Cross-attention calculation includes: S4.1 Feature Space Mapping: Select any two feature vectors and map them to query vector, key vector and value vector respectively through a learnable linear transformation matrix; S4.2 Attention Weight Calculation: By scaling the dot product operation and combining it with the Softmax function, the attention weight matrix at each position is calculated to measure the correlation between the two. S4.3 Cross-modal fusion representation generation: The calculated attention weights are used to weight and aggregate the value vectors to output the cross-modal fusion representation vector.

8. The method for early intelligent inspection of pine wilt disease based on land-air multimodal fusion according to claim 1, characterized in that: The chlorophyll absorption trough blue shift is the difference between the absorption trough wavelength of the sample and the healthy reference value of the same variety. When the difference is less than the threshold, it is judged as a suspected indicator of early infection. The water absorption depth ratio is the ratio of the water absorption depth of the sample to the healthy reference value. When the ratio is less than the threshold, it is judged as a suspected indicator of early infection. The lignin-cellulose absorption characteristic offset is the difference between the average spectral slope of the sample and the healthy reference value. When the absolute value of the difference is greater than the threshold, it is judged as a suspected indicator of early infection.

Images