Crop monitoring method and system based on multi-mode and adaptive optics of unmanned aerial vehicle

Abstract

The invention provides a crop monitoring method and system based on unmanned aerial vehicle multi-mode and adaptive optics, and belongs to the field of precision agriculture. The system adopts an active architecture design, a hyperspectral imaging unit, a Raman spectrum probe and a self-adaptive optical module are subjected to multi-modal integration, an intelligent sensing system capable of being carried on an unmanned aerial vehicle platform is constructed, and efficient collection of crop canopy data is achieved. In the data acquisition process, the system actively corrects wavefront distortion caused by atmospheric turbulence through the adaptive optical module, and the stability of optical signals is ensured. The collected multi-modal data is subjected to feature level fusion through a deep learning algorithm, and hyperspectral and Raman spectrum feature parameters are synchronously analyzed, so that accurate evaluation of crop health conditions and early warning of diseases and insect pests are realized, and a reliable technical solution is provided for modern agriculture.

Description

Technical Field

[0001] This invention belongs to the field of precision agriculture, specifically relating to a crop monitoring method and system based on UAV multimodal and adaptive optics. Background Technology

[0002] Crop health monitoring and early disease identification technologies have become core elements for improving agricultural production efficiency. Existing technologies mainly include three traditional methods: diagnostic methods based on human visual judgment, laboratory chemical analysis methods, and non-destructive monitoring methods using single-spectral techniques. Human visual diagnostic methods have significant technical drawbacks: their results heavily rely on the experience of professionals, leading to subjective judgment bias and low detection efficiency, especially in identifying early microscopic disease characteristics in crops. Chemical analysis methods require transporting crop samples to specialized laboratories for chromatography-mass spectrometry (GC-MS) analysis, which has three limitations: destructive testing characteristics; long detection cycles leading to poor timeliness; and high costs limiting widespread application. Although UAV-borne multispectral and hyperspectral remote sensing technologies and Raman spectroscopy have developed in recent years, key technical bottlenecks such as insufficient detection accuracy and poor environmental adaptability still exist, making it difficult to meet the real-time monitoring needs of precision agriculture.

[0003] Existing crop health monitoring technologies mainly fall into two categories: physical characterization and chemical composition analysis. Physical characterization primarily utilizes multispectral and hyperspectral imaging to assess crop health status by analyzing its reflectance spectral characteristics. Chemical composition analysis employs methods such as Raman spectroscopy and mass spectrometry to analyze crop molecular structure characteristics and disease indicators. Specifically, multispectral imaging, as a typical method of physical characterization, suffers from the following limitations: detection threshold settings rely excessively on empirical parameters; the effective detection limit only identifies macroscopically visible diseases; and it lacks the ability to detect microscopic molecular-level lesions. On the other hand, while Raman spectroscopy can acquire molecular vibrational fingerprints, its limitations in signal-to-noise ratio and sampling rate prevent it from meeting the real-time requirements of high-speed operation on UAV platforms and from providing information on large-scale spatial distribution.

[0004] A search revealed application CN120599500A, an intelligent pest and disease monitoring system based on multimodal data fusion from unmanned aerial vehicles (UAVs). This system utilizes a UAV equipped with a multi-source heterogeneous data acquisition module to acquire optical, multispectral, and thermal infrared data in parallel. After wireless link encoding and format adaptation, the data is stored on the ground. The system employs an improved YOLO algorithm to construct a multi-level feature pyramid and an adaptive convolutional kernel architecture for feature extraction and pest and disease candidate region generation from the multimodal data. It leverages an optimized long short-term memory network combined with parameters such as pest and disease growth cycles to achieve time-series dynamic analysis and deep feature mining. Through a weight-adaptive fusion strategy, the system performs decision-level fusion of the results from the two algorithms to output the final monitoring data. Furthermore, various correlation formulas are designed in each stage of the system to ensure the synergy of parameters in data acquisition, algorithm processing, and other aspects, thereby achieving efficient and accurate pest and disease monitoring and providing technical support for agricultural pest and disease control.

[0005] The invention disclosed in CN120599500A uses a drone equipped with multi-source heterogeneous sensors to collect optical, multispectral, and thermal infrared data in parallel, which is then wirelessly transmitted and stored on the ground. It utilizes an improved YOLO algorithm to construct a multi-level feature pyramid and adaptive convolutional kernels to extract multimodal features and generate candidate regions for pests and diseases. An optimized LSTM network is then combined to analyze temporal dynamic features. Finally, the results are integrated at the decision level through a weighted adaptive fusion strategy, supplemented by multi-stage correlation formulas and coordinated parameters, to achieve efficient and accurate pest and disease monitoring. Although both are pest and disease monitoring systems based on drone platforms, this invention and the invention disclosed in CN120599500A differ significantly in the following key aspects:

[0006] (1) Multimodal data types and microscopic feature detection: The invention with publication number CN120599500A only integrates macroscopic physical feature data such as optical, multispectral and thermal infrared, which belongs to the category of traditional physical characterization technology and cannot obtain molecular-level information of crop diseases. However, this invention innovatively integrates multimodal technology of hyperspectral imaging and Raman spectroscopy, and directly detects the molecular vibrational fingerprint of crop leaves through Raman spectroscopy probe, realizing accurate identification of early microscopic molecular changes of crop diseases, filling the technical gap that existing systems cannot perform early molecular-level detection of diseases.

[0007] (2) Atmospheric Interference and Beam Quality Assurance Mechanism: The invention with publication number CN120599500A did not address issues such as wavefront distortion, beam spread, and signal attenuation caused by atmospheric turbulence during long-distance data transmission from UAVs. Its system is susceptible to atmospheric interference in complex farmland environments, leading to a significant decrease in optical data quality. In contrast, this invention utilizes an adaptive optics system and a closed-loop control of a Shaker-Hartmann wavefront sensor and a MEMS deformable mirror to correct wavefront distortion in real time, significantly improving the clarity of hyperspectral imaging and the intensity of Raman signals, thus ensuring the quality of data acquisition in complex environments.

[0008] (3) Multimodal data fusion architecture and feature fusion depth: The invention with publication number CN120599500A uses an improved YOLO algorithm to construct a multi-level feature pyramid and an adaptive convolutional kernel architecture for feature extraction, and performs time-series analysis through an optimized LSTM. However, it does not clearly define the deep fusion mechanism for multimodal data (optical / multispectral / thermal infrared), relying only on a decision-level fusion strategy, resulting in low reliability of feature association. In contrast, this invention uses a dual-stream deep learning network architecture (3D-CNN for hyperspectral data + 1D-CNN for Raman data), combined with an attention mechanism to dynamically fuse microscopic and macroscopic features. Through feature-level deep fusion, it achieves pixel-level accurate diagnosis with a classification accuracy of over 95%, which is significantly better than the simple decision-level fusion strategy of existing systems. Summary of the Invention

[0009] This invention aims to solve the problems of the prior art mentioned above. It proposes a crop monitoring method and system based on UAV multimodal and adaptive optics. The technical solution of this invention is as follows:

[0010] A crop monitoring method based on UAV multimodal and adaptive optics includes the following steps:

[0011] Step 1: Collect crop canopy data using a drone equipped with a hyperspectral camera and a Raman spectral probe; the hyperspectral camera is used to acquire the spatial-spectral characteristics of the crop canopy, and the Raman spectral probe is used to acquire the molecular vibrational characteristics of the crop leaves;

[0012] Step 2: Use an adaptive optics system to correct wavefront distortion caused by atmospheric turbulence in real time;

[0013] Step 3: Use a deep learning model to fuse hyperspectral and Raman features to achieve crop health status assessment and early diagnosis of diseases and pests;

[0014] Step 4: Construct a spatiotemporal distribution model of crop health status and analyze the migration and accumulation patterns of diseases.

[0015] Furthermore, in step 2, the adaptive optics system includes a wavefront sensor and a deformable mirror, used to measure and correct wavefront distortion caused by atmospheric turbulence in real time, and the wavefront slope formula... Where f is the wavefront slope matrix generated by the microlens focal length of approximately 5 mm, and x i ,y i This represents the actual spot coordinates of the same sub-aperture under wavefront distortion; x ref,i ,y ref,i This represents the ideal spot coordinates of the i-th sub-aperture when there is no distortion.

[0016] Furthermore, in step 3, the deep learning model is a two-stream network, including a 3D-CNN for processing hyperspectral data and a 1D-CNN for processing Raman data. The diagnostic results are output through a feature fusion layer, specifically including:

[0017] 3D-CNN extracts features from hyperspectral data using three-dimensional convolutional kernels, with the kernel size being k. h ×k w ×k d , where k h k w k is the size of the convolution kernel in the spatial dimension. d The kernel size is H×W×B, where H and W are the spatial resolutions and B is the number of bands. The convolution operation formula is:

[0018] Among them W mnp b represents the weights of the 3D convolution kernel. mnp For bias terms; i,j represent the spatial positions of the output feature map, corresponding to the spatial resolution of the hyperspectral data; d represents the spectral positions of the output feature map, corresponding to the band indices of the hyperspectral data; X (i+m)(j+n)p In the diagram, (i+m) and (j+n) represent the spatial positions of the input hyperspectral data. Here, m is the sliding offset of the convolution kernel in the spatial height direction, n is the sliding offset of the convolution kernel in the spatial width direction, and p is the position of the input hyperspectral data in the spectral dimension, corresponding to the sliding offset of the convolution kernel in the spectral dimension.

[0019] 1D-CNN extracts features from Raman spectral data using one-dimensional convolutional kernels of size K. l The Raman spectral data has a size of L×D, where L is the number of sampling points and D is the Raman shift dimension. The convolution formula is:

[0020]

[0021] X (t+m)dThis represents the eigenvalue at the (t+m)th sampling point and the dth Raman shift dimension in the input Raman spectral data; b m This represents the bias parameter at the corresponding position during the convolution process.

[0022] Subsequently, an attention mechanism is used in the feature fusion layer to dynamically integrate the two types of features, and the attention weight α is calculated:

[0023]

[0024] Where score is the feature scoring function, and the final fused feature F fusion For: F fusion =α·F hyperspectral +(1-α)·F Raman During the model training phase, an active learning strategy is introduced to assess sample uncertainty by calculating the model prediction entropy H(p). This represents the contribution of a single category probability to "prediction uncertainty (entropy)". Samples with high entropy values ​​are preferentially selected for manual labeling; a weighted cross-entropy loss function is used to address the imbalance of disease categories. The loss function is: Where w c The class weights are inversely proportional to the number of samples, y c For real labels, To predict probabilities, an active learning loop is performed.

[0025] Furthermore, in step 4, the spatiotemporal distribution model combines multimodal remote sensing data, meteorological data, and soil data to analyze the migration and accumulation patterns of diseases and predict future distribution trends; the specific implementation steps are as follows:

[0026] 1) Multi-source data integration: Acquire UAV hyperspectral imagery and Raman spectral data, and output disease type and severity (level 1-5) through automatic identification model; Simultaneously collect farmland meteorological data (temperature, humidity, rainfall, wind speed) and soil property data (pH, nutrients, texture), and generate environmental covariate layers through interpolation and inversion.

[0027] 2) Feature extraction and quantification: Extract the geometric features (area, perimeter) and spatial aggregation indicators of lesions, and construct composite features by combining hyperspectral red edge slope and Raman characteristic peaks; calculate environmental risk factors (relative humidity cumulative index RHCI, temperature-humidity interaction term THI, soil nutrient deficit index SNDI) to quantify the causes of disease.

[0028] 3) Spatiotemporal dynamic modeling: Geographically weighted regression analysis is used to analyze the spatial non-stationary relationship between diseases and the local environment, and hotspot areas are identified by Moran's index; based on spatiotemporal cubes, LSTM is used to model the cumulative effect of single-point diseases, or ST-GCN is used to capture the propagation path between plots (such as wind-driven diffusion).

[0029] 4) Risk classification and output: Integrate disease severity, environmental factors and vegetation index to construct a comprehensive health index (CHI), and classify it into four levels: healthy (>0.7), sub-healthy (0.5-0.7), risky (0.3-0.5) and high-risk (<0.3); generate dynamic heat maps and decision reports to guide precise prevention and control.

[0030] A crop monitoring method and system based on UAV multimodal and adaptive optics, comprising:

[0031] 1) Multimodal data acquisition module, including a hyperspectral camera and a Raman spectroscopy probe, used to acquire the spatial-spectral characteristics and molecular vibrational characteristics of crop canopy;

[0032] 2) Adaptive optics system module, including wavefront sensor and deformable mirror, for real-time correction of wavefront distortion caused by atmospheric turbulence;

[0033] 3) Data processing module, used to preprocess the acquired hyperspectral and Raman data, including noise removal, spectral normalization and feature extraction;

[0034] 4) Automatic identification module, including a dual-stream deep learning network, integrates hyperspectral and Raman features to output crop health status assessment and pest and disease diagnosis results;

[0035] 5) Spatiotemporal distribution modeling module, used to construct a spatiotemporal distribution model of crop health status and analyze the migration and accumulation patterns of diseases.

[0036] Furthermore, the hyperspectral camera covers the visible to near-infrared spectral range and employs a narrowband filter to improve the spectral resolution of crop disease characteristics. The Raman spectral probe uses a 785nm laser band and is combined with a spectral matching algorithm to identify different types of disease characteristics. The spectral matching algorithm refers to a technique that determines the category or characteristics of an unknown sample by calculating the similarity between an unknown spectrum and known spectra in a reference spectral library. Its core steps include: first, constructing a reference library containing various standard spectra (such as spectra of healthy crops and different diseases); then, preprocessing the unknown spectrum (such as crop disease detection spectra) by denoising and normalization; then, using methods such as Euclidean distance and correlation coefficients to measure the similarity between the two; and finally, determining the category (such as crop health status or disease type) corresponding to the unknown spectrum based on the similarity results, thereby achieving accurate identification.

[0037] Furthermore, the adaptive optics system employs a Shaker-Hartmann wavefront sensor and a MEMS deformable mirror, with the wavefront sensor sampling frequency at 1kHz and the deformable mirror response time at 1ms, dynamically correcting wavefront distortion.

[0038] Furthermore, the data processing module uses Savitzky-Golay filtering to remove hyperspectral noise, employs polynomial fitting for Raman baseline correction, and extracts key features through PCA dimensionality reduction.

[0039] Furthermore, the automatic recognition model employs transfer learning technology, fine-tunes the pre-trained model, and continuously optimizes the model performance through an active learning strategy.

[0040] The advantages and beneficial effects of this invention are as follows:

[0041] 1. This invention innovates multimodal fusion and high-precision diagnostic capabilities: Through multimodal data fusion (such as the synergistic analysis of hyperspectral macroscopic lesion features and Raman microscopic molecular features), it achieves comprehensive diagnostic capabilities for crop diseases.

[0042] This end-to-end precision diagnosis, from "macro-symptom localization" to "micro-causal identification," increases the accuracy of early disease identification (such as latent powdery mildew) to over 95%, significantly outperforming the 70%-85% of existing technologies. This provides reliable technical support for early warning of pests and diseases.

[0043] 2. Dynamic Optical Correction and Adaptability to Complex Environments: This invention introduces an adaptive optics system (Shack-Hartmann wavefront sensor + MEMS deformable mirror) to a UAV platform for the first time. By measuring and correcting wavefront distortion caused by atmospheric turbulence in real time (RMS error corrected from λ / 2 to below λ / 10), it solves the problems of light spot diffusion and signal attenuation caused by long-distance beam transmission (5-30 meters) in traditional UAV hyperspectral and Raman detection. This technology improves the clarity of hyperspectral imaging by 30% and the signal-to-noise ratio of Raman signals by 50%-80%, ensuring adaptability in complex farmland environments (wind speed ≥10m / s, fluctuating light).

[0044] The data quality stability at ±30% breaks through the stringent environmental stability limitations of existing equipment, providing a universal solution for large-scale and rapid monitoring by UAVs.

[0045] 3. Lightweight Intelligent Model and Efficient Decision Support: This invention employs a dual-stream deep learning network (3D-CNN for hyperspectral data and 1D-CNN for Raman data) to fuse multimodal features and introduces an attention mechanism to dynamically optimize feature weights. Combined with transfer learning and active learning strategies, a high-precision, highly generalizable automatic identification model is constructed. In field experiments, this model achieved a disease classification accuracy of over 95%, with an inference speed ≤80ms / frame, supporting real-time processing and decision output from UAVs. Simultaneously, by analyzing the migration and accumulation patterns of diseases through a spatiotemporal distribution model, it provides agricultural sectors with an integrated early warning solution encompassing "disease type - spatial distribution - risk level," facilitating precise pesticide application (reducing pesticide usage by 30%-50%) and optimal resource allocation, resulting in significant socio-economic benefits. Attached Figure Description

[0046] Figure 1 This is a diagram of the crop health monitoring system architecture according to a preferred embodiment of the present invention;

[0047] Figure 2 An adaptive optics system architecture diagram provided for an embodiment of the present invention.

[0048] Figure 3 This is a flowchart of the automatic recognition model training process provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0050] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0051] This invention designs a novel multimodal crop health monitoring sensor system, which innovatively integrates a hyperspectral imaging module and a Raman spectroscopy detection unit into a UAV platform. The system's optical acquisition unit captures the three-dimensional morphological features, surface texture, and reflectance spectral information of the plant canopy using a high-resolution spectroscopic camera. Simultaneously, the laser-induced Raman spectroscopy module uses a specific wavelength excitation source to irradiate leaf tissue, acquiring its molecular vibrational characteristic spectrum. By matching and analyzing this spectrum with a standard disease spectral database, accurate identification of disease types is achieved. The UAV platform's mobile monitoring capabilities support autonomous inspection of large-scale farmland and transmit multi-source sensor data back in real time via a high-speed data transmission link.

[0052] To address the impact of atmospheric turbulence on optical signals, the system integrates a real-time adaptive optical compensation device. This device employs high-speed wavefront detection technology (sampling frequency up to 1kHz) and, in conjunction with a programmable gate array, calculates wavefront distortion compensation parameters in real time, optimizing the laser beam wavefront error from the initial λ / 2 to within λ / 10, significantly improving optical imaging quality and the signal-to-noise ratio of the Raman signal.

[0053] To address the challenges of spectral detection in complex farmland environments, this study proposes a multi-source data fusion method based on deep learning. The designed dual-branch neural network employs a feature encoding-classification decoding architecture, processing visual and spectral features separately to achieve sub-pixel-level disease identification. Hyperspectral imaging provides macroscopic spatial distribution information, while Raman spectroscopy reveals differences in microscopic molecular composition. The synergistic analysis of these two technologies enables the system to distinguish between healthy and diseased tissues with an accuracy exceeding 95%, far surpassing conventional detection techniques.

[0054] The study also established a dynamic crop health assessment model, integrating multi-temporal remote sensing observations, environmental meteorological parameters, and soil characteristic data to construct a spatiotemporal prediction system for disease transmission. This model draws on hydrodynamic principles, comprehensively considering environmental factors such as airflow, temperature and humidity changes, and precipitation distribution to simulate the spread patterns of pathogens under different climatic conditions. Empirical studies show that high temperature and high humidity environments significantly accelerate disease transmission, while drought and low temperature conditions are conducive to epidemic control. This prediction model can provide quantitative evidence for regional agricultural risk management.

[0055] This system integrates cutting-edge technologies such as UAV remote sensing, molecular spectroscopy, intelligent computing, and spatiotemporal modeling to achieve non-destructive, efficient, and accurate monitoring of crop health status. Compared to traditional laboratory testing methods, this system offers advantages such as non-contact operation, full automation, and wide coverage, making it widely applicable for disease monitoring of major food crops. Its innovative multi-source data fusion architecture ensures reliable detection even in complex farmland environments, providing agricultural management departments with a scientific decision-making tool. This technological framework has good scalability and can be applied to cash crops and fruit tree cultivation in the future, providing technical support for the sustainable development of modern agriculture.

[0056] like Figure 1 As shown, the crop health monitoring system provided in this embodiment of the invention includes the following modules:

[0057] S101, a multimodal data acquisition module, is used to obtain the spatial-spectral characteristics and molecular vibrational characteristics of crop canopy;

[0058] S102, Adaptive Optics System Module, is used to correct wavefront distortion caused by atmospheric turbulence in real time;

[0059] S103, the data processing module, is used to preprocess the acquired hyperspectral and Raman data, including noise removal, spectral normalization, and feature extraction.

[0060] S104, Automatic Identification Module, uses a dual-stream deep learning network to automatically identify and diagnose crop health status;

[0061] S105, the spatiotemporal distribution modeling module, is used to construct a spatiotemporal distribution model of crop health status and analyze the migration and accumulation patterns of diseases.

[0062] This invention designs a novel multimodal crop health monitoring sensor system, which innovatively integrates a hyperspectral imaging module and a Raman spectroscopy detection unit into a UAV platform. The system's optical acquisition unit captures the three-dimensional morphological features, surface texture, and reflectance spectral information of the plant canopy using a high-resolution spectroscopic camera. Simultaneously, the laser-induced Raman spectroscopy module uses a specific wavelength excitation source to irradiate leaf tissue, acquiring its molecular vibrational characteristic spectrum. By matching and analyzing this spectrum with a standard disease spectral database, accurate identification of disease types is achieved. The UAV platform's mobile monitoring capabilities support autonomous inspection of large-scale farmland and transmit multi-source sensor data back in real time via a high-speed data transmission link.

[0063] To address the impact of atmospheric turbulence on optical signals, the system integrates a real-time adaptive optical compensation device. This device employs high-speed wavefront detection technology (sampling frequency up to 1kHz) and, in conjunction with a programmable gate array, calculates wavefront distortion compensation parameters in real time, optimizing the laser beam wavefront error from the initial λ / 2 to within λ / 10, significantly improving optical imaging quality and the signal-to-noise ratio of the Raman signal.

[0064] To address the challenges of spectral detection in complex farmland environments, this study proposes a multi-source data fusion method based on deep learning. The designed dual-branch neural network employs a feature encoding-classification decoding architecture, processing visual and spectral features separately to achieve sub-pixel-level disease identification. Hyperspectral imaging provides macroscopic spatial distribution information, while Raman spectroscopy reveals differences in microscopic molecular composition. The synergistic analysis of these two technologies enables the system to distinguish between healthy and diseased tissues with an accuracy exceeding 95%, far surpassing conventional detection techniques.

[0065] The study also established a dynamic crop health assessment model, integrating multi-temporal remote sensing observations, environmental meteorological parameters, and soil characteristic data to construct a spatiotemporal prediction system for disease transmission. This model draws on hydrodynamic principles, comprehensively considering environmental factors such as airflow, temperature and humidity changes, and precipitation distribution to simulate the spread patterns of pathogens under different climatic conditions. Empirical studies show that high temperature and high humidity environments significantly accelerate disease transmission, while drought and low temperature conditions are conducive to epidemic control. This prediction model can provide quantitative evidence for regional agricultural risk management.

[0066] This system integrates cutting-edge technologies such as UAV remote sensing, molecular spectroscopy, intelligent computing, and spatiotemporal modeling to achieve non-destructive, efficient, and accurate monitoring of crop health status. Compared to traditional laboratory testing methods, this system offers advantages such as non-contact operation, full automation, and wide coverage, making it widely applicable for disease monitoring of major food crops. Its innovative multi-source data fusion architecture ensures reliable detection even in complex farmland environments, providing agricultural management departments with a scientific decision-making tool. This technological framework has good scalability and can be applied to cash crops and fruit tree cultivation in the future, providing technical support for the sustainable development of modern agriculture.

[0067] The multimodal data fusion architecture provided in this embodiment of the invention is specifically implemented as follows:

[0068] The system acquires spatial-spectral feature data of crop canopy (such as lesion shape, texture, and spectral reflectance changes) using a hyperspectral camera mounted on a drone, and simultaneously collects molecular vibrational feature data of crop leaves (such as the characteristic peak intensity and position of pathogen metabolites and cell wall components) using a Raman spectroscopy probe. Both types of data are input into a dual-stream deep learning network for processing: the hyperspectral data is processed by a 3D-CNN (three-dimensional convolutional neural network) to extract joint spatial-spectral features, while the Raman data is processed by a 1D-CNN (one-dimensional convolutional neural network) to extract molecular vibrational features. Then, an attention mechanism is used in the feature fusion layer to dynamically integrate the two types of features. The weights of the hyperspectral and Raman features are automatically adjusted according to the disease type and development stage (e.g., early-stage diseases emphasize Raman molecular features, while mid-to-late-stage diseases emphasize hyperspectral macroscopic features). Finally, the system outputs disease classification results (e.g., healthy, powdery mildew, rust, etc.) and severity assessment through the fused high-dimensional feature vector, achieving collaborative analysis of macroscopic lesion localization and microscopic etiology identification, significantly improving detection accuracy and reliability.

[0069] like Figure 2 As shown, the specific implementation process of the adaptive optics system provided in this embodiment of the invention is as follows: The system consists of a Shaker-Hartmann wavefront sensor, a MEMS deformable mirror, and an FPGA control unit. The wavefront sensor acquires the position of the light spot after the incident beam is focused by the microlenses at a high frequency of 1kHz through a 10×10 microlens array. After being recorded by a CMOS sensor, the offset of each sub-aperture light spot relative to the ideal reference position is calculated, and then the offset is calculated using the wavefront slope formula. Where f is the microlens focal length of approximately 5mm, the generated wavefront slope matrix reflects the wavefront distortion distribution caused by atmospheric turbulence in real time. After receiving the wavefront slope matrix, the FPGA control unit calculates the required driving voltage signals for each unit of the MEMS deformable mirror using the region method or the mode method, and sends control commands through a high-speed interface. The deformable mirror dynamically adjusts the local curvature of the mirror surface according to the voltage signal, and completes the wavefront distortion correction in the next sampling cycle, correcting the root mean square error of the wavefront from λ / 2 to λ under turbulent conditions to below λ / 10, significantly improving the beam quality. According to the test, in farmland environments with wind speed ≥10m / s and light fluctuation ±30%, the system can reduce the diameter of the hyperspectral imaging spot from 200μm to 50μm, reduce the spectral reflectance noise by 30%-50%, and improve the Raman spectral signal-to-noise ratio from <5 to >10, ensuring the clarity of hyperspectral imaging and the intensity of Raman signals under long-distance transmission, and providing a stable optical basis for crop disease detection.

[0070] like Figure 3 As shown, the automatic recognition model training process provided in this embodiment of the invention is specifically implemented as follows: First, transfer learning technology is used to initialize the weight parameters of the dual-stream network (3D-CNN for hyperspectral data and 1D-CNN for Raman data) based on the pre-trained ResNet-50 model to alleviate the problem of insufficient small sample data in complex farmland environments; in the feature extraction stage, the 3D-CNN uses a three-dimensional convolutional kernel (size k) h ×k w ×k d , where k h k w k is the size of the convolution kernel in the spatial dimension. d To extract features from hyperspectral data (with H×W×B as the convolution kernel size for the spectral dimension), the convolution operation formula is as follows:

[0071] Among them W mnp b represents the weights of the 3D convolution kernel. mnp The bias term is used; 1D-CNN uses a one-dimensional convolutional kernel (size K). l Feature extraction is performed on Raman spectral data (size L×D, where L is the number of sampling points and D is the Raman shift dimension). The convolution formula is:

[0072]

[0073] Subsequently, an attention mechanism is used in the feature fusion layer to dynamically integrate the two types of features, and the attention weight α is calculated:

[0074]

[0075] Where score is the feature scoring function (e.g., based on feature vector magnitude or MLP network output), and the final fused feature F fusion For: F fusion =α·F hyperspectral +(1-α)·F Raman During the model training phase, an active learning strategy is introduced to assess sample uncertainty by calculating the model prediction entropy H(p). Samples with high entropy values ​​are selected for manual annotation first; a weighted cross-entropy loss function is used to address the imbalance of disease categories. The loss function is as follows: Where w c For class weights (inversely proportional to the number of samples), y c For real labels, To predict probabilities, an active learning loop is performed (sample selection → manual annotation → weighted training → sample selection again… until termination).

[0076] → Output model. The final model achieves a disease classification accuracy of over 95% on the test set, which is significantly better than the single-modal model (hyperspectral about 82%, Raman about 78%), thus completing the construction of high-precision identification and diagnosis capabilities for crop diseases.

[0077] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.

[0078] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A crop monitoring method based on UAV multimodal and adaptive optics, characterized in that, Includes the following steps: Step 1: Collect crop canopy data using a drone equipped with a hyperspectral camera and a Raman spectral probe; the hyperspectral camera is used to acquire the spatial-spectral characteristics of the crop canopy, and the Raman spectral probe is used to acquire the molecular vibrational characteristics of the crop leaves; Step 2: Use an adaptive optics system to correct wavefront distortion caused by atmospheric turbulence in real time; Step 3: Use a deep learning model to fuse hyperspectral and Raman features to achieve crop health status assessment and early diagnosis of diseases and pests; Step 4: Construct a spatiotemporal distribution model of crop health status and analyze the migration and accumulation patterns of diseases.

2. The crop monitoring method based on UAV multimodal and adaptive optics as described in claim 1, characterized in that, In step 2, the adaptive optics system includes a wavefront sensor and a deformable mirror, used to measure and correct wavefront distortion caused by atmospheric turbulence in real time. The wavefront slope formula is... Where f is the wavefront slope matrix generated by the microlens focal length of approximately 5 mm; x i ,y i This represents the actual spot coordinates of the same sub-aperture under wavefront distortion; x ref,i ,y ref,i This represents the ideal spot coordinates of the i-th sub-aperture when there is no distortion.

3. The crop monitoring method based on UAV multimodal and adaptive optics as described in claim 1, characterized in that, In step 3, the deep learning model is a two-stream network, comprising a 3D-CNN for processing hyperspectral data and a 1D-CNN for processing Raman data. The diagnostic results are output through a feature fusion layer, specifically including: 3D-CNN extracts features from hyperspectral data using three-dimensional convolutional kernels, with the kernel size being k. h ×k w ×k d , where k h k w k is the size of the convolution kernel in the spatial dimension. d The kernel size is H×W×B, where H and W are the spatial resolutions and B is the number of bands. The convolution operation formula is: Among them W mnp b represents the weights of the 3D convolution kernel. mnp For bias terms; i,j represent the spatial positions of the output feature map, corresponding to the spatial resolution of the hyperspectral data; d represents the spectral positions of the output feature map, corresponding to the band indices of the hyperspectral data; X (i+m)(j+n)p In the diagram, (i+m) and (j+n) represent the spatial positions of the input hyperspectral data; where m is the sliding offset of the convolution kernel in the spatial height direction, n is the sliding offset of the convolution kernel in the spatial width direction, and p is the position of the input hyperspectral data in the spectral dimension, corresponding to the sliding offset of the convolution kernel in the spectral dimension. 1D-CNN extracts features from Raman spectral data using one-dimensional convolutional kernels of size K. l The Raman spectral data has a size of L×D, where L is the number of sampling points and D is the Raman shift dimension. The convolution formula is: X (t+m)d This represents the eigenvalue at the (t+m)th sampling point and the dth Raman shift dimension in the input Raman spectral data; b m This represents the bias parameter at the corresponding position during the convolution process. Subsequently, an attention mechanism is used in the feature fusion layer to dynamically integrate the two types of features, and the attention weight α is calculated: Where score is the feature scoring function, and the final fused feature F fusion For: F fusion =α·F hyperspectral +(1-α)·F Raman During the model training phase, an active learning strategy is introduced to assess sample uncertainty by calculating the model prediction entropy H(p). This represents the contribution of a single category probability to "prediction uncertainty (entropy)"; samples with high entropy values ​​are preferentially selected for manual labeling; a weighted cross-entropy loss function is used to address the imbalance of disease categories, and the loss function is: Where w c The class weights are inversely proportional to the number of samples, y c For real labels, To predict probabilities, an active learning loop is performed.

4. The crop monitoring method based on UAV multimodal and adaptive optics as described in claim 1, characterized in that, In step 4, the spatiotemporal distribution model combines multimodal remote sensing data, meteorological data, and soil data to analyze the migration and accumulation patterns of diseases and predict future distribution trends; the specific implementation steps are as follows: 1) Multi-source data integration: Acquire UAV hyperspectral images and Raman spectral data, and output the disease type and severity (level 1-5) through automatic identification model; Simultaneously collect farmland meteorological data such as temperature, humidity, rainfall, and wind speed, and soil property data such as pH, nutrients, and texture, and generate environmental covariate layers through interpolation and inversion. 2) Feature extraction and quantification: Extract the geometric features of lesion area, perimeter and spatial aggregation index, and construct composite features by combining hyperspectral red edge slope and Raman characteristic peaks; Calculate environmental risk factors to quantify disease inducing factors, namely the relative humidity cumulative index (RHCI), temperature-humidity interaction term (THI), and soil nutrient deficit index (SNDI). 3) Spatiotemporal dynamic modeling: Geographically weighted regression analysis is used to analyze the spatial non-stationary correlation between diseases and the local environment, and hotspot areas are identified through Moran's index; Based on the spatiotemporal cube, LSTM is used to model the cumulative effect of single-point diseases, or ST-GCN is used to capture the propagation path between plots. 4) Risk classification and output: Integrate disease severity, environmental factors and vegetation index to construct a comprehensive health index (CHI), and classify it into four levels: healthy (>0.7), sub-healthy (0.5-0.7), risky (0.3-0.5) and high-risk (<0.3); generate dynamic heat maps and decision reports to guide precise prevention and control.

5. A crop monitoring system based on UAV multimodal and adaptive optics, characterized in that, include: 1) Multimodal data acquisition module, including a hyperspectral camera and a Raman spectroscopy probe, used to acquire the spatial-spectral characteristics and molecular vibrational characteristics of crop canopy; 2) Adaptive optics system module, including wavefront sensor and deformable mirror, for real-time correction of wavefront distortion caused by atmospheric turbulence; 3) Data processing module, used to preprocess the acquired hyperspectral and Raman data, including noise removal, spectral normalization and feature extraction; 4) Automatic identification module, including a dual-stream deep learning network, integrates hyperspectral and Raman features to output crop health status assessment and pest and disease diagnosis results; 5) Spatiotemporal distribution modeling module, used to construct a spatiotemporal distribution model of crop health status and analyze the migration and accumulation patterns of diseases.

6. The crop monitoring system based on UAV multimodal and adaptive optics as described in claim 5, characterized in that, The hyperspectral camera covers the visible to near-infrared spectral range and uses a narrowband filter to improve the spectral resolution of crop disease characteristics. The Raman spectral probe uses a 785nm laser band and is combined with a spectral matching algorithm to identify different types of disease characteristics. The spectral matching algorithm refers to the technique of determining the category or characteristics of unknown samples by calculating the similarity between unknown spectra and known spectra in a reference spectral library. Its core steps include: first, constructing a reference library containing various standard spectra; then, performing noise reduction and normalization preprocessing on the unknown spectra; then, using Euclidean distance and correlation coefficient methods to measure the similarity between the two; and finally, determining the category corresponding to the unknown spectra based on the similarity results, thereby achieving accurate identification.

7. The crop monitoring system based on UAV multimodal and adaptive optics as described in claim 6, characterized in that, The adaptive optics system employs a Shaker-Hartmann wavefront sensor and a MEMS deformable mirror. The wavefront sensor has a sampling frequency of 1 kHz, and the deformable mirror has a response time of 1 ms, dynamically correcting wavefront distortion.

8. The crop monitoring system based on UAV multimodal and adaptive optics as described in claim 5, characterized in that, The data processing module uses Savitzky-Golay filtering to remove hyperspectral noise, employs polynomial fitting for Raman baseline correction, and extracts key features through PCA dimensionality reduction.

9. The crop monitoring system based on UAV multimodal and adaptive optics as described in claim 8, characterized in that, The automatic recognition model employs transfer learning technology, fine-tunes the pre-trained model, and continuously optimizes the model performance through an active learning strategy.

Images