A cotton disease and pest grade identification method and system based on spectral and laser radar images

Abstract

The application discloses a cotton disease and pest grade recognition method and system based on spectrum and laser radar images, and belongs to the technical field of agricultural remote sensing. In view of the problems that single spectrum monitoring is greatly disturbed by light and is difficult to capture three-dimensional features such as leaf curling, a multi-modal collaborative perception fusion scheme is provided. The method synchronously collects spectrum images and laser radar data of a cotton canopy by using a UAV. After pretreatment and space-time registration, a multi-branch feature extraction network is constructed to extract fine spectrum features, spectrum image texture features and CHM data of the laser radar respectively. By designing a cross-modal attention module, the complementary weights of different modal features are dynamically calculated and weighted fusion is performed. The fused high-dimensional joint features are input into the constructed RCG-Net model, and an end-to-end cotton disease and pest damage grade is output, so that the disease and pest grade of the multi-scale cotton plant remote sensing data of the UAV is accurately recognized, and technical support is provided for disease and pest monitoring in intelligent agriculture.

Description

Technical Field

[0001] This invention relates to the field of agricultural remote sensing technology, specifically to a method and system for identifying the severity of cotton pests and diseases based on spectral and lidar images. Background Technology

[0002] Currently, cotton pest and disease monitoring mainly relies on manual observation or single remote sensing data. However, in existing technologies, single-spectral remote sensing struggles to distinguish between "same-spectral objects and different objects," resulting in low monitoring efficiency. Furthermore, single-spectral remote sensing is easily affected by factors such as ambient light, shadow, and temperature, making it difficult to capture three-dimensional features such as color differences and leaf curling caused by Verticillium wilt and aphids. While research on estimating the threat level of cotton pests and diseases under single laboratory conditions is relatively mature, corresponding research on diagnosing cotton pests and diseases in field environments is still lacking. Moreover, there is a scarcity of combining UAV spectral and lidar multi-source remote sensing data detection. Existing technologies lack effective fusion of multi-source heterogeneous data and strategies to improve the monitoring accuracy of different pests and diseases.

[0003] Currently, existing research has focused on determining the severity of cotton pests and diseases using cotton canopy spectral curves, spectral images, and lidar data. Canopy spectral curves reflect the growth status of cotton and its spectral response characteristics under pest and disease stress. Furthermore, since pests and diseases also cause apparent changes in cotton canopy images, canopy texture features can effectively assist models in identifying pest and disease severity levels. In addition, lidar data further supplements the spatial structural features of cotton, thus effectively improving the accuracy of pest and disease severity identification. However, few studies have achieved effective fusion of these three modalities. Moreover, due to the massive volume of multi-source data, failure to accurately select key features can easily lead to a surge in feature space dimensions, increasing the risk of model overfitting and causing a decrease in recognition accuracy.

[0004] Therefore, how to achieve rapid and accurate identification of the occurrence level of cotton pests and diseases has become a key technical problem that urgently needs to be solved in related fields. To address this problem, this application proposes a cotton pest and disease level identification method based on multimodal data fusion. By complementing hyperspectral, multispectral, and lidar data, and combining multi-branch feature extraction and cross-modal attention fusion, an RCG-Net is constructed to achieve end-to-end identification, overcoming interference from light, soil, and shadow, and capturing structural features such as leaf curling and plant dwarfing. Summary of the Invention

[0005] In view of this, the present invention provides a real-time identification system and device for cotton pests and diseases based on spectral features and lidar, so as to realize a method for identifying the level of cotton pests and diseases based on spectral features and lidar.

[0006] To achieve the above objectives, the present invention provides the following steps: 1. A method for identifying cotton pest and disease severity levels based on spectral and lidar imagery, comprising the following steps: S1. Multi-source data acquisition and processing: Using a drone platform equipped with a hyperspectral camera, a multispectral camera and a lidar camera (LiDAR), hyperspectral images, multispectral images and lidar data of the cotton canopy are acquired simultaneously; at the same time, the cotton pest and disease situation at ground sampling points is collected as ground value labels, including the degree of cotton aphid infestation, the degree of red spider mite damage and the degree of Verticillium wilt infestation, and a priori dataset is constructed. S2. Data Preprocessing and Registration: Radiometric calibration and reflectance conversion are performed on hyperspectral and multispectral images. Savitzky-Golay smoothing is used to smooth hyperspectral and multispectral images. Noise reduction, normalization, and canopy height model (CHM) generation are performed on lidar data. Spatiotemporal registration of multi-source data is performed using geospatial information (POS) to construct a multimodal dataset. S3. Multi-source feature calculation and extraction: Construct a multi-branch feature extraction network. The hyperspectral branch extracts the fine spectral features of the hyperspectral image, the multispectral image branch extracts the spectral features, leaf texture and color features of the multispectral image, and the structural branch extracts the canopy height, curl and topological structure features of the lidar data. S4. Cross-modal feature fusion and recognition: Design a cross-modal attention module to spatially align LiDAR data and spectral features at the same scale. Calculate complementary weights of different modal features through the cross-modal attention module. Input the weighted fused features of the aligned multimodal features into a fully connected layer to generate a high-dimensional joint feature representation. S5. Pest and Disease Level Identification: Construct an RCG-Net model and train it using a multimodal dataset. Input the high-dimensional joint feature representation into the fully connected layer of the RCG-Net model and output the cotton pest and disease level identification results for the current monitoring area through a mapping function.

[0007] In step S1, the prior dataset includes spectral images, lidar images and cotton ground pest and disease data. The OSAVI vegetation index is used to remove soil and shadow background, and the vegetation index (OSAVI) is constructed by extracting RE bands and NIR bands from the multispectral bands. In step S1, the specific method for constructing the prior dataset and vegetation indices is as follows: The Optimized Soil-Adjusted Vegetation Index (OSAVI) was used to remove the influence of soil background and shadow on the cotton canopy spectrum, and the red and near-infrared bands (NIR) in the multispectral images were extracted to construct spectral indices. The formula for calculating OSAVI is: ; Wherein, NIR is the reflectance in the near-infrared band, and Red is the reflectance in the red band.

[0008] In step S2, the specific steps of data preprocessing include: Diffuse plate correction: Reflectance correction of hyperspectral images is performed using a standard diffuse plate. The correction model is as follows: ; In the formula, S1 is the hyperspectral image radiance value, S2 is the standard diffuse reflectance, and S3 is the standard diffuse reflectance value. Spectral data: Savitzky-Golay smoothing is used to reduce noise interference in the spectral imagery, resulting in the original spectral data; LiDAR data: The original point cloud data is classified using the Triangular Irregular Network (TIN) algorithm to distinguish between ground points and non-ground points; the ground points are interpolated using the TIN interpolation algorithm to generate a digital elevation model (DEM); the non-ground points are interpolated using the Kriging interpolation algorithm to generate a digital surface model (DSM); and the canopy height model (CHM) is generated through the difference operation between the DSM and the DEM (DSM-DEM).

[0009] In step S3, the specific contents of the multi-branch feature extraction network and the multi-temporal vegetation index are as follows: Multi-temporal vegetation index calculation: Based on multi-temporal spectral images, a set of spectral features is calculated, including the Normalized Difference Vegetation Index (NDVI), Normalized Difference Red Edge Index (NDRE), Difference Vegetation Index (DVI), Enhanced Vegetation Index (EVI), and Structure Insensitive Pigment Index (SIPI). In texture feature extraction: texture features of images are extracted based on gray-level co-occurrence matrix (GLCM). Texture features include at least: energy, contrast, entropy, homogeneity, and correlation. In the hyperspectral feature extraction branch: a hybrid architecture of one-dimensional convolutional neural network (1D-CNN) is adopted, which utilizes depthwise separable convolution to extract spectral features between bands and capture the specific absorption peak changes caused by aphid damage; In the multispectral feature extraction branch, a two-dimensional residual network architecture is adopted for multispectral images. After the image input is processed by convolution and max pooling, it enters the stacked two-dimensional residual module. In order to enhance the expression of lesion and texture features, an ECA attention module is introduced after the residual module to realize the adaptive adjustment of channel weights. In the structural branch, the spatial neighborhood information of the CHM of the LiDAR data is aggregated using graph convolutional neural network (GCN) to capture the leaf curling and plant dwarfing characteristics caused by cotton aphids or cotton spider mites, thus making up for the deficiencies of two-dimensional images in structural representation.

[0010] In step S4, the specific method for cross-modal feature fusion is as follows: Feature Mapping: A cross-modal attention layer is designed to map hyperspectral features, multispectral features, and LiDAR structural features to feature labels of a unified dimension. Using the key, value, and query matrix in the attention mechanism, the importance weights of features from different sensor data are dynamically calculated. Based on the calculated weights, modal features sensitive to pests and diseases are adaptively enhanced, and the weights of LiDAR features and structural features are increased to achieve complementary enhancement of different modal features.

[0011] The multi-branch heterogeneous residual network RCG-Net model constructed in step S5 has a specific network architecture that includes parallel spectral branches, texture branches, and structural branches. A cotton pest and disease severity identification system based on spectral and lidar imagery, applied to the aforementioned cotton pest and disease severity identification method based on spectral and lidar imagery, includes: Data acquisition module: used to acquire hyperspectral images, multispectral images and lidar data of the cotton canopy, and to obtain the corresponding ground pest and disease true value labels; Data processing and registration module: used to perform radiometric calibration, reflectivity conversion and CHM calculation on the multi-source images, and perform spatiotemporal registration to obtain a multimodal dataset; Multi-source feature extraction module: Employs a multi-branch network to extract fine spectral features, texture and color features, and topological features from the multimodal dataset; Cross-modal feature fusion module: used to calculate the feature weights of each modality through the cross-modal attention module, perform spatial alignment and weighted fusion, and generate a high-dimensional joint feature representation; Model training and recognition module: Construct and train the RCG-Net model, input the high-dimensional joint feature representation into the model, and output the cotton pest and disease level.

[0012] A computing platform includes at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a cotton pest and disease level identification method based on spectral and lidar imagery as described above.

[0013] A machine-readable storage medium is characterized in that it stores executable instructions, which, when executed, cause the machine to perform a cotton pest and disease level identification system based on spectral and lidar images as described above.

[0014] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a method and apparatus for real-time identification of cotton pest and disease images based on multimodal features, such as... Figure 1 As shown, it has the following beneficial effects: This invention integrates hyperspectral, multispectral, and LiDAR data, overcoming the limitations of traditional single-spectral monitoring technologies which are susceptible to interference from lighting conditions, soil background, and shadows. Based on hyperspectral and multispectral data, it captures the subtle spectral differences exhibited by cotton under various pest and disease stresses. By introducing the OSAVI vegetation index, it effectively eliminates soil background noise. Combined with CHM data obtained from LiDAR, it can accurately capture the physical structural changes caused by pests and diseases, such as leaf curling and plant dwarfing, significantly improving the accuracy of identifying early-stage and mild pests and diseases. This invention innovatively designs a multi-branch feature extraction network that includes a spectral branch (1D-CNN), a multispectral image branch (ResNeXt-CBAM-MSFE), and a structural branch (GCN). Combined with a cross-modal attention module, it can adaptively adjust the weights of each modality feature according to different environmental conditions, effectively solving the feature conflict problem in multi-source data fusion and achieving robust discrimination of the damage level of cotton pests and diseases. Compared with existing single-spectrum monitoring methods, this application can effectively overcome the interference of light, soil, and shadow, while capturing three-dimensional structural features such as leaf curling and plant dwarfing. It has higher identification accuracy and stronger robustness, and can realize accurate and rapid identification of cotton pest and disease levels in field environments, and has the prospect of industrial application. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a general technical roadmap of the method of the present invention; Figure 2 Model architecture diagram; 256*256 multispectral data plot; Figure 3 The graph shows the hyperspectral reflectance after SG processing. Figure 4 This is a graph of LiDAR data; Figure 5 The diagram shows the RCG-Net model. Figure 6 Diagram of CBAM module; Figure 7 MSFE module diagram; Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] A method for identifying cotton pest and disease severity levels based on spectral and lidar imagery, comprising the following steps: S1. Multi-source data acquisition and processing: Using a drone platform equipped with a hyperspectral camera, a multispectral camera and a lidar camera (LiDAR), hyperspectral images, multispectral images and lidar data of the cotton canopy are acquired simultaneously; at the same time, the cotton pest and disease situation at ground sampling points is collected as ground value labels, including the degree of cotton aphid infestation, the degree of red spider mite damage and the degree of Verticillium wilt infestation, and a priori dataset is constructed. S2. Data Preprocessing and Registration: Radiometric calibration and spectral reflectance conversion are performed on hyperspectral and multispectral images; Savitzky-Golay smoothing is used to smooth hyperspectral and multispectral images; denoising, normalization, and canopy height model (CHM) generation are performed on lidar data; spatiotemporal registration of multi-source data is performed using geospatial information (POS) to construct a multimodal dataset; S3. Multi-source feature calculation and extraction: Construct a multi-branch feature extraction network. The hyperspectral branch uses a one-dimensional convolutional neural network (1D-CNN) to extract fine spectral features of the hyperspectral spectrum; the multispectral image branch uses ResNeXt-CBAM-MSFE to extract leaf texture and color features from visible light / multispectral images; the structure branch uses a graph convolutional neural network (GCN) to process lidar data and extract canopy height, curl, and topological structure features. S4. Cross-modal feature fusion and recognition: Design a cross-modal attention module to spatially align lidar features and spectral features at the same scale. Calculate the complementary weights of different modal features through the cross-modal attention module. Input the weighted fused features of the aligned multimodal features into a fully connected layer to generate a high-dimensional joint feature representation. S5. Pest and Disease Level Identification: Construct an RCG-Net model and train it using the multimodal dataset. Input the high-dimensional joint feature representation into the fully connected layer of the RCG-Net model and output the cotton pest and disease level identification results for the current monitoring area through a mapping function.

[0019] In step S1, the prior dataset includes spectral images, lidar images and cotton ground pest and disease data. The OSAVI vegetation index is used to remove soil and shadow background, and the RE band and NIR band in the multispectral band are extracted to construct the vegetation index (OSAVI). In step S1, the specific method for constructing the prior dataset and vegetation indices is as follows: The Optimized Soil-Adjusted Vegetation Index (OSAVI) was used to remove the influence of soil background and shadow on the cotton canopy spectrum, and the red and near-infrared bands (NIR) in the multispectral images were extracted to construct spectral indices.

[0020] The formula for calculating OSAVI is as follows: ; Wherein, NIR is the reflectance in the near-infrared band, and Red is the reflectance in the red band.

[0021] In step S2, the specific steps of the data preprocessing include: Diffuse plate correction: Reflectance correction of hyperspectral images is performed using a standard diffuse plate. The correction model is as follows: ; In the formula, S1 is the hyperspectral image radiance value, S2 is the standard diffuse reflectance, and S3 is the standard diffuse reflectance value; Spectral data: Savitzky-Golay smoothing was used to reduce noise interference in the spectral images, resulting in the original spectral data; LiDAR data: The original point cloud data is classified using the Triangular Irregular Network (TIN) algorithm to distinguish between ground points and non-ground points; the ground points are interpolated using the TIN interpolation algorithm to generate a digital elevation model (DEM); the non-ground points are interpolated using the Kriging interpolation algorithm to generate a digital surface model (DSM); and the canopy height model (CHM) is generated through the difference operation between the DSM and the DEM (DSM-DEM).

[0022] In step S3, the specific contents of the multi-branch feature extraction network and the multi-temporal vegetation index are as follows: Multi-temporal vegetation index calculation: Based on multi-temporal spectral images, a set of spectral features is calculated, including the Normalized Difference Vegetation Index (NDVI), Normalized Difference Red Edge Index (NDRE), Difference Vegetation Index (DVI), Enhanced Vegetation Index (EVI), and Structure Insensitive Pigment Index (SIPI). Texture feature extraction: Based on the gray-level co-occurrence matrix (GLCM), texture features of the image are extracted. Texture features include: energy, contrast, entropy, homogeneity and correlation. Information entropy is a random indicator that measures the amount of information in an image; it reflects the non-uniformity and complexity of the image texture. ; in, p i The image contains gray levels i The probability, N It is the total number of pixels in the image; Contrast reflects the sharpness of an image and the depth of texture grooves; ; in, P ( i , j ) is the position in the gray-level co-occurrence matrix ( i , j The co-occurrence probability on ); energy: ; entropy: ; Correlation: ; Inverse moment: ; in,( i, j ) is the position in the gray-level co-occurrence matrix ( i, j The co-occurrence probability on ); In the hyperspectral feature extraction branch: In the spectral branch, a one-dimensional convolutional neural network (1D-CNN) architecture is adopted. The spectral features between bands are extracted using depthwise separable convolution. The changes in specific absorption peaks caused by aphids, spider mites, and Verticillium wilt are captured. From the perspective of biophysical mechanism analysis, after cotton is damaged by cotton aphids, its leaf mesophyll cell structure is destroyed, its chlorophyll content is sharply reduced, and its canopy water is lost, which will produce a clear stress response in a specific spectral range. In the multispectral image branch, a two-dimensional residual network architecture is adopted for spectral images and texture features. After the image input is processed by convolution and max pooling, it enters the stacked two-dimensional residual module. To enhance the expression of lesions and texture features, an ECA attention module is introduced after the residual module to achieve adaptive adjustment of channel weights. In the structural branch, a graph convolutional neural network (GCN) is used to aggregate the spatial neighborhood information of the CHM to capture the leaf curling and plant dwarfing characteristics caused by cotton aphids or cotton spider mites, thus making up for the shortcomings of two-dimensional images in structural representation.

[0023] In step S4, the specific method for cross-modal feature fusion is as follows: An RCG-Net model is constructed, where feature mapping includes mapping hyperspectral features, texture features, and LiDAR structural features to feature labels of a unified dimension. A cross-modal attention layer is designed, utilizing the key, value, and query matrices in the attention mechanism to dynamically calculate the feature importance weights from different sensor data. Based on the calculated weights, modal features sensitive to pests and diseases are adaptively enhanced. When poor lighting conditions lead to a decrease in the signal-to-noise ratio of spectral features, the weights of LiDAR features and structural features are adaptively increased to achieve complementary enhancement of different modal features. At the feature-level fusion level, the weighted fusion calculation formula is as follows, where... Let be the final fused feature matrix, where For the final high-dimensional joint feature representation; , , These are the globally adaptive weight coefficients for the hyperspectral, multispectral, and structural features output by the cross-modal attention module, respectively. For the first i Weights of each hyperspectral feature channel These are the corresponding hyperspectral characteristic values; For the first j Weights of each multispectral feature channel, These are the corresponding multispectral characteristic values; The topological features of the canopy are extracted from lidar data. The cross-modal attention module dynamically evaluates the importance weights of spectral features, texture features, and structural features in the current scene by calculating query, key, and value matrices.

[0024] The multi-branch heterogeneous residual network RCG-Net constructed in step S5 has a specific network architecture that includes parallel hyperspectral branches, multispectral image branches, and structural branches. A cotton pest and disease severity identification system based on spectral and lidar imagery is applied to the aforementioned cotton pest and disease severity identification method based on spectral and lidar imagery, comprising: Data acquisition module: used to acquire hyperspectral images, multispectral images and lidar data of the cotton canopy, and obtain the corresponding ground truth labels; Data processing and registration module: used to perform radiometric calibration, reflectivity conversion and CHM calculation on the multi-source images, and perform spatiotemporal registration to obtain a multimodal dataset; Multi-source feature extraction module: Employs a multi-branch network to extract fine spectral features, texture and color features, and topological features from the multimodal dataset; Cross-modal feature fusion module: used to calculate the feature weights of each modality through the cross-modal attention module, perform spatial alignment and weighted fusion, and generate a high-dimensional joint feature representation; Model training and recognition module: Construct and train the RCG-Net model, input the high-dimensional joint feature representation into the model, and output the cotton pest and disease level.

[0025] The system can identify the damage caused by pests and diseases in a single image captured by a drone in real time, and clarify the types of pests and diseases and the extent of infection.

[0026] Multimodal feature weights are calculated, and features with different influence weights are set by analyzing the dataset to ensure recognition accuracy and efficiency under various pest and disease image conditions. Simultaneously, the weights of each modality or model are dynamically adjusted based on real-time information such as user feedback and system performance evaluation. A correlation matrix between feature performance and local scene context is established. During real-time inference, if a specific feature subset is detected to exhibit excellent discriminative power in the current specific scene, a dynamic attention mechanism is triggered to directionally increase the weight coefficient of that modality or feature node, ensuring that the model achieves an optimal balance between global recognition accuracy and computational efficiency under multi-scale pest and disease image conditions.

[0027] The cross-modal feature fusion and recognition module aims to eliminate the heterogeneity between multi-source remote sensing data and achieve deep synergy between cotton field phenotypes and pest and disease characteristics. First, through spatial coordinate transformation and interpolation algorithms, the three-dimensional structural features extracted by lidar, plant height, number of fruiting branches, etc., are spatially aligned with the spectral features extracted by hyperspectral imaging at the same scale. Then, a feature-level fusion network based on an attention mechanism is introduced. Combining the dynamic weight vector output from the previous module, the aligned multimodal features are weighted and concatenated or subjected to inner product operations to generate a high-dimensional joint feature representation. Finally, the joint features are input into a fully connected layer or classifier, and the cotton health status of the current monitoring area is output through a mapping function, completing end-to-end multimodal pest and disease identification.

[0028] A computing platform includes: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a cotton pest and disease level identification method based on spectral and lidar imagery.

[0029] A machine-readable storage medium storing executable instructions that, when executed, cause a machine to perform the aforementioned cotton pest and disease level identification system based on spectral and lidar imagery.

[0030] Example 1: RCG-Net Model Construction: This invention discloses a method for identifying the severity of cotton pests and diseases based on spectral and lidar imagery, such as... Figure 1 As shown, the specific steps are as follows: (1) Multimodal data acquisition and preprocessing; (2) Dynamic augmentation of training samples; (3) Construction of a multimodal feature fusion network model; (4) Model training and optimization strategies; (5) Identification of disease severity levels.

[0031] Specifically, in step (1) multimodal data acquisition and preprocessing, this method uses an unmanned aerial vehicle platform to collect data of three different modes of cotton canopy: multispectral images, hyperspectral data and CHM data.

[0032] Multispectral imagery: Contains information from multiple spectral bands, along with various vegetation indices calculated based on these bands, forming a high-dimensional image data input. Texture features: Texture features of the NIR bands in the multispectral imagery are extracted using a gray-level co-occurrence matrix, with its spatial scale aligned with the multispectral imagery. Hyperspectral data: Stored in tabular form (Excel file), with each row representing a sample point and containing multiple consecutive narrow-band reflectance values, accurately reflecting the response characteristics of vegetation under different spectra. CHM is a single-channel image; its pixel values ​​represent the altitude of the corresponding ground point, effectively reflecting the canopy height and growth of cotton plants.

[0033] In the preprocessing stage, all image data were uniformly resized to 256x256 pixels. By reading TIF format images, pixel values ​​were normalized and converted to Tensor format. Simultaneously, hyperspectral data was matched with image samples based on their IDs. Finally, the entire dataset was divided into training, validation, and test sets using stratified sampling in a 7:1.5:1.5 ratio to ensure a balanced distribution of samples representing different disease severity levels across the datasets.

[0034] Specifically, in step (2) of dynamic augmentation of training samples, to improve the generalization ability of the model and prevent overfitting, this invention designs an online data augmentation strategy. In each batch of model training, a series of random transformation operations are performed on the input training set multispectral images and CHM images with a 50% probability, such as... Figure 2 As shown, it includes: Random Flip: Randomly flips the image horizontally or vertically.

[0035] Random rotation: Rotate the image at random angles within the range of 0 to 360°.

[0036] Random cropping and scaling: Randomly crop 80% of the area from the image and restore it to the original 256x256 size to simulate observation at different scales.

[0037] Random erasure: Randomly select a region on the image (occupying 20% ​​of the smallest side) and set its pixel value to zero to enhance the model's robustness to target occlusion.

[0038] Specifically, in step (3) of constructing the multimodal feature fusion network model, this invention employs a deep learning network called RCG-Net. The core idea of ​​this network is to design a dedicated feature extractor for each data modality, and then intelligently weight and fuse the extracted high-level features through an attention fusion module to ultimately achieve accurate classification. This network mainly consists of three parallel feature extraction branches and an attention fusion classification module.

[0039] Specifically, before field data collection, healthy cotton plants with typical traits were marked in each experimental plot, and RTK real-time differential positioning was used to locate each sampling point and obtain absolute location information. During measurement, experimental personnel entered the cotton field to manually collect data at each sampling point, collecting data on different pests and diseases. The pest and disease data collection standards were based on national standards. The cotton variety was Xinluzao 62, a herbaceous resistant cotton, with a planting density of 12,000 plants / mu, and the soil type was loam.

[0040] In a specific example, the hyperspectral data acquired in this invention were all captured by a Rikola hyperspectral camera, and ground data analysis was performed on different sampling points selected in the cotton. The hyperspectral data acquisition was performed using a DJIM 600 Pro drone, with a Ronin-MX gimbal carrying a SENOP Rikola frame-type hyperspectral imager (Finland). This imager has a wavelength range of 500nm-900nm and carries a relative radiance sensor. This invention sets the spectral resolution to 9nm, acquiring hyperspectral images of 45 bands and 1010×1010 pixels. During the drone acquisition process, the hyperspectral imager lens was kept vertically downwards, the flight altitude was set to 50m, and the flight speed was 3.9m / s.

[0041] The multispectral data was captured using a DJI Mavic 3 Multispectral Edition camera equipped with a four-channel multispectral lens (Red, Green, Nir, and RedEdge). This study used this device to capture 12m multispectral data and multispectral data for various growth stages. The lateral and lateral overlap rates were both 80%. All data underwent radiometric correction using 64% diffuse reflectance. Combined with the local solar azimuth angle, spectral data were collected between 12:00 and 2:00 PM on the same day. Cotton irrigation was controlled during data acquisition to avoid spectral anomalies caused by moisture reflection.

[0042] This study used DJI's L2 LiDAR to acquire cotton LiDAR data, taking into account the high-density cotton planting characteristics of the Letuyi experimental field. The imaging parameters were: 5 echoes, laser pulse emission frequency of 240 Hz, and flight altitude of 12 m, with distortion correction selected to ensure modeling accuracy.

[0043] In a specific instance, such as Figure 5 As shown, the first branch is the multispectral feature extraction branch, ResNeXt-CBAM-MSFE, which is responsible for processing multispectral imagery. Its basic architecture is the ResNeXt50-32x4d model, with depth optimization. First, its initial convolutional layers are modified to accept multi-channel input composed of multiple spectral bands and vegetation indices. Second, after the three core residual stages (layer 1, layer 2, and layer 3) of the model, convolutional block attention modules (CBAMs) are embedded, such as... Figure 6 As shown, the CBAM module can sequentially apply attention weights to the feature maps in both channel and spatial dimensions, enabling the network to adaptively focus on the spectral band combinations most sensitive to pest and disease stress and the most critical lesion spatial regions. Finally, the last residual stage of the network is replaced with the multi-scale feature extraction module MSFE_Block, as shown. Figure 7As shown, this module uses four parallel convolutional branches with different receptive fields (1x1 convolution, 3x3 convolution, stacked 3x3 convolution, and global pooling) to simultaneously capture disease features at different scales, thereby enhancing the ability to identify disease and pest infestation areas with different morphologies.

[0044] ; ; ; The second branch is the hyperspectral feature extraction branch (1D_CNN), which is a one-dimensional convolutional neural network specifically designed to process hyperspectral reflectance data in one-dimensional vector form, such as... Figure 3 As shown. By stacking multiple Conv1d, BatchNorm1d, and MaxPool1d layers, this network can effectively mine subtle changes and feature combinations in hyperspectral curves, extracting deep spectral features that can distinguish cotton plants with different health conditions.

[0045] The third branch is the CHM feature extraction branch (GCN), which is used to extract structural features from CHM images representing plant height, such as... Figure 4 As shown, this branch innovatively combines Convolutional Neural Networks (CNNs) and Graph Convolutional Networks (GCNs). First, a shallow CNN network performs preliminary feature extraction on CHM images. Then, the extracted feature map is treated as a graph structure, where each pixel is a node. By calculating the similarity between features of nodes in the feature map, an adjacency matrix is ​​dynamically constructed. Next, using graph convolution operations, the network can aggregate information from spatially non-adjacent but feature-similar nodes. This design enables the model to capture growth structure information such as non-local abnormal undulations or dwarfing of the plant canopy caused by pests and diseases.

[0046] At the end of the network is the attention feature fusion and classification module. This module uses 256-dimensional feature vectors from the three branches mentioned above. , , They are stacked together. The formula is... .

[0047] An attention layer learns from these three feature vectors, dynamically assigning an attention weight to each modality. This weight reflects the contribution of each data modality to the final judgment given the current input sample. For example, when spectral features are significant, the model will assign... and Higher weights. A final fused feature vector is obtained by weighted summation of the feature vectors and their corresponding attention weights. Finally, this fused vector is passed through a classifier with Dropout, which outputs the probability of four severity levels of cotton pest and disease damage.

[0048] Specifically, in step (4) model training and optimization strategy, the present invention adopts a comprehensive optimization scheme to ensure the efficiency and stability of the model.

[0049] Loss function and optimizer: The cross-entropy loss function is used as the objective function, and the AdamW optimizer is selected to update the model parameters. This optimizer integrates a weight decay mechanism to effectively prevent the model from overfitting.

[0050] Learning rate scheduling: A cosine annealing learning rate scheduler is used. This strategy causes the learning rate to decrease in a cosine function manner during training, enabling rapid convergence in the early stages of training and fine-tuning of model parameters with a smaller learning rate in the later stages. This helps the model escape local optima and find better solutions.

[0051] Sample Balance: To address the imbalance in the number of samples across different disease levels in the dataset, a weighted random sampler is used during training. This sampler calculates sampling weights based on the reciprocal of the number of samples for each category, increasing the probability of the less common category (usually severe disease) being selected in each training cycle, thus ensuring that the model achieves consistent learning performance across all disease levels.

[0052] The software and hardware environment configurations used in this experiment are shown in Table 1. Table 1 Model Operating Environment GPU NVIDIA GeForce RTX 4090 24G CPU Intel(R) Xeon(R) Gold 6248R CPU@3.0GHz operating system Windows 10 CUDA 12.9 Design Language 3.9 Deep learning framework PyTorch Specifically, in step (5) of identifying the severity level of pests and diseases, the model ultimately outputs a four-class classification result, corresponding to four levels of pests and diseases: Level 0 (healthy), Level 1 (mild), Level 2 (moderate), and Level 3 (severe). This level classification corresponds to severity_0 to severity_3 in the input data. During the model inference stage, the multimodal data of the cotton sample to be tested is input, and the model can automatically output its corresponding severity level of the disease.

[0053] In summary, this invention constructs a multimodal deep learning framework that integrates multiple customized feature extractors (ResNeXt-CBAM-MSFE, 1D-CNN, GCN), and combines an attention-weighted fusion mechanism and advanced training optimization strategies. This framework can fully utilize the complementary information from multispectral, hyperspectral, and CHM data to achieve high-precision and automated identification of the severity of cotton pests and diseases.

[0054] This invention also discloses a cotton pest and disease severity identification system based on spectral and lidar imagery, comprising: Data acquisition module: used to acquire hyperspectral images, multispectral images and lidar data of the cotton canopy, and obtain the corresponding ground truth labels; Data processing and registration module: used to perform radiometric calibration, reflectivity conversion and CHM calculation on the multi-source images, and perform spatiotemporal registration to obtain a multimodal dataset; Multi-source feature extraction module: Employs a multi-branch network to extract fine spectral features, texture and color features, and topological features from the multimodal dataset; Cross-modal feature fusion module: used to calculate the feature weights of each modality through the cross-modal attention module, perform spatial alignment and weighted fusion, and generate a high-dimensional joint feature representation; Model training and recognition module: Construct and train the RCG-Net model, input the high-dimensional joint feature representation into the model, and output the cotton pest and disease level.

[0055] An electronic device includes at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the above-described multimodal image recognition method for cotton pests and diseases based on multi-level feature fusion.

[0056] Electronic devices may include, but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smartphones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronics devices, etc.

[0057] A machine-readable storage medium storing executable instructions, which, when executed, cause the machine to perform the aforementioned multimodal image recognition method for cotton pests and diseases based on multi-level feature fusion.

[0058] In summary, the cotton pest and disease severity identification method and system based on spectral and lidar imagery proposed in this invention provides an effective multimodal collaborative sensing solution for crop phenotypic monitoring and pest and disease diagnosis in complex field environments. This invention achieves adaptive fusion of multi-source heterogeneous data in an end-to-end model within the field of agricultural remote sensing, effectively overcoming the challenges of traditional single-spectrum data being susceptible to interference from complex lighting and failing to capture distortions in the three-dimensional physical structure of plants. Furthermore, it has significant practical application and reference value for smart agriculture precision plant protection scenarios requiring high robustness and high-precision identification of pest and disease infestation levels.

[0059] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the invention; furthermore, those skilled in the art will recognize that, based on the ideas of the invention, there will be changes in specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the invention.

Claims

1. A method for identifying cotton pest and disease severity levels based on spectral and lidar imagery, characterized in that, The method includes the following steps: S1. Multi-source data acquisition and processing: Using a drone platform equipped with a hyperspectral camera, a multispectral camera and a lidar camera (LiDAR), hyperspectral images, multispectral images and lidar data of the cotton canopy are acquired simultaneously; at the same time, the cotton pest and disease situation at ground sampling points is collected as ground value labels, including the degree of cotton aphid infestation, the degree of red spider mite damage and the degree of Verticillium wilt infestation, and a priori dataset is constructed. S2. Data Preprocessing and Registration: Radiometric calibration and reflectance conversion are performed on hyperspectral and multispectral images. The Savitzky-Golay smoothing algorithm is used to smooth the hyperspectral and multispectral images. The lidar data is denoised, normalized, and a canopy height model (CHM) is generated. The geolocation information (POS) is used to perform spatiotemporal registration on multi-source data to construct a multimodal dataset. S3. Multi-source feature calculation and extraction: Construct a multi-branch feature extraction network. The hyperspectral branch extracts the fine spectral features of the hyperspectral image, the multispectral image branch extracts the spectral features, leaf texture and color features of the multispectral image, and the structural branch extracts the canopy height, curl and topological structure features of the lidar data. S4. Cross-modal feature fusion and recognition: Design a cross-modal attention module to spatially align LiDAR data and spectral features at the same scale. Calculate complementary weights of different modal features through the cross-modal attention module. Input the weighted fused features of the aligned multimodal features into a fully connected layer to generate a high-dimensional joint feature representation. S5. Pest and Disease Level Identification: Construct an RCG-Net model and train it using a multimodal dataset. Input the high-dimensional joint feature representation into the fully connected layer of the RCG-Net model and output the cotton pest and disease level identification results for the current monitoring area through a mapping function.

2. The method for identifying cotton pest and disease levels based on spectral and lidar imagery according to claim 1, characterized in that, In step S1, the prior dataset includes spectral images, lidar images and cotton ground pest and disease data. The OSAVI vegetation index is used to remove soil and shadow background, and the vegetation index (OSAVI) is constructed by extracting RE bands and NIR bands from the multispectral bands. In step S1, the specific method for constructing the prior dataset and vegetation indices is as follows: The Optimized Soil-Adjusted Vegetation Index (OSAVI) was used to remove the influence of soil background and shadow on the cotton canopy spectrum, and the red and near-infrared bands (NIR) in the multispectral images were extracted to construct spectral indices. The formula for calculating OSAVI is: ； Wherein, NIR is the reflectance in the near-infrared band, and Red is the reflectance in the red band.

3. The method for identifying cotton pest and disease levels based on spectral and lidar imagery according to claim 1, characterized in that, In step S2, the specific steps of data preprocessing include: Diffuse plate correction: Reflectance correction of hyperspectral images is performed using a standard diffuse plate. The correction model is as follows: ； In the formula, S1 is the hyperspectral image radiance value, S2 is the standard diffuse reflectance, and S3 is the standard diffuse reflectance value. Spectral data: Savitzky-Golay smoothing is used to reduce noise interference in the spectral imagery, resulting in the original spectral data; LiDAR data: The original point cloud data is classified using the Triangular Irregular Network (TIN) algorithm to distinguish between ground points and non-ground points; the ground points are interpolated using the TIN interpolation algorithm to generate a digital elevation model (DEM); the non-ground points are interpolated using the Kriging interpolation algorithm to generate a digital surface model (DSM); and the canopy height model (CHM) is generated through the difference operation between the DSM and the DEM (DSM-DEM).

4. The method for identifying cotton pest and disease levels based on spectral and lidar imagery according to claim 1, characterized in that, In step S3, the specific contents of the multi-branch feature extraction network and the multi-temporal vegetation index are as follows: Multi-temporal vegetation index calculation: Based on multi-temporal spectral images, a set of spectral features is calculated, including the Normalized Difference Vegetation Index (NDVI), Normalized Difference Red Edge Index (NDRE), Difference Vegetation Index (DVI), Enhanced Vegetation Index (EVI), and Structure Insensitive Pigment Index (SIPI). In texture feature extraction: texture features of images are extracted based on gray-level co-occurrence matrix (GLCM). Texture features include at least: energy, contrast, entropy, homogeneity, and correlation. In the hyperspectral feature extraction branch: a hybrid architecture of one-dimensional convolutional neural network (1D-CNN) is adopted, which utilizes depthwise separable convolution to extract spectral features between bands and capture the specific absorption peak changes caused by aphid damage; In the multispectral feature extraction branch, a two-dimensional residual network architecture is adopted for multispectral images. After the image input is processed by convolution and max pooling, it enters the stacked two-dimensional residual module. In order to enhance the expression of lesion and texture features, an ECA attention module is introduced after the residual module to realize the adaptive adjustment of channel weights. In the structural branch, the spatial neighborhood information of the CHM of the LiDAR data is aggregated using graph convolutional neural network (GCN) to capture the leaf curling and plant dwarfing characteristics caused by cotton aphids or cotton spider mites, thus making up for the deficiencies of two-dimensional images in structural representation.

5. The method for identifying cotton pest and disease levels based on spectral and lidar imagery according to claim 1, characterized in that, In step S4, the specific method for cross-modal feature fusion is as follows: Feature Mapping: A cross-modal attention layer is designed to map hyperspectral features, multispectral features, and LiDAR structural features to feature labels of a unified dimension. Using the key, value, and query matrix in the attention mechanism, the importance weights of features from different sensor data are dynamically calculated. Based on the calculated weights, modal features sensitive to pests and diseases are adaptively enhanced, and the weights of LiDAR features and structural features are increased to achieve complementary enhancement of different modal features.

6. The method for identifying cotton pest and disease levels based on spectral and lidar imagery according to claim 1, characterized in that, The multi-branch heterogeneous residual network RCG-Net model constructed in step S5 has a specific network architecture that includes parallel spectral branches, texture branches, and structural branches.

7. A cotton pest and disease severity identification system based on spectral and lidar imagery, applied to the cotton pest and disease severity identification method based on spectral and lidar imagery as described in any one of claims 1 to 6, characterized in that, include: Data acquisition module: used to acquire hyperspectral images, multispectral images and lidar data of the cotton canopy, and to obtain the corresponding ground pest and disease true value labels; Data processing and registration module: used to perform radiometric calibration, reflectivity conversion and CHM calculation on the multi-source images, and perform spatiotemporal registration to obtain a multimodal dataset; Multi-source feature extraction module: Employs a multi-branch network to extract fine spectral features, texture and color features, and topological features from the multimodal dataset; Cross-modal feature fusion module: used to calculate the feature weights of each modality through the cross-modal attention module, perform spatial alignment and weighted fusion, and generate a high-dimensional joint feature representation; Model training and recognition module: Construct and train the RCG-Net model, input the high-dimensional joint feature representation into the model, and output the cotton pest and disease level.

8. A computing platform, characterized in that... The method includes: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a cotton pest and disease level identification method based on spectral and lidar imagery as described in any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that... It stores executable instructions, which, when executed, cause the machine to perform a cotton pest and disease level identification system based on spectral and lidar images as described in claim 7.

Images