Crop disease and pest recognition method based on hyperspectral and LiDAR data

Abstract

The application discloses a crop disease and pest identification method based on hyperspectral and LiDAR data, and belongs to the technical field of pattern recognition. Firstly, a cascaded multi-scale convolution module is used to extract features of hyperspectral images and LiDAR data. Then, a fusion process is modeled as a sequential decision process, and the optimal classification target is promoted step by step. Secondly, four key constraint conditions, i.e., spectral fidelity, feature consistency, structure preservation and resolution matching degree, are monitored in real time at the feature level, and the fusion strategy is adjusted according to the monitored conditions, so that the disease and pest identification information of the two modalities can be fully preserved in the whole fusion process. Finally, the optimal complementary fusion strategy of the hyperspectral image and the LiDAR is obtained through a phased decision, so as to improve the disease and pest identification precision.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, and in particular relates to a method for identifying crop diseases and pests based on hyperspectral and LiDAR data. Background Technology

[0002] Outbreaks of crop diseases and pests not only severely impact grain yields but also continuously threaten food security. In large-scale agricultural production, traditional manual inspection methods suffer from high costs, low efficiency, and the inability to achieve large-scale real-time monitoring, failing to meet the demands of modern agriculture for precise identification and control of diseases and pests. With the development of hyperspectral imaging and LiDAR (Light Detection and Ranging) technologies, increasing research is combining these two technologies for intelligent monitoring of crop diseases and pests. Hyperspectral images provide the spectral characteristics of crops, enabling accurate identification of diseases and pests, while LiDAR data provides elevation information, aiding in the identification of spatial morphological changes in crops using hyperspectral images. The combination of these two technologies provides more comprehensive data support for the monitoring and identification of diseases and pests.

[0003] However, fusing hyperspectral images with LiDAR data for crop pest and disease identification still faces a series of challenges. First, the two modalities exhibit significant heterogeneity. Hyperspectral images primarily provide spectral information about crops, while LiDAR data provides elevation information; their data types and dimensionalities differ considerably. How to effectively fuse these two heterogeneous data types and extract effective features for pest and disease identification is a key issue. Most existing fusion methods are based on static fusion strategies, such as attention-based weighted fusion or gated network fusion. While these methods can improve identification accuracy to some extent, the complexity of crop types, pest and disease types, and environmental changes in farmland environments means that static fusion strategies cannot be flexibly adjusted according to the task scenario, easily overlooking important features and thus affecting identification accuracy.

[0004] Secondly, existing methods typically assume that all pest and disease species and their spectral characteristics are known, and that the data is static during training. However, in real agricultural environments, the occurrence of pests and diseases is influenced by multiple factors, and their spatial and temporal distribution exhibits high uncertainty. This necessitates that the identification model possess dynamic adaptability. Existing static fusion methods cannot cope with such dynamic changes, and therefore struggle to address variations in pest and disease species, crop varieties, and growing environments in real agricultural settings. Summary of the Invention

[0005] To address the problems existing in the background technology, the purpose of this invention is to provide a method for identifying crop diseases and pests based on hyperspectral and LiDAR data. Addressing the issues of static fusion strategies and modal heterogeneity in existing hyperspectral image and LiDAR data fusion processes, this invention provides a solution that models the multimodal feature fusion process as a sequential decision-making process.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The present invention provides a method for identifying crop diseases and pests based on hyperspectral and LiDAR data, comprising: Feature extraction was performed on the acquired hyperspectral images and LiDAR data to obtain hyperspectral image features and LiDAR features, respectively. The hyperspectral image features and LiDAR features are input into a sequential decision actor-critic network. The hyperspectral image features and LiDAR features are fused using the current fusion weights output by the sequential decision actor-critic network to generate the current fused features. The current fusion features are evaluated according to the constraint parsing mechanism, and the current fusion weights are corrected in real time based on the evaluation results to obtain the optimal fusion weights; The hyperspectral image features and LiDAR features are fused according to the optimal fusion weight to generate the optimal fused features, and the optimal fused features are classified to obtain the crop disease and pest identification results.

[0007] Furthermore, the feature extraction from the acquired hyperspectral image and LiDAR data includes: Feature extraction is performed on the acquired hyperspectral image, and the steps are as follows: For the hyperspectral image Dimensionality reduction is performed to obtain the reduced hyperspectral data. ; The dimensionality-reduced hyperspectral data The input hyperspectral image feature extraction network comprises a 1×1 convolutional block, a multi-scale convolutional block, a pyramid pooling module, and a fusion block connected sequentially. First, the hyperspectral data is processed through the 1×1 convolutional block. Extracting preliminary features Then, spatial features at different scales are captured through multi-scale convolutional blocks. The multi-resolution context information is aggregated through the pyramid pooling module to obtain the aggregated spatial features. The initial features are then integrated into the fusion module. Spatial features after aggregation The images are fused to output spectral-spatial coupled hyperspectral image features. ; Feature extraction is performed on the acquired LiDAR data, and the steps are as follows: LiDAR data is input into a LiDAR feature extraction network, which includes a 1×1 convolutional block, a multi-scale convolutional block, and a pyramid pooling module connected in sequence. First, preliminary features are extracted from the LiDAR data using the 1×1 convolutional block. Then, spatial features at different scales are captured using the multi-scale convolutional block. Finally, the pyramid pooling module aggregates multi-resolution contextual information to obtain the aggregated features, which serve as the LiDAR features output by the LiDAR feature extraction network. .

[0008] Furthermore, the sequential decision actor-critic network includes an actor network and a critic network; Based on hyperspectral image features and LiDAR features Construct the state vector at the current moment The state vector includes hyperspectral image features. LiDAR features Historical decision-making records and constraint state information The historical decision records Includes historical fusion weights; The actor network is based on the state vector Output the probability distribution of the current fusion weights, and generate the current fusion weights by sampling; The critic network is used to evaluate the value of the current fusion weights and assist in updating the actor network.

[0009] Furthermore, the evaluation metrics for the constraint resolution mechanism include at least one of spectral fidelity, feature consistency, structure preservation, and resolution matching. The spectral fidelity is calculated based on spectral angle mapping and is used to measure the spectral similarity between the current fused feature and the hyperspectral image feature. The feature consistency is calculated based on normalized L2 distance and is used to measure the degree of similarity in feature space between the current fused feature and the LiDAR feature; The structure preservation degree is calculated based on the structure similarity index and is used to measure the spatial structure preservation ability of the current fused features and the hyperspectral image features; The resolution matching degree is calculated based on gradient difference and is used to measure the ability of the current fused feature and the hyperspectral image feature to preserve details in terms of gradient magnitude and direction.

[0010] Furthermore, the current fusion features are evaluated according to the constraint resolution mechanism, including: The current evaluation metrics for fusion feature calculation are spectral fidelity, feature consistency, structure preservation, and resolution matching. When the value of any evaluation index exceeds the corresponding preset threshold range, the current fusion weight is adjusted until the fusion feature generated by the corrected fusion weight meets the requirements of each preset threshold.

[0011] Furthermore, a decision-making and control module is constructed to adjust the current fusion weights in real time based on the evaluation results. Specifically, it calculates spectral fidelity, feature consistency, structure preservation, and resolution matching based on the current fusion features. These parameters are then used as inputs to the decision-making and control module, and the resulting constraint weight distribution is output after passing through two fully connected layers. Minimize weight distribution with ideal weight The mean squared error between them is used to guide the parameter update direction of the actor network in the sequential decision actor-critic network.

[0012] Furthermore, the optimal fused features are classified to obtain the crop disease and pest identification results. The process is as follows: First, a two-dimensional adaptive average pooling layer is applied to the optimal fused features to perform global spatial aggregation of the fused features: ; Then it is fed into two layers of fully connected networks: ; in, It is the optimal fusion feature. This represents two-dimensional adaptive average pooling. These are the features after pooling. It is the first fully connected layer, with ReLU activation function. Dropout is a regularization layer, and p represents the dropout rate. The second fully connected layer, It is a category score vector; Category score vector The predicted category is obtained by taking the maximum value. The result of the identification of crop diseases and pests is represented as follows: =argmax( ); Here, argmax represents taking the maximum value.

[0013] Furthermore, during training, the cross-entropy loss function is used. Optimize network parameters: ; in, It is the number of samples. It is a sample Category The true label, which is 0 or 1. The model predicts the sample. Category The probability of.

[0014] Furthermore, the crop diseases and pests mentioned include potato virus Y, wheat grain nematode, corn spider mite, wheat aphid, wheat stripe rust, powdery mildew, and grape downy mildew.

[0015] This invention also provides a crop pest and disease identification system based on hyperspectral and LiDAR data, comprising: The feature extraction module is used to extract features from the acquired hyperspectral images and LiDAR data respectively, to obtain hyperspectral image features. and LiDAR features ; A sequential decision actor-critic network, comprising an actor network and a critic network, is used to process the hyperspectral image features. and LiDAR features As input to the sequential decision actor-critic network, the current fusion weights are output through the actor network; The fusion module, based on the current fusion weights, adjusts the hyperspectral image features. and LiDAR features The hyperspectral image features are fused to obtain the current fused features; and the hyperspectral image features are then adjusted based on the optimal fusion weights. and LiDAR features The optimal fusion feature is obtained by performing fusion. The constraint parsing module is used to evaluate the current fusion feature and correct the current fusion weight in real time based on the evaluation result to obtain the optimal fusion weight; A classifier network classifies the optimal fused features and outputs the identification results of crop diseases and pests.

[0016] Furthermore, the feature extraction module includes a hyperspectral image feature extraction network and a LiDAR feature extraction network; A dimensionality reduction unit is used to perform principal component analysis dimensionality reduction on the hyperspectral image; A hyperspectral image feature extraction network, consisting of sequentially connected 1×1 convolutional blocks, multi-scale convolutional blocks, pyramid pooling modules, and fusion blocks, is used to extract spectral-spatial coupled hyperspectral image features from dimensionality-reduced hyperspectral data. The LiDAR feature extraction network includes a 1×1 convolutional block, a multi-scale convolutional block, and a pyramid pooling module connected in sequence, used to extract LiDAR features from the LiDAR data.

[0017] Furthermore, the crop diseases and pests mentioned include potato virus Y, wheat grain nematode, corn spider mite, wheat aphid, wheat stripe rust, powdery mildew, and grape downy mildew. Among them, potato virus Y or wheat grain nematode can cause symptoms such as yellowing and reddening, corn spider mite or wheat aphid can cause wilting symptoms, and wheat stripe rust, powdery mildew, or grape downy mildew can cause pustules and tissue lesions.

[0018] Beneficial Effects: Compared with existing technologies, this invention provides a method for identifying crop diseases and pests based on hyperspectral and LiDAR data. First, cascaded multi-scale convolution and pyramid pooling are used to extract heterogeneous features from hyperspectral images and LiDAR data, obtaining spectral-spatial coupling features and elevation-structure features. Second, a sequential decision actor-commentator network is designed to model the feature fusion process as a sequential decision-making process. Then, the bimodal features are input into the sequential decision actor-commentator network, and at each time step, the Actor outputs the fusion weights, which are then corrected in real time by a constraint parsing mechanism to generate fusion features that satisfy four constraints: spectral fidelity, feature consistency, structure preservation, and resolution matching. Finally, through phased decision control, spectral separability is preserved in the initial stage, modal weights are balanced in the middle stage, and structural details are optimized in the later stage to achieve accurate identification of diseases and pests. This invention transforms the multimodal fusion process into a sequential decision problem, overcoming the limitations of static weight fusion in adapting to complex farmland scenarios and improving the accuracy and stability of disease and pest identification in open and dynamic environments. Attached Figure Description

[0019] Figure 1 This is a block diagram illustrating the principle of the crop disease and pest identification method of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0021] The method for identifying crop diseases and pests based on hyperspectral and LiDAR data provided by this invention has the following specific principle: Figure 1As shown, firstly, a multimodal feature embedding module is constructed to extract features from hyperspectral and LiDAR data respectively, obtaining hyperspectral image features and LiDAR features. Secondly, a sequential decision actor-critic mechanism is designed to model the feature fusion process as a sequential decision problem. The model autonomously decides fusion weights, progressing towards the optimal fusion strategy step by step. The Actor-Critic framework from reinforcement learning is introduced, enabling the model to intelligently adjust the fusion strategy between hyperspectral images and LiDAR data. In each training phase, the model adaptively adjusts the fusion strategy based on the features of the current data and task requirements, thereby improving the model's adaptability and recognition accuracy in different agricultural environments. Next, a constraint parsing mechanism is constructed to monitor the key constraints of spectral fidelity, feature consistency, structure preservation, and resolution matching of the fused features in real time during the feature fusion process. The fusion strategy is intelligently adjusted to ensure explicit assurance of the quality of the fused features, ensuring that key features for disease and pest identification are not lost during the fusion process. Finally, a decision control module is proposed. By adjusting the constraint importance configuration in stages, a complete fusion strategy evolution of spectral preservation, modal balance, and structural optimization is achieved. This ensures that different optimization objectives are focused on at different training stages, realizing the optimal complementary fusion of HSI and LiDAR data to improve the accuracy of pest and disease identification. This ensures the accuracy and stability of the final pest and disease identification. The crop pests and diseases in this specific implementation include potato virus Y, wheat grain nematode, corn spider mite, wheat aphid, wheat stripe rust, powdery mildew, and grape downy mildew. Among them, potato virus Y or wheat grain nematode can cause symptoms such as yellowing and reddening, corn spider mite or wheat aphid can cause wilting symptoms, and wheat stripe rust, powdery mildew, or grape downy mildew can cause pustules and tissue lesions.

[0022] The method for identifying crop diseases and pests based on hyperspectral and LiDAR data of the present invention specifically includes the following steps:

[0023] Step 1, Multimodal Feature Extraction: Spectral correlation and elevation basis features are extracted from hyperspectral images and LiDAR data using 1x1 convolution, respectively. Then, through four-way multi-scale convolution, pyramid pooling, and fusion convolution, the hyperspectral image features with spectral spatial coupling and the LiDAR features with elevation structure are obtained.

[0024] (1) Feature extraction of hyperspectral images is performed as follows: For hyperspectral images Principal component analysis (PCA) was applied for dimensionality reduction to obtain the dimensionality-reduced hyperspectral image data. ,in, Represents hyperspectral images The number of channels, Indicates the number of channels The number of channels after dimensionality reduction express The set of real numbers of dimension , express The set of real numbers of dimension s, where s represents the spatial size of the hyperspectral image and LiDAR data. Dimensionally reduced hyperspectral image data. and raw LiDAR data The data are then fed into the feature extraction network. express The set of real numbers of dimension , Represents LiDAR data The number of channels.

[0025] Two basic units are constructed: convolutional blocks and multi-scale convolutional blocks. A 1×1 convolutional block combines convolution, batch normalization, and ReLU activation, and can be represented as follows: Multi-scale convolutional blocks capture spatial features at different scales by using convolutions of 1×1, 3×3, 5×5, and 7×7 kernel sizes in parallel. They are defined as follows: ,in Indicates the kernel size as The branch, in this specific implementation The values ​​are 1, 3, 5, and 7. The fusion function enables the feature extraction network to respond to ground objects of different sizes simultaneously, thus enhancing the scale invariance of the feature extraction network.

[0026] A three-stage module is proposed for extracting features from hyperspectral images. The first stage is spectral feature extraction; given a hyperspectral image after dimensionality reduction via principal component analysis (PCA). First, two 1×1 convolutions are applied to extract spectral features. , Indicate spectral characteristics The number of channels. 1×1 convolution focuses on learning the correlation between bands without introducing spatial context, which is crucial for distinguishing ground features with similar spectral characteristics. Second stage: multi-scale spatial feature capture; then, spectral features... Input multi-scale convolutional blocks to capture spatial features at different scales. : ; in, Represents multi-scale convolutional blocks; Effectively identify ground features of different sizes and shapes. The third stage: global aggregation and feature fusion; subsequently, pyramid pooling blocks (PPM) are used to aggregate global contextual information at multiple resolutions (1×1, 2×2, 3×3, and 6×6) to obtain the pyramid-pooled spatial features. : ; Finally, spectral features are connected through fusion blocks. and spatial features The final hyperspectral image features are obtained. ,in, Indicating hyperspectral image features Number of channels: ; in, Indicates a merge block; (2) Feature extraction of LiDAR data is performed as follows: For extracting LiDAR features, input LiDAR data First, a 1×1 convolutional block is applied to extract initial features, namely elevation response basis features. Then, these initial features are processed through multi-scale convolutional blocks to capture multi-scale spatial information. Similar to the hyperspectral image branch, pyramid pooling blocks (PPM) are used to aggregate the global context to obtain the final LiDAR features. , Represents LiDAR features The number of channels.

[0027] Step 2, Constrained Evolutionary Sequential Decision Fusion: The obtained bimodal features are used as the initial state vector and input into the sequential decision actor-critic network. The bimodal features are hyperspectral image features. and LiDAR features The Actor network outputs actions and corrects them in real time through constraint parsing. The Sequential Decision-Making Actor-Critic network uses the corrected actions to generate fused features, while recording classification accuracy and constraint loss as historical decision information. Complete experience tuples are stored in the experience pool, and the Actor network and the online Q network (Critic network) are continuously updated. The target Q network synchronizes the parameters of the online Q network through a soft update mechanism, realizing constraint evolution sequential decision fusion.

[0028] Among them, the sequential decision actor-critic network uses a custom fusion environment, which consists of a state space, an action space, a reward function, and transition probabilities; state space The state space contains descriptive information about the entire environment and is the foundation for agent decision-making. The state in the sequential decision actor-commentator network includes not only HSI features from hyperspectral images and LiDAR features, but also historical decision information (i.e., previous modality fusion weights) and constraint state information.

[0029] Action space The action space determines the actions an agent can take, i.e., how to choose the fusion weights for each modality. Specifically, the action in a sequential decision actor-critic network is a continuous weight vector.

[0030] reward function The reward function is an indicator for judging the quality of an agent's behavior. A composite reward function is designed, consisting of rewards at multiple time scales, including immediate rewards, mid-term rewards, long-term rewards, and penalties for breach of contract.

[0031] Transition probability State transition function Indicates how to start from the current state at time t Execute actions The state then transitions to the state at time t+1. .

[0032] State observation and action generation include: First, at the start of each training iteration, the agent obtains the current state from the environment. : ; in, and These are hyperspectral image features and LiDAR features, respectively. It is the agent's historical decision record, including previous fusion weights. It is the constraint state information related to feature fusion. The state vector fully captures the feature distribution and fusion history of the current sample, providing complete information for the agent's decision-making.

[0033] Next, the Actor network processes the state vectors and outputs the action distribution parameters. , constraint values ​​predicted by the constraint head : ; in, and These are the mean and standard deviation of the action distribution, respectively. The parameter is The policy network. It adopts a parameterized Gaussian distribution representation strategy, which can simultaneously explore and utilize, adapting to the characteristic differences of different land cover types.

[0034] Finally, the action at time t is randomly generated using the reparameterization technique. : ; Among them, actions Including the current fusion weights of hyperspectral data Current fusion weights with LiDAR data , It is a standard normal distribution The random noise sampled in the middle, where I represents the identity matrix, Represents element-wise product. The function compresses the output to the range [-1, 1] to ensure that the generated action is within the valid range.

[0035] The reward includes: the total reward at time t. From instant rewards Mid-term rewards Long-term rewards and penalties for breach of contract It consists of four parts: ; in, and It is a discount factor for medium- and long-term rewards, balancing the importance of goals at different time scales.

[0036] In each interaction with the environment, the agent changes from state to state. Execute action The state transition yields And receive a reward Generate experience tuples They are stored in a priority experience pool, with priority calculated based on constraint loss and sorted for early sampling. Indicates a terminator.

[0037] Sequential Decision Actors-Critic Network Updates include: First, the sequential decision actor-critic network evaluates the value of the current fusion strategy through a Q-network, i.e., calculates the target value. To mitigate the overestimation problem; the Q-network includes an online Q-network and a dual-objective Q-network, the dual-objective Q-network including a first objective Q-network. and the second target Q network Calculate the target value The minimum value of the two outputs is taken, which is expressed as: ; in, Indicates the target Q network The parameters, It is a discount factor. Represents a target policy network and the next state Sample the next action , It's a temperature parameter that controls the strength of entropy regularization. The predicted value equals the immediate reward plus any discounted reward that might be received in the future (if the task is not yet finished).

[0038] Online Q network By minimizing the online Q-value and the objective value Learning from the differences between them, the loss function of the online Q-network is: ; in, For online Q network The parameters, It is the importance sampling weight of the i_sampleth empirical sample, which corrects the bias introduced by priority sampling. Indicates online Q network exist The online Q-value is calculated and output. Batch samples sampled from the experience replay pool D Calculate the expected value of the weighted mean square error.

[0039] Then, the policy network It is responsible for generating new fusion weight decisions, learning by maximizing the online Q-value while maintaining appropriate decision randomness. (Online Q-network) It is a general function approximator. The policy network simultaneously learns the optimal fusion weights and constrained predictions: ; ; ; in, Indicates the target Q network State-action pair The value estimate, These are the constraint values ​​predicted by the policy network. It is the target constraint value. These are constraint loss weights. Policy loss. It consists of two parts: a weighted sum of policy entropy and online Q-value. and constrained prediction loss The policy network can not only generate high-reward actions (with fused weights), but also predict the impact of these actions on constraint satisfaction.

[0040] Temperature parameters By minimizing the following loss Automatic adjustment: ; in, Indicates the expectation. This indicates that weighted actions are sampled and fused from the action distribution output by the target policy network. , It is the target entropy value, usually set as the negative of the action space dimension (i.e., The temperature parameter can be adaptively adjusted, increasing when the policy entropy is below the target value (encouraging more exploration) and decreasing when the policy entropy is above the target value (encouraging more utilization).

[0041] Finally, the target Q network Slowly track the online Q network through soft updates. parameters : ; in, These are soft update coefficients that control the target Q-network. Tracking online Q network The slow update mechanism improves training stability and prevents drastic fluctuations in online Q-value estimation.

[0042] The constraint resolution mechanism construction includes: First, after completing action generation and adjusting the fusion weights, perform preliminary feature fusion: ; in, Indicates the current fusion characteristics, and These are the current fusion weights for hyperspectral data and LiDAR data. It is a fusion network consisting of a convolutional layer, batch normalization, and a ReLU activation function. It maps weighted concatenated features to a unified fusion representation, allowing the two modalities to contribute different levels of information based on the characteristics of the current scene while maintaining their relative independence.

[0043] Then, design the spectral fidelity. Feature consistency Structural retention Matching degree with resolution Four categories of fusion characteristic evaluation indicators to comprehensively assess fusion quality:

[0044] Spectral fidelity is evaluated based on Spectral Angle Mapper (SAM) to assess the fidelity of fused features to hyperspectral image features: ; in, Indicates spectral fidelity. Indicating hyperspectral image features With fusion features The inner product, The norm is represented. The spectral angle is normalized to the [0,1] interval. The smaller the angle, the greater the spectral similarity between the fused features and the original hyperspectral image features.

[0045] Feature consistency, based on normalized L2 distance, measures the degree to which fused features preserve the spatial structure of the lidar. ; in, Indicates feature consistency. Features of lidar With fusion features Euclidean distance, Features of lidar norm, It is a small constant to prevent division by zero errors. Normalization prevents high-value features from dominating distance calculations and ensures fair comparison of features at different scales.

[0046] The structural preservation rating assesses the spatial structural preservation capability of fused features based on the improved Structural Similarity Index (SSIM). ; in, Indicates structural retention. and These are the mean values ​​of the hyperspectral image features and the fused features, respectively. and These are the standard deviations of hyperspectral image features and fused features, respectively. It is the covariance of hyperspectral image features and fused features. and It is a stability constant. The Structural Similarity Index (SSIM) considers three aspects: brightness, contrast, and structure, ensuring the complete preservation of feature boundaries and texture features after fusion.

[0047] Resolution matching evaluates the ability to preserve details of fused features based on gradient difference calculations. ; in, Indicates resolution matching degree. Hyperspectral image features gradient magnitude, LiDAR features gradient magnitude, Hyperspectral image features gradient direction, LiDAR features The gradient direction, with weights of 0.7 and 0.3, reflects the importance of amplitude differences relative to directional differences. The first term assesses gradient amplitude differences, and the second term assesses gradient direction differences, ensuring that small-scale ground features are not smoothed or lost.

[0048] Finally, constraint-guided action adjustments are performed. While the initial actions generated by the policy network are based on the optimization direction, they may not meet specific constraints. For example, it's necessary to ensure that the fusion policy conforms to physical constraints while exploring the optimal solution. In HSI and LiDAR data fusion classification tasks, reinforcement learning, in order to maximize expected rewards, inevitably explores some destructive actions that impair the unique features of each modality, thereby reducing the quality of fused features. Therefore, ineffective or harmful fusion actions must be avoided.

[0049] The constraint parsing mechanism works in real time on each batch and each sample to ensure that each fusion operation meets the basic constraint requirements so that the fusion strategy can be adjusted in real time and the results of each constraint evaluation are recorded in the sequential decision actor-commentator network loss history. This historical data is then used by the decision control module.

[0050] Step 3, Phased Decision Control and Final Identification: A phased learning strategy is adopted, and the learning rate and exploration rate are dynamically adjusted according to the constraint state. Conservative updates are used in complex scenarios, while exploration is accelerated in simple scenarios. Finally, the learned optimal fusion strategy is used to accurately identify pest and disease pixels. Specifically, this includes: Step 31: The decision control module adopts a phased learning strategy, prioritizing the preservation of spectral separability in the initial stage, balancing the weights of the two modes in the middle stage, and focusing on structural optimization in the later stage. Step 32: Dynamically adjust the learning rate and exploration rate based on the constraint state, making conservative updates in complex scenarios and accelerating exploration in simple scenarios; Step 33: Use the learned optimal fusion strategy to accurately identify pests and diseases.

[0051] The decision-making and control module includes the following components: The decision-making and control module is used for long-term learning direction planning. Instead of directly modifying the fusion weights, this module focuses on shifts in learning priorities and changes in learning parameters at each stage to guide the learning direction of the sequential decision actor-critic network.

[0052] Spectral fidelity received by the decision control module Feature consistency Structural retention Matching degree with resolution as input vector , is represented as: ; First, the input is mapped to the hidden space through the first fully connected layer: ; in, and These are the weights and bias parameters of the first fully connected layer.

[0053] Subsequently, it is transformed into a constrained weight distribution through a second fully connected layer. : ; in, and These are the weights and bias parameters of the second fully connected layer. The softmax function ensures that the sum of all constraint weights is 1, forming an effective weight distribution.

[0054] Next, a phased learning strategy is implemented, dynamically adjusting the ideal weights according to the training progress. : Secondly, the decision-making and control module is trained using supervised learning to minimize the weight distribution. with ideal weight The mean square error between them.

[0055] Finally, the learning hyperparameters are dynamically adjusted to adapt to the learning difficulty of different regions: ; ; in, and These are the learning rate and exploration rate at the current time step, respectively. and These are the baseline values ​​for the learning rate and the exploration rate, respectively. and It is an adjustment function based on constraint states. The Sequential Decision Actor-Critic Network adopts a more cautious learning strategy when facing difficult-to-classify regions, while actively exploring and finding the optimal fusion strategy in simple regions. The optimal fusion strategy is a fusion weight allocation strategy learned by the Sequential Decision Actor-Critic Network that maximizes long-term cumulative rewards (i.e., balances classification accuracy and constraint satisfaction). This strategy assigns optimal fusion weights to hyperspectral features and LiDAR features, enabling the fused features to achieve the best pest and disease recognition performance while satisfying constraints.

[0056] The classification system includes: Obtain the features after fusion using the optimal fusion strategy Subsequently, the classifier network is responsible for mapping the samples to their corresponding land cover categories. The classifier network designed in this invention contains multiple components to fully utilize fused feature information.

[0057] The classification process can be represented as: ; in, Represents a classifier. It is the predicted category; In the classifier network, firstly, a two-dimensional adaptive average pooling layer is used to perform global spatial aggregation of the fused features: ; in, It is a target fusion feature. This represents two-dimensional adaptive average pooling. These are the features after pooling. It refers to the batch size. It is a feature The number of channels, and It is the height and width of the feature map that enable the classifier network to focus on overall characteristics rather than local changes.

[0058] Next, the features are flattened and fed into a two-layer fully connected network: ; in, It is the first fully connected layer with the ReLU activation function, and Dropout is a regularization layer (p=0.5 indicates a 50% dropout rate). The second fully connected layer, It is a category score vector.

[0059] Finally, the category score vector The predicted category is obtained by taking the maximum value. , is represented as: =argmax( ) During training, the cross-entropy loss function is used to optimize the classifier network parameters: ; in, It is the sample size. It is a sample Category The true label, which is 0 or 1. The model predicts the sample. Category The probability of this. The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for identifying crop diseases and pests based on hyperspectral and LiDAR data, characterized in that, include: Feature extraction was performed on the acquired hyperspectral images and LiDAR data to obtain hyperspectral image features and LiDAR features, respectively. The hyperspectral image features and LiDAR features are input into a sequential decision actor-critic network. Based on the current fusion weights output by the sequential decision actor-critic network, the hyperspectral image features and LiDAR features are fused to generate the current fused features. The current fusion features are evaluated according to the constraint parsing mechanism, and the current fusion weights are corrected in real time based on the evaluation results to obtain the optimal fusion weights; The hyperspectral image features and LiDAR features are fused according to the optimal fusion weight to generate the optimal fused features, and the optimal fused features are classified to obtain the crop disease and pest identification results.

2. The method for identifying crop diseases and pests according to claim 1, characterized in that, The feature extraction process for the acquired hyperspectral images and LiDAR data includes: Feature extraction is performed on the acquired hyperspectral image, and the steps are as follows: Hyperspectral images Dimensionality reduction is performed to obtain the dimensionality-reduced hyperspectral data. ; The dimensionality-reduced hyperspectral data The input hyperspectral image feature extraction network comprises a 1×1 convolutional block, a multi-scale convolutional block, a pyramid pooling module, and a fusion block connected sequentially. First, the hyperspectral data is processed through the 1×1 convolutional block. Extracting preliminary features Then, spatial features at different scales are captured through multi-scale convolutional blocks. The multi-resolution context information is aggregated through the pyramid pooling module to obtain the aggregated spatial features. In the fusion module, the preliminary features are... Spatial features after aggregation The fusion process is performed to output spectral-spatial coupled hyperspectral image features. ; Feature extraction is performed on the acquired LiDAR data, and the steps are as follows: LiDAR data is input into a LiDAR feature extraction network, which includes a 1×1 convolutional block, a multi-scale convolutional block, and a pyramid pooling module connected in sequence. First, preliminary features are extracted from the LiDAR data using the 1×1 convolutional block. Then, spatial features at different scales are captured using the multi-scale convolutional block. Finally, the pyramid pooling module aggregates multi-resolution contextual information to obtain the aggregated features, which serve as the LiDAR features output by the LiDAR feature extraction network. .

3. The method for identifying crop diseases and pests according to claim 1, characterized in that, The sequential decision actor-critic network includes an actor network and a critic network; Based on hyperspectral image features and LiDAR features Construct the state vector at the current moment The state vector Including hyperspectral image features LiDAR features Historical decision-making records and constraint state information The historical decision records Includes historical fusion weights; The actor network is based on the state vector Output the probability distribution of the current fusion weights, and generate the current fusion weights by sampling; The critic network is used to evaluate the value of the current fusion weights and assist in updating the actor network.

4. The method for identifying crop diseases and pests according to claim 1, characterized in that, The evaluation metrics for the constraint resolution mechanism include at least one of spectral fidelity, feature consistency, structure preservation, and resolution matching. The spectral fidelity is calculated based on spectral angle mapping and is used to measure the spectral similarity between the current fused feature and the hyperspectral image feature. The feature consistency is calculated based on normalized L2 distance and is used to measure the degree of similarity in feature space between the current fused feature and the LiDAR feature; The structure preservation degree is calculated based on the structure similarity index and is used to measure the spatial structure preservation ability of the current fused features and the hyperspectral image features; The resolution matching degree is calculated based on gradient difference and is used to measure the ability of the current fused feature and the hyperspectral image feature to preserve details in terms of gradient magnitude and direction.

5. The method for identifying crop diseases and pests according to claim 4, characterized in that, The current fusion features are evaluated according to the constraint resolution mechanism, including: The current evaluation metrics for fusion feature calculation are spectral fidelity, feature consistency, structure preservation, and resolution matching. When the value of any evaluation index exceeds the corresponding preset threshold range, the current fusion weight is adjusted until the fusion feature generated by the corrected fusion weight meets the requirements of each preset threshold.

6. The method for identifying crop diseases and pests according to claim 4, characterized in that, A decision-making and control module is constructed to adjust the current fusion weights in real time based on the evaluation results. Specifically, it calculates spectral fidelity, feature consistency, structure preservation, and resolution matching based on the current fusion features. These parameters are then used as inputs to the decision-making and control module, and the resulting constraint weight distribution is output after passing through two fully connected layers. Minimize weight distribution with ideal weight The mean squared error between them is used to guide the parameter update direction of the actor network in the sequential decision actor-critic network.

7. The method for identifying crop diseases and pests according to claim 1, characterized in that, The optimal fused features are classified to obtain the crop disease and pest identification results. The process is as follows: First, a two-dimensional adaptive average pooling layer is applied to the optimal fused features to perform global spatial aggregation of the fused features: ； Then it is fed into two layers of fully connected networks: ； in, It is the optimal fusion feature. This represents two-dimensional adaptive average pooling. These are the features after pooling. It is the first fully connected layer, with ReLU activation function. Dropout is a regularization layer, and p represents the dropout rate. The second fully connected layer, It is a category score vector; Category score vector The predicted category is obtained by taking the maximum value. The result of the identification of crop diseases and pests is represented as follows: =argmax( ); Here, argmax represents taking the maximum value.

8. The method for identifying crop diseases and pests according to claim 1, characterized in that, During training, the cross-entropy loss function is used. Optimize network parameters: ； in, It is the number of samples. It is a sample Category The true label, which is 0 or 1. The model predicts the sample. Category The probability of.

9. The method for identifying crop diseases and pests according to claim 1, characterized in that, The crop diseases and pests mentioned include potato virus Y, wheat grain nematode, corn spider mite, wheat aphid, wheat stripe rust, powdery mildew, and grape downy mildew.

Images