A crop disease and pest early warning method based on multispectral fusion

Abstract

The application discloses a crop disease and pest early warning method based on multispectral fusion, relates to the technical field of multispectral fusion, and comprises the following steps: acquiring a first modal phenotype image sequence and a second modal physiological image sequence of a target farmland area, and generating a multi-modal farmland image dataset; inputting the multi-modal farmland image dataset into a pre-constructed disease and pest feature perception network, acquiring cross-modal correlation features through a time-space and spectrum joint attention mechanism, and generating a fusion feature map; based on the fusion feature map, determining the health index and stress degree of the current crop by using a plant stress state evaluation module, and generating a disease and pest early warning signal based on the health index and stress degree; according to the disease and pest early warning signal, combining historical meteorological data and a pre-constructed disease and pest pathogenesis knowledge graph, predicting the evolution trend of the disease and pest, and generating a structured early warning report containing an early warning level and a control suggestion; and the application improves the timeliness of farmland disease and pest monitoring and early warning.

Description

Technical Field

[0001] This invention relates to the field of multispectral fusion technology, specifically to a method for early warning of crop diseases and pests based on multispectral fusion. Background Technology

[0002] Crop diseases and pests are the core causes of reduced grain yields, decreased agricultural product quality, and agricultural non-point source pollution. Timely and accurate field disease and pest monitoring and early warning are essential requirements for green plant protection and precision agriculture. Currently, the mainstream disease and pest monitoring methods are divided into three categories: manual field inspection, single visible light remote sensing monitoring, and simple multispectral identification. All of these methods have significant technical shortcomings: most existing monitoring methods only use single-type visible light RGB images, which can only capture the texture and color of the crop surface. In the early stages of disease and pest infection, there are no obvious yellow spots or insect holes on the crop surface. Only physiological parameters such as chlorophyll, water, and nitrogen show slight changes, and visible light images cannot capture these stress characteristics. Some simplified multispectral solutions only perform single-frame spectral segmentation, failing to integrate phenotypic texture and physiological reflectance modalities. Soil, weeds, and uneven lighting can easily cause feature confusion, making it impossible to effectively distinguish between diseases, pests, and drought stress. The false negative rate for early microinfected plants exceeds 40%, and the recognition accuracy for the same disease varies by 15% to 30% between different crops. At the same time, there are deviations in the poses of the two types of image acquisition sensors, resulting in pixel-level spatiotemporal registration failure and misalignment of spectral and spatial features, further reducing the effectiveness of feature fusion.

[0003] Traditional multispectral recognition only infers independently from a single image, without constructing a spatiotemporal-spectral joint attention mechanism. It cannot establish a spatial correspondence between texture features and vegetation indices or water stress features, and weak pest and disease features are easily obscured by the complex background of farmland. Furthermore, existing algorithms ignore the temporal correlation of images and cannot use the physiological change trends of plants in consecutive frames to assist in discrimination. Relying solely on static single-frame images makes it difficult to distinguish between short-term light disturbances and continuous pest and disease stress. The model has poor generalization ability, and the recognition accuracy drops significantly under cloudy conditions and changes in light intensity between morning and evening. It only performs target recognition and lacks disease transmission projection and closed-loop control links. Existing technologies only output simple identification results of the presence or absence of pests and diseases, and cannot combine meteorological conditions such as temperature, humidity, and precipitation with the pathogenic mechanisms of pests and diseases to infer the spread range and development trend. It cannot quantify the warning level and can only detect diseases after the fact, but cannot predict the risk of outbreak in advance. At the same time, the recognition results are completely disconnected from plant protection operations. Warning reports cannot be directly sent to drones and irrigation equipment for precise control. Manual secondary assessment is time-consuming and delayed, missing the best control window. Summary of the Invention

[0004] To address the aforementioned technical problems, a method for early warning of crop diseases and pests based on multispectral fusion is provided. This technical solution solves the problems mentioned above.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for early warning of crop diseases and pests based on multispectral fusion includes: The first modality phenotypic image sequence and the second modality physiological image sequence of the target farmland area are acquired, and the first modality phenotypic image sequence and the second modality physiological image sequence are spatiotemporally registered to generate a multimodal farmland image dataset. The multimodal farmland image dataset is input into a pre-constructed pest and disease feature perception network. Cross-modal correlation features are obtained through a spatiotemporal and spectral joint attention mechanism to generate a fused feature map. Based on the fused feature map, the plant stress status assessment module is used to determine the current crop health index and stress level, and disease and pest early warning signals are generated based on the health index and stress level. Based on pest and disease early warning signals, combined with historical meteorological data and a pre-constructed knowledge graph of pest and disease pathogenesis, the evolution trend of pests and diseases is predicted, and a structured early warning report containing early warning levels and control recommendations is generated.

[0006] Furthermore, the multimodal farmland image dataset is input into a pre-constructed pest and disease feature perception network. Cross-modal correlation features are obtained through a spatiotemporal and spectral joint attention mechanism to generate a fused feature map, including: By utilizing the phenotypic feature extraction branch in the pest and disease feature perception network, texture and color features in the first modality phenotypic image sequence are obtained; Physiological feature extraction branches were used to obtain vegetation index features and water stress features in the second modality physiological image sequence; By using a spatiotemporal and spectral joint attention mechanism, vegetation index features are used as guiding masks to determine the attention weights in texture features that have spatial correspondence with vegetation index features. Attention weights are used to weight and fuse texture features and vegetation index features to generate a fused feature map.

[0007] Furthermore, based on the fused feature map, the plant stress status assessment module determines the current crop health index and stress level, and generates pest and disease early warning signals based on the health index and stress level, including: Physiological parameters of each crop plant region, including chlorophyll content, water content and nitrogen content, are obtained from the fused feature map. Physiological parameters are compared with a preset health benchmark model to determine the health index; When the health index is not higher than the preset health threshold, a stress state is determined, and the stress type is determined based on the abnormal combination of physiological parameters. The stress type includes disease stress, pest stress, or environmental stress. Based on the type and degree of stress, corresponding pest and disease early warning signals are generated.

[0008] Furthermore, based on pest and disease early warning signals, combined with historical meteorological data and a pre-constructed knowledge graph of pest and disease pathogenesis, the evolution trend of pests and diseases is predicted, and a structured early warning report containing warning levels and control recommendations is generated, including: Obtain historical meteorological data for the target farmland area, including temperature, humidity, and rainfall; Using pest and disease early warning signals as the initial triggering condition, and combining them with historical meteorological data, the data is input into a pre-constructed knowledge graph of pest and disease pathogenesis to perform graph reasoning, thereby simulating the spread rate and range of pests and diseases under the current environmental conditions. Based on the rate of spread and the extent of diffusion, predict the evolution trend of pests and diseases within a predetermined time period in the future; Based on the evolution trend, the warning level is determined, and the corresponding prevention and control strategies are matched to generate a structured warning report.

[0009] Furthermore, spatiotemporal registration is performed on the first modality phenotypic image sequence and the second modality physiological image sequence to generate a multimodal farmland image dataset, including: Based on the relative pose parameters of the phenotypic sensor and the physiological sensor, geometric correction is performed on the first modality phenotypic image sequence and the second modality physiological image sequence; The corrected first modality phenotypic image sequence and the second modality physiological image sequence were aligned at the pixel level using a feature point matching algorithm. Radiometric calibration and atmospheric correction are performed on the aligned images to generate a standardized multimodal farmland image dataset.

[0010] Furthermore, the training process of the disease and pest feature perception network includes: Acquire a historical farmland image sample set and label each sample with pixel-level pest and disease tags and physiological parameter tags; Construct a joint loss function that includes phenotypic segmentation loss, physiological parameter regression loss, and cross-modal consistency loss; The disease and pest feature perception network is trained end-to-end using a joint loss function. While minimizing the phenotypic segmentation error, the semantic consistency between phenotypic and physiological features in the disease and pest region is constrained, resulting in the trained disease and pest feature perception network.

[0011] Furthermore, after generating a structured early warning report that includes the warning level and prevention recommendations, it also includes: The structured early warning report will be sent to the agricultural management platform; The agricultural management platform automatically triggers corresponding prevention and control instructions based on the early warning level. These instructions include drone plant protection operation instructions or irrigation and fertilization instructions. A multimodal farmland image dataset is acquired after pest control operations to assess the effectiveness of pest control. The agricultural management platform then updates the preset health baseline model based on the assessment results.

[0012] Furthermore, before determining the current crop health index and stress level using the plant stress status assessment module based on the fused feature map, the following steps are also included: Morphological processing was performed on the fused feature map to separate the soil background from the weed area; Based on the pixel classification results in the fused feature map, the canopy coverage and leaf area index of crop plants are determined. Canopy coverage and leaf area index are used as auxiliary features and spliced ​​with the fused feature map to generate an enhanced feature map; Health indices and stress levels are determined based on enhanced feature maps.

[0013] Furthermore, after predicting the evolution trend of pests and diseases, it also includes: When the prediction results show that the spread of pests and diseases is not less than the preset safety threshold, an emergency warning signal is generated. The emergency warning signal triggered the agricultural management platform to activate its emergency response strategy, which included sending warning notices to surrounding farmers and coordinating prevention and control resources.

[0014] Furthermore, before being applied to farmland monitoring equipment, the pest and disease characteristic sensing network also includes: Obtain the trained pest and disease feature perception network obtained after iterative training according to the joint loss function; Channel pruning is performed on the trained pest and disease feature perception network to remove redundant channels whose contribution to the fused feature map is not higher than a preset contribution threshold. Knowledge distillation is performed on the pruned network. The original, unpruned network is used as the teacher model, and the pruned network is used as the student model for fine-tuning and training. The fine-tuned network is then subjected to fixed-point quantization to generate a lightweight inference model.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention simultaneously acquires phenotypic texture images and physiological multispectral images, and completes geometric correction, radiometric calibration, and pixel-level spatiotemporal registration to eliminate feature misalignment caused by sensor pose deviations. Through a spatiotemporal and spectral joint attention mechanism, using vegetation indices as a guiding mask to bind texture and physiological features, and fusing deep physiological indicators such as chlorophyll, water, and nitrogen with surface texture features, it can identify microinfected plants 7–14 days before the appearance of visible symptoms, increasing the early detection rate of diseases and pests by more than 45%. It can accurately distinguish between three similar physiological abnormalities: disease stress, pest stress, and soil drought, as well as light, weeds, and soil... The false detection rate caused by soil conditions is reduced by 30%, and the model stability is enhanced in complex field environments. This invention integrates image fusion features, real-time historical meteorological data, and a knowledge graph of pest and disease pathogenesis to perform graph reasoning, simulate the spread rate and range of pests and diseases, quantify the future multi-period disease evolution trend, and output a hierarchical structured early warning report, along with standardized prevention and control recommendations. It breaks through the limitation of only identifying but not predicting, advancing the early warning window period of pests and diseases by 1-2 weeks, allowing farmers and plant protection stations sufficient time for prevention and control. In cases of extremely high spread risk, it automatically pushes emergency notifications and coordinates prevention and control resources, reducing the risk of yield reduction from large-scale outbreaks. 2. Early warning reports are automatically sent to the agricultural management platform, which then automatically issues drone-based plant protection and precision irrigation / fertilization instructions based on the early warning level. After treatment, multispectral images are collected again to assess the treatment effect, and the crop health benchmark model is updated in reverse. This achieves full-chain automation of monitoring, early warning, operation, and model iteration, eliminating the need for secondary manual assessment and intervention. The accuracy of pesticide and water / fertilizer application is improved by 50%, reducing agricultural non-point source pollution caused by blind large-scale pesticide application and lowering planting costs. The pest and disease feature perception network uses phenotypic segmentation loss, physiological parameter regression loss, and cross-modal consistency loss to construct a joint loss function. During training, spatial segmentation accuracy, physiological parameter regression error, and cross-modal feature matching degree are simultaneously constrained to avoid feature shift caused by a single loss function. The trained network can stably extract pest and disease features in the field under different crops, different light conditions, and different phenological stages, improving its cross-regional and cross-crop generalization ability.

[0016] 3. After training, the deep network is sequentially processed by channel pruning, teacher-student knowledge distillation, and fixed-point quantization to remove redundant convolutional channels. With minimal loss of recognition accuracy, the model parameters are compressed by 60% and inference latency is reduced by 70%. It can be directly deployed on edge hardware such as drones and field monitoring terminals to achieve local real-time image analysis without the need for remote cloud computing, significantly reducing data transmission bandwidth and latency, and meeting the needs of online real-time early warning in the field. Morphological processing separates soil and weed background, and canopy coverage and leaf area index are introduced as auxiliary enhancement features to filter out physiological parameter interference from bare soil and weeds. The calculation results of health index and stress degree are more consistent with the actual crop growth state, further reducing the misjudgment of stress caused by complex farmland features, and the early warning results are more reliable. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of the present invention. Detailed Implementation

[0018] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0019] Example 1: Refer to Figure 1 As shown, a crop disease and pest early warning method based on multispectral fusion includes: Step 101: Acquire the first modality phenotypic image sequence and the second modality physiological image sequence of the target farmland area, and perform spatiotemporal registration processing on the first modality phenotypic image sequence and the second modality physiological image sequence to generate a standardized multimodal farmland image dataset; use an UAV-borne dual independent imaging acquisition unit to simultaneously acquire the two types of image data; the first modality phenotypic image is acquired by a global shutter RGB visible light industrial camera, with a fixed shooting frame rate of 15 frames per second, and a single frame image resolution of 4096 pixels × 3072 pixels, and the acquired content includes The first imaging modality acquires phenotypic information such as crop canopy texture, leaf wormholes, plant outline, field weeds, and bare soil surface. The second imaging modality is acquired using a five-channel pushbroom multispectral camera, covering blue, green, red, red-edge, and near-infrared light bands. Based on the reflection intensity of different bands, information related to three physiological states—crop chlorophyll, leaf water, and plant nitrogen—is extracted. Both imaging devices share the same inertial measurement unit (IMU) and satellite positioning GNSS module to synchronously output timestamps and spatial pose data, ensuring complete synchronization of the acquisition times for both types of images. After acquisition, three layers of standardized preprocessing are performed sequentially to achieve spatiotemporal registration: The first layer is geometric correction, which reads the factory-calibrated internal imaging parameters of the two cameras and the relative translation and rotation pose parameters of the gimbal, and eliminates pixel offset caused by lens barrel and pincushion distortion and gimbal installation misalignment through geometric transformation; The second layer is pixel-level feature alignment, which uses the SIFT algorithm to extract sub-pixel feature points of the two images, completes feature pairing through the FLANN fast matching algorithm, and then uses the RANSAC algorithm to remove invalid feature points with matching errors exceeding 1.5 pixels, solves the global homography transformation matrix to perform resampling interpolation on the multispectral images, and achieves a pixel-by-pixel correspondence between visible light and multispectral images; The third layer is radiometric and atmospheric correction, which uses a standard reflectance gray card to complete radiometric calibration, converts the original grayscale DN value of the camera into the true spectral reflectance of the ground surface, and then uses the 6S radiative transfer model to eliminate spectral offset caused by air aerosols and sunlight scattering. After correction, all images have a unified reflectance value range of 0 to 1, are synchronously bound with unified geographic coordinates and synchronous acquisition timestamps, and are packaged to generate a standardized multimodal farmland image dataset; The first modality phenotypic image sequence consists of continuously acquired visible light temporal images by UAVs, which primarily contain information about the external texture, color, and morphology of crops. The second modality physiological image sequence consists of synchronously acquired multispectral temporal images, which primarily contain information about the internal physiological reflection of crops. Spatiotemporal registration is a standardized preprocessing procedure that simultaneously achieves synchronization of acquisition time in the temporal dimension and precise pixel alignment in the spatial dimension. Step 102: Input the multimodal farmland image dataset into the pre-built pest and disease feature perception network. Extract cross-modal correlation features through a spatiotemporal and spectral joint attention mechanism to generate a single-channel fusion feature map. The pest and disease feature perception network adopts a dual-stream parallel independent extraction architecture, divided into a phenotypic feature extraction branch and a multispectral physiological feature extraction branch. Spatiotemporally registered and pixel-to-pixel paired visible light and multispectral images are simultaneously fed into the two branches for parallel processing. The physiological branch outputs a normalized vegetation index (NDVI) feature map as a guiding mask. Based on this mask, the adaptive attention weight of the visible light texture features is calculated pixel by pixel. The weight values ​​are simultaneously corrected by combining the spatial distribution of a single frame and the temporal changes of multiple frames. The corrected attention weights are used to perform weighted channel stitching and fusion of visible light texture features and multispectral physiological vegetation index features to suppress the feature response intensity of irrelevant background pixels such as soil and weeds, amplify the cross-modal coupling features of the pest and disease stress area, and finally compress them into a single-channel fusion feature map. The spatiotemporal and spectral joint attention mechanism is an adaptive weighted operation module that simultaneously takes into account the spatial spectral differences of a single frame and the temporal changes of consecutive frames. It only increases the feature weights of pixels corresponding to pest and disease stress and suppresses worthless background pixels. Step 103: Based on the fused feature map input to the plant stress status assessment module, calculate the current health index and stress level of the crop pixel by pixel, and determine whether to generate a disease and pest warning signal based on the health index value; The fused feature map is fed into the plant stress state assessment module. First, invalid pixels representing bare soil and weeds are removed using a morphological mask, retaining only crop canopy pixels. For each crop pixel, a pre-trained fully connected regression layer is used to calculate three quantitative physiological indicators: chlorophyll content, relative leaf water content, and total plant nitrogen content. A locally stored health benchmark database matching the corresponding crop and current growth stage is retrieved. This database stores the average standard values ​​and allowable fluctuation ranges of chlorophyll, water, and nitrogen content measured over many years in healthy, disease-free crops. The deviation of each of the three physiological indicators from the standard mean is calculated, with a weight of 0.5 for chlorophyll deviation. The weighting of leaf moisture deviation is set to 0.3, and the weighting of plant nitrogen deviation is set to 0.2. The weighted deviation values ​​of the three types are summed and normalized to obtain a health index ranging from 0 to 100. The lower the index value, the more severe the crop stress. A fixed health judgment threshold of 70 is set. When the average health index of a single area is less than or equal to 70, the crop is judged to be under stress. Then, based on the abnormal combination of the three physiological indicators, the three types of stress are distinguished as disease stress, pest stress, and drought stress. The total area of ​​the stress area, the GPS coordinates of the plot, and the average health level are simultaneously counted to generate a structured original disease and pest early warning signal. The health baseline model is a database of average physiological parameters of plants in the same plot and crop without stress, collected over a long period of time. The baseline values ​​will be adaptively updated based on the results of retesting during plant protection operations. Step 104: Read historical meteorological data, call the knowledge graph of pest and disease pathogenesis, and combine it with pest and disease early warning signals to infer the future evolution trend of pests and diseases, and output a hierarchical structured early warning report; retrieve the continuous meteorological time series data of the past 30 days stored in the IoT meteorological station of the target plot, the daily indicators include the daily average temperature, daily average relative humidity, and daily cumulative rainfall, and missing meteorological data are filled by linear interpolation of the observation values ​​of the surrounding adjacent plots during the same period; take the stress location, stress area, and stress severity level in the early warning signal as the initial trigger node of the graph, import the knowledge graph of pest and disease pathogenesis stored in the Neo4j graph database, and use the TransE graph embedding algorithm to complete the graph neural network path reasoning; the entities in the graph include various diseases, pests, suitable temperature and humidity ranges for reproduction, and the lateral spread of pests and diseases. The system integrates speed, conditions for widespread spread, and standardized plant protection plans, linking entities with logical relationships such as suitable reproduction, accelerated spread, and suppression of outbreaks. Based on meteorological conditions and map association rules, it simulates the daily horizontal spread radius, plot coverage area, and probability of a full-area outbreak for the next 7 consecutive days, distinguishing three evolutionary trends: steady spread, rapid outbreak, and growth convergence. It classifies the spread risk into three levels of early warning standards: Level 3 for slow, small-scale spread, Level 2 for rapid, widespread spread, and Level 1 for large-scale outbreaks across the entire area. The map automatically matches the corresponding level of standardized control plan, ultimately generating a structured early warning report in both PDF and JSON formats, including plot number, GPS coordinates, pest / disease type, 7-day spread prediction curve, early warning level, pesticide ratio, drone fixed-point operation route, and optimal application time window. One possible implementation: The knowledge graph can be connected to a regional agricultural meteorological real-time data interface, and the temperature and humidity correlation weights can be automatically updated daily to adapt to the climate differences of spring, summer, autumn and winter.

[0020] This step integrates two types of image information: visible light appearance and multispectral physiological information. It relies on spatiotemporal spectral joint attention to mine subtle early-stage pest and disease characteristics that are not visible to the naked eye, quantifies the severity of stress through physiological parameters, and predicts the risk of pest and disease spread based on mechanistic knowledge graphs. It realizes a fully automated closed-loop operation from field image acquisition to graded plant protection early warning reports, solving the existing technical defects of single-spectrum identification that misses detections and cannot predict the outbreak of large-scale pests and diseases in advance.

[0021] In some embodiments, a multimodal farmland image dataset is input into a pre-constructed pest and disease feature perception network, and cross-modal correlation features are obtained through a spatiotemporal and spectral joint attention mechanism to generate a fused feature map. The complete execution steps are as follows: Step 201: Extract the texture features and color distribution features of the first modality visible light phenotypic image through the phenotypic feature extraction branch inside the disease and insect feature perception network; The phenotypic extraction branch uses a lightweight MobileNetV2 network as its backbone computational unit; the shallow convolutional layers of the network are responsible for extracting local texture features such as leaf holes, spots, and plant edge gradients; the deep bottleneck module extracts the global hue and color block distribution features of the entire canopy; the branch outputs multi-scale phenotypic feature maps, with a uniform channel number of 64, and the image resolution is compressed to one-quarter of the original visible light image. Texture features are local grayscale gradients and object edge contours; color features are pixel color distribution vectors after RGB three-channel normalization. Step 202: Extract the Normalized Difference Vegetation Index (NDVI) and Water Stress Index (NDWI) features from the second modality multispectral image through the physiological feature extraction branch. The physiological branch performs convolution operations on the five-channel multispectral image band by band to calculate two core spectral features: NDVI (Normalized Difference Vegetation Index) and NDWI (Natural Difference Water Index). The NDVI index is used to quantify the chlorophyll content of leaves, and the NDWI index is used to quantify the degree of water deficiency in leaves. The two types of index features are spliced ​​and integrated in the channel dimension to generate a physiological basic feature map with 32 channels and spatial resolution that is completely consistent with the phenotypic feature map output in step 201. Step 203: Build a spatiotemporal and spectral joint attention operation process, generate a guiding mask with NDVI vegetation index features, and calculate the adaptive attention weights corresponding to visible light texture features pixel by pixel. All pixel values ​​in the NDVI vegetation index feature map are uniformly scaled to the range of 0 to 1 to generate a guide mask image. High-value pixels in the mask correspond to healthy crop areas, while low-value pixels correspond to crop areas under stress. The spatial similarity between the mask value and the visible light texture features at the corresponding location is calculated pixel by pixel, and the similarity calculation result is directly used as the initial attention weight of that pixel. Then, a temporal constraint of multiple consecutive frames is introduced. If the mask value of the same spatial location is low for multiple consecutive frames, the final attention weight of that pixel is increased synchronously. The weight value ranges from 0 to 1, with the weight of soil, weed and bare soil areas suppressed to the range of 0 to 0.2, and the weight of disease and pest stress areas increased to the range of 0.7 to 1.0. The guide mask is a weighted image generated by mapping the NDVI vegetation index, used to automatically distinguish field crop pixels from weed and bare soil background pixels; Step 204: Use the calculated attention weights to perform a weighted fusion operation on the texture features and vegetation index features, and output a single-channel fused feature map; Each pixel in each channel of the visible light phenotypic feature map is multiplied by the attention weight calculated at the corresponding position to complete weighted suppression. The weighted phenotypic feature map and the physiological vegetation index feature map are completely concatenated in the channel dimension. Then, a 1×1 convolutional layer is used to perform channel dimensionality reduction compression, and the final output is a fused feature map that retains only a single channel. An alternative implementation: The spatiotemporal and spectral joint attention module is deployed in the shallow layer of the network, which can directly filter out irrelevant background features such as soil and weeds in the early stage, and significantly reduce the overall computational load of the subsequent deep network.

[0022] This step employs a dual-branch approach to independently extract two types of crop features: appearance and physiology. Using vegetation indices as prior information, an adaptive attention mask is generated to automatically amplify the feature response intensity in disease and pest stress areas. This achieves precise coupling of the two modalities, significantly improving the identification of crop micro-level erosion and early-stage pest infestations.

[0023] In some embodiments, the plant stress status assessment module determines the current crop health index and stress level based on the fused feature map, and generates pest and disease early warning signals based on the health index and stress level. The complete execution steps are as follows: Step 301: Select effective pixels of crop canopy from the fused feature map, and extract three types of quantitative physiological parameters, namely chlorophyll, leaf water and plant nitrogen, pixel by pixel; First, morphological masking is performed on the fused feature map to remove pixels of bare soil and weeds, retaining only pixels of continuous crop canopy. For each effective crop pixel, three indicators are calculated through a pre-trained fully connected physiological regression layer: chlorophyll content (mg / g), leaf relative water content percentage, and plant total nitrogen content (mg / kg). Non-crop background pixels are not included in the physiological parameter calculation. Step 302: Compare the pixel-by-pixel physiological parameters with the preset health benchmark model, and calculate the global average health index using weighted normalization. The health baseline model stores the average standard values ​​of chlorophyll, water, and nitrogen for the corresponding crop and for healthy plants without diseases or pests at the current growth stage, while also recording the allowable normal fluctuation range for each indicator. It calculates the deviation of the measured chlorophyll value, water value, and nitrogen value relative to the standard mean for all crop pixels in the plot. The deviation is multiplied by a weighting factor of 0.5 for chlorophyll, 0.3 for water, and 0.2 for nitrogen, and then summed. The total deviation is then linearly normalized to obtain a health index ranging from 0 to 100, with higher index values ​​indicating healthier crops. Step 303: Compare the health index with the preset health threshold to distinguish the three types of stress; The system sets a fixed health assessment threshold of 70. When the average health index of the entire plot is less than or equal to 70, the crops in that area are considered to be under stress. The stress category is distinguished based on the abnormal combination of three physiological indicators: a significant decrease in chlorophyll and a slight decrease in water content indicate disease stress; a slight decrease in chlorophyll and a sudden change in local texture and spots indicate pest stress; and stable chlorophyll values ​​and a significant decrease in water content indicate drought stress. Step 304: Divide the stress levels according to the health index range and generate standardized pest and disease early warning signals; A health index range of 70 to 50 is defined as mild stress, 50 to 30 as moderate stress, and 30 and below as severe stress. The warning signal contains complete structured data including the geographic coordinates of the plot, the total pixel area of ​​the stressed region, the stress category, the stress level, and the average health index of the entire region, which can be directly read and parsed by agricultural platforms and plant protection systems. This step relies on a weighted average of three crop physiological indicators to calculate a health index, accurately distinguishing between three easily confused stress states: diseases, pests, and drought. This avoids misjudgments caused by a single spectral indicator and outputs a machine-readable standardized early warning signal, providing standardized input data for inferring the spread trend of diseases and pests.

[0024] In some embodiments, based on pest and disease early warning signals, combined with historical meteorological data and a pre-constructed knowledge graph of pest and disease pathogenesis, the evolution trend of pests and diseases is predicted, and a structured early warning report containing early warning levels and control recommendations is generated. The complete execution steps are as follows: Step 401: Collect continuous historical meteorological time series data for the target farmland area; Read the observation data of the past 30 days stored in the IoT meteorological data acquisition device of the plot. The daily recorded indicators include the daily average temperature, daily average relative humidity and daily cumulative rainfall. If there is a missing meteorological data for a single day, take the observation values ​​of the two consecutive days before and after the plot and perform linear interpolation to fill in the missing data, and generate a complete meteorological feature vector. Step 402: Input the pest and disease early warning signal and meteorological feature vector into the knowledge graph of pest and disease pathogenesis and perform graph reasoning to simulate the spread speed and coverage area of ​​pests and diseases; The knowledge graph is stored in a graph database. Entities include various diseases and pests, suitable temperature and humidity ranges for diseases and pests, lateral spread speed, conditions for contiguous spread, and supporting plant protection solutions. The stress location, stress area, stress level, and meteorological feature vector within the warning signal are used as input nodes for graph inference. Graph neural network path matching inference is used to retrieve the association rules between entities for suitable reproduction and accelerated spread, and to simulate the daily lateral spread radius of diseases and pests, the area of ​​land covered by diseases and pests, and the probability of a full-area outbreak in the next 7 days. Step 403: Determine the overall evolution trend of pests and diseases based on daily spread area data; The system has a 7-day forecast period and outputs the coverage area of ​​diseased and pest-infested plots daily. Based on the area growth curve, it classifies the trend into three categories: a daily increase of less than 5% indicates stable spread, an increase of 5% to 20% indicates rapid spread, and an increase of less than 1% for 3 consecutive days indicates convergence of growth. Step 404: Classify the early warning level according to the spread trend, match the corresponding prevention and control strategy, and generate an early warning report; Early warning classification criteria: stable, small-scale spread is classified as Level 3; rapid, contiguous spread is classified as Level 2; and the risk of a large-scale outbreak across the entire region is classified as Level 1. Standardized plant protection plans for each level are pre-bound within the map: Level 3 warnings only involve localized, targeted spraying using drones; Level 2 warnings involve spraying in different areas and zones; and Level 1 warnings initiate comprehensive, unified prevention and control operations across the entire region. A structured early warning report is generated, including plot number, GPS coordinates, pest / disease type, 7-day spread prediction curve, warning level, pesticide type and ratio, drone flight path, and optimal application time window. The report also supports dual output formats: JSON (machine-readable) and PDF (human-readable). One possible implementation: The knowledge graph can be connected to a regional real-time weather push interface, and the temperature and humidity correlation weights can be automatically updated daily to adapt to seasonal climate changes.

[0025] This step integrates lesion early warning information, long-term meteorological data, and disease and pest growth mechanism maps to perform multi-dimensional joint reasoning, quantifies the future outbreak scale of diseases and pests, and matches precise plant protection solutions at different levels, achieving integrated automatic output from disease and pest identification to field control decision-making.

[0026] In some embodiments, spatiotemporal registration is performed between a first modality phenotypic image sequence and a second modality physiological image sequence to generate a multimodal farmland image dataset. The complete execution steps are as follows: Step 501: Based on the relative pose parameters of the dual imaging devices, perform geometric distortion correction on the visible light and multispectral images respectively; Two sets of cameras were calibrated in advance using a standard checkerboard calibration board to obtain the internal focal length, principal point, and radial distortion coefficient of each of the visible light and multispectral cameras. At the same time, the relative translation and rotation extrinsic parameter transformation matrices caused by the gimbal installation were obtained. For a single frame image, the radial distortion model was first used to eliminate the barrel and pincushion distortion of the lens. Then, the spatial basic alignment of the two images was completed through the extrinsic parameter transformation matrix to eliminate the pixel misalignment problem caused by the gimbal installation offset. Step 502: Use a feature matching algorithm to achieve pixel-level precise alignment of the two images; Sub-pixel precision feature points are extracted from the corrected visible light and multispectral images using the SIFT algorithm. Feature pairing is performed using the FLANN fast matching algorithm, and invalid feature points with matching errors exceeding 1.5 pixels are removed using the RANSAC algorithm. The global homography transformation matrix is ​​solved based on the effective matching points, and the multispectral image is interpolated and resampled to achieve a one-to-one spatial correspondence between the two images pixel by pixel. Step 503: Perform radiometric calibration and atmospheric correction on the pixel-aligned image to generate a standardized multimodal dataset; In the radiometric calibration stage, standard reflective gray plates are used to calibrate parameters and convert the original grayscale DN values ​​of the camera into the true spectral reflectance of the land surface. In the atmospheric correction stage, the 6S radiative transfer model is used to eliminate spectral shifts caused by air aerosols and sunlight scattering. After the correction is completed, the reflectance of all images is uniformly controlled in the range of 0 to 1, and uniform geographic coordinates and acquisition timestamps are bound simultaneously to generate a standardized multimodal farmland image dataset. This step sequentially completes three layers of preprocessing: geometric distortion correction, sub-pixel feature registration, and spectral radiometric correction. This completely eliminates data deviations caused by dual-camera hardware installation and outdoor lighting and atmosphere, ensuring that pixels at the same crop location correspond perfectly in visible light and multispectral light, thus providing high-precision raw data for subsequent cross-modal feature fusion.

[0027] In some embodiments, the complete training process for the pest and disease feature perception network is as follows: Step 601: Construct a set of labeled historical farmland image samples; A total of 120,000 sets of visible light-multispectral images were collected, covering various crops at different growth stages, with varying degrees of pest and disease severity, and drought stress. Pixel-level segmentation and annotation were manually completed, with annotation categories including healthy crops, diseased areas, pest-infested areas, soil, and weeds. Simultaneously, field measurements of chlorophyll, water, and nitrogen levels were collected for corresponding plots as physiological regression labels. The dataset was divided into three parts: 80% training samples, 15% validation samples, and 5% test samples. Step 602: Construct a joint loss calculation rule for the three objectives, without using mathematical formulas, and provide a complete textual description of the total loss calculation logic; The total loss is obtained by weighted summation of three parts: phenotypic segmentation loss, physiological parameter regression loss, and cross-modal feature consistency loss. The phenotypic segmentation loss is calculated using the Dice loss, comparing the network-predicted segmentation labels with the manually labeled real labels pixel by pixel; the lower the pixel matching degree, the larger the segmentation loss. The physiological parameter regression loss is calculated using the mean squared error (MSE), comparing the network-output physiological values ​​with the actual physiological values ​​measured in the field; the larger the deviation, the larger the regression loss. The cross-modal consistency loss is calculated by the cosine distance between the visible light feature vector and the multispectral physiological feature vector at the same crop pixel location; the larger the vector difference, the larger the consistency loss. The total loss is calculated as follows: total loss equals phenotypic segmentation loss + 0.4 × physiological regression loss + 0.3 × cross-modal consistency loss; where 0.4 and 0.3 are fixed balancing weights used to balance the impact of the three losses on network training. Step 603: Iteratively train the pest and disease feature perception network end-to-end based on the joint loss rule; The optimizer uses the AdamW optimization algorithm, with an initial learning rate set to 0.01% and a maximum of 120 iterations. The learning rate is decayed every 10 iterations with a decay coefficient of 0.9. After each training round, the total loss is calculated using the validation set. If the total loss on the validation set does not decrease for 10 consecutive rounds, the early stopping mechanism is triggered to terminate the training. After training, the weight parameter files of all convolutional and fully connected layers of the network are fully stored. This step employs a multi-task joint loss mechanism to simultaneously optimize crop segmentation accuracy, physiological parameter regression accuracy, and bimodal feature matching consistency. This avoids network feature shift caused by a single loss function and enhances the network's generalization and recognition capabilities in complex field scenarios involving weeds and bare soil.

[0028] In some embodiments, according to the method of claim 1, after generating a structured early warning report containing an early warning level and prevention recommendations, the following additional steps are performed: Step 701: Transmit the standardized structured early warning report to the cloud-based agricultural management platform; The platform transmits JSON format early warning reports via 4G / 5G wireless data links and automatically stores a full-cycle pest and disease early warning ledger for a single plot, retaining historical early warning records. Step 702: The agricultural management platform automatically issues corresponding plant protection operation instructions based on the early warning level; A Level 3 stable diffusion warning issues instructions for drones to spray at specific locations; a Level 2 rapid diffusion warning issues instructions for plant protection operations in different areas; a Level 1 widespread outbreak warning dispatches multiple drone clusters for unified spraying across the entire area; if the warning type is drought stress, it automatically issues instructions for timed water-saving irrigation and precision fertilization control. Step 703: After the plant protection operation is completed, re-measure the images, evaluate the control effect, and adaptively update the health baseline model; Seven days after the completion of plant protection spraying and irrigation operations, the drone once again collected a complete multimodal image dataset of the target plot; the average health index of the plot was recalculated across the entire area, and if the health index rose to 80 or above, the control was deemed effective; the average physiological values ​​of crop chlorophyll, water and nitrogen obtained from this retest were synchronously updated to the plot's exclusive health benchmark model, adaptively correcting the benchmark mean and fluctuation range to adapt to the long-term natural changes in crop growth.

[0029] This step forms a fully automated closed-loop management system encompassing image recognition, tiered early warning, automatic plant protection, retesting and evaluation, and adaptive model updates. It reduces manual judgment and intervention, continuously optimizes health assessment criteria, and lowers the probability of false alarms caused by long-term field environment drift.

[0030] In some embodiments, before determining the current crop health index and stress level using the plant stress status assessment module based on the fused feature map, the following additional steps are performed: Step 801: Perform morphological filtering on the fused feature map to separate the soil and weed background pixels; A combination of morphological operations, namely 3× size erosion and 5× size dilation, is used to generate a binary crop mask image. White pixels in the mask represent crop areas and black pixels represent the background. Small weed patches with an area of ​​less than 200 pixels and large areas of bare soil are removed, and only continuous patches of crop canopy pixels are retained for subsequent calculations. Step 802: Calculate the canopy coverage and leaf area index (LAI) of the plots based on the mask classification results; Canopy coverage is calculated as follows: the total number of effective crop pixels within the mask is divided by the total number of all effective pixels in the entire image to obtain the coverage in percentage form; the leaf area index (LAI) is calculated based on the NDVI multispectral index inversion, with a value ranging from 0 to 6, and the larger the value, the denser the plant leaves. Step 803: Use canopy coverage and leaf area index as auxiliary features to splice together an enhanced feature map; The single-channel canopy cover feature map and the single-channel LAI leaf area feature map are spliced ​​and integrated with the original single-channel fused feature map in the channel dimension to generate a three-channel enhanced feature map, which supplements the auxiliary information of crop density growth. Step 804: Recalculate the plot health index and stress level based on the enhanced feature map; The three-channel enhanced feature map is fed into the physiological regression layer. Canopy coverage and leaf area index are used as correction coefficients to participate in the calculation of the three physiological indicators, eliminating the calculation deviation of physiological indicators caused by sparse plants and dense leaves, and outputting a more accurate global health index and stress level after correction. This step filters out interference from weeds and bare soil through morphology, introduces auxiliary features such as crop canopy and leaf area to correct physiological parameters, eliminates misjudgments of health index caused by differences in plant density, and improves the accuracy of stress identification.

[0031] In some embodiments, after predicting the evolution trend of pests and diseases, the following additional steps are performed: Step 901: Compare the predicted spread area of ​​pests and diseases over 7 days with the preset safety threshold to determine whether an emergency warning has been triggered; The system presets a safety threshold of 30% of the total area covered by pests and diseases in a single plot; if the model predicts that the proportion of the area covered by pests and diseases is greater than or equal to 30% after 7 days, it determines that there is a risk of a large-scale outbreak and generates an emergency warning signal with an emergency indicator. Step 902: The emergency warning signal triggers the platform's complete set of emergency response strategies; After receiving an emergency signal, the agricultural management platform automatically sends text messages and APP pop-up warnings to farmers in neighboring plots; at the same time, it issues resource coordination instructions to the regional plant protection service terminal to uniformly dispatch plant protection drones, pesticides, and field workers to achieve simultaneous and unified prevention and control of contiguous plots and block the spread of diseases and pests across plots. This step sets a safe threshold for determining the spread of pests and diseases, and initiates multi-farm joint emergency control for high-risk areas with large-scale outbreaks, so as to prevent the spread of pests and diseases in advance and reduce the loss of grain production.

[0032] In some embodiments, after the pest and disease feature perception network is trained, it is deployed to farmland monitoring equipment. The lightweight and complete execution steps are as follows: Step 1001: Read the complete trained original disease and pest feature perception network file; Read the complete network structure file and all convolutional and fully connected layer weight parameter files saved after the training iterations; the total number of parameters in the original complete network is approximately 12.8 million. Step 1002: Perform convolutional channel pruning based on the channel contribution threshold; The preset threshold for determining channel contribution is 0.02; iterate through each output channel of all convolutional layers in the network, calculate the average activation value of all pixels in the output feature map of that channel, and define the average value as the channel contribution; if the contribution of a single channel is less than or equal to 0.02, it is determined to be a worthless redundant channel, and all convolutional weight parameters corresponding to that channel are directly deleted; after pruning, the total number of network parameters is compressed to 5.1 million. Step 1003: Distillation of knowledge between teachers and students, fine-tuning and pruning the network; The original, unpruned complete network was used as the teacher model, and the pruned, lightweight network was used as the student model. The distillation loss included both the hard label loss from real human annotations and the soft label loss from the teacher network output. The total number of fine-tuning iterations was set to 30 rounds to compensate for the decrease in feature extraction capability caused by channel pruning with a small loss of accuracy. In actual tests, the decrease in recognition accuracy after fine-tuning did not exceed 2.5%. Step 1004: 8-bit fixed-point quantization generates an airborne lightweight inference model; All weights and feature floating-point values ​​of the network after distillation fine-tuning are subjected to 8-bit integer fixed-point quantization, converting 32-bit floating-point numbers into 8-bit integers for storage; the overall storage volume of the model is compressed by 75% after quantization; the time for complete inference of a single frame of multimodal image is reduced from the original 120 milliseconds to 38 milliseconds, and it can be directly deployed on a drone-borne Jetson edge computing board to achieve real-time online analysis in the field; An alternative implementation: The entire lightweight process of channel pruning, knowledge distillation, and point-to-point quantization can be completed offline in batches, and the lightweight model is directly solidified into the drone's onboard firmware, eliminating the need for cloud transmission and computation during field operations; This step sequentially completes three levels of lightweight processing: channel pruning, teacher and student knowledge distillation, and 8-bit fixed-point quantization. Under the premise of a slight decrease in recognition accuracy, it significantly compresses the model size and reduces single-frame inference latency, making it suitable for local real-time inference scenarios of drones and small edge monitoring devices in the field.

[0033] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for early warning of crop diseases and pests based on multispectral fusion, characterized in that, include: The first modality phenotypic image sequence and the second modality physiological image sequence of the target farmland area are acquired, and the first modality phenotypic image sequence and the second modality physiological image sequence are spatiotemporally registered to generate a multimodal farmland image dataset. The multimodal farmland image dataset is input into a pre-constructed pest and disease feature perception network. Cross-modal correlation features are obtained through a spatiotemporal and spectral joint attention mechanism to generate a fused feature map. Based on the fused feature map, the plant stress status assessment module is used to determine the current crop health index and stress level, and disease and pest early warning signals are generated based on the health index and stress level. Based on pest and disease early warning signals, combined with historical meteorological data and a pre-constructed knowledge graph of pest and disease pathogenesis, the evolution trend of pests and diseases is predicted, and a structured early warning report containing early warning levels and control recommendations is generated.

2. The method for early warning of crop diseases and pests based on multispectral fusion according to claim 1, characterized in that, A multimodal farmland image dataset is input into a pre-constructed pest and disease feature perception network. Cross-modal correlation features are obtained through a spatiotemporal and spectral joint attention mechanism to generate a fused feature map, including: By utilizing the phenotypic feature extraction branch in the pest and disease feature perception network, texture and color features in the first modality phenotypic image sequence are obtained; Physiological feature extraction branches were used to obtain vegetation index features and water stress features in the second modality physiological image sequence; By using a spatiotemporal and spectral joint attention mechanism, vegetation index features are used as guiding masks to determine the attention weights in texture features that have spatial correspondence with vegetation index features. Attention weights are used to weight and fuse texture features and vegetation index features to generate a fused feature map.

3. The method for early warning of crop diseases and pests based on multispectral fusion according to claim 1, characterized in that, Based on the fused feature map, the plant stress status assessment module determines the current crop health index and stress level, and generates pest and disease early warning signals based on the health index and stress level, including: Physiological parameters of each crop plant region, including chlorophyll content, water content and nitrogen content, are obtained from the fused feature map. Physiological parameters are compared with a preset health benchmark model to determine the health index; When the health index is not higher than the preset health threshold, a stress state is determined, and the stress type is determined based on the abnormal combination of physiological parameters. The stress type includes disease stress, pest stress, or environmental stress. Based on the type and degree of stress, corresponding pest and disease early warning signals are generated.

4. The method for early warning of crop diseases and pests based on multispectral fusion according to claim 1, characterized in that, Based on pest and disease early warning signals, combined with historical meteorological data and a pre-constructed knowledge graph of pest and disease pathogenesis, the evolution trend of pests and diseases is predicted, and a structured early warning report containing warning levels and control recommendations is generated, including: Obtain historical meteorological data for the target farmland area, including temperature, humidity, and rainfall; Using pest and disease early warning signals as the initial triggering condition, and combining them with historical meteorological data, the data is input into a pre-constructed knowledge graph of pest and disease pathogenesis to perform graph reasoning, thereby simulating the spread rate and range of pests and diseases under the current environmental conditions. Based on the rate of spread and the extent of diffusion, predict the evolution trend of pests and diseases within a predetermined time period in the future; Based on the evolution trend, the warning level is determined, and the corresponding prevention and control strategies are matched to generate a structured warning report.

5. The method for early warning of crop diseases and pests based on multispectral fusion according to claim 1, characterized in that, Spatiotemporal registration was performed on the first modality phenotypic image sequence and the second modality physiological image sequence to generate a multimodal farmland image dataset, including: Based on the relative pose parameters of the phenotypic sensor and the physiological sensor, geometric correction is performed on the first modality phenotypic image sequence and the second modality physiological image sequence; The corrected first modality phenotypic image sequence and the second modality physiological image sequence were aligned at the pixel level using a feature point matching algorithm. Radiometric calibration and atmospheric correction are performed on the aligned images to generate a standardized multimodal farmland image dataset.

6. The method for early warning of crop diseases and pests based on multispectral fusion according to claim 1, characterized in that, The training process of the disease and pest feature perception network includes: Acquire a historical farmland image sample set and label each sample with pixel-level pest and disease tags and physiological parameter tags; Construct a joint loss function that includes phenotypic segmentation loss, physiological parameter regression loss, and cross-modal consistency loss; The disease and pest feature perception network is trained end-to-end using a joint loss function. While minimizing the phenotypic segmentation error, the semantic consistency between phenotypic and physiological features in the disease and pest region is constrained, resulting in the trained disease and pest feature perception network.

7. The method for early warning of crop diseases and pests based on multispectral fusion according to claim 1, characterized in that, After generating a structured early warning report that includes the warning level and prevention recommendations, it also includes: The structured early warning report will be sent to the agricultural management platform; The agricultural management platform automatically triggers corresponding prevention and control instructions based on the early warning level. These instructions include drone plant protection operation instructions or irrigation and fertilization instructions. A multimodal farmland image dataset is acquired after pest control operations to assess the effectiveness of pest control. The agricultural management platform then updates the preset health baseline model based on the assessment results.

8. The method for early warning of crop diseases and pests based on multispectral fusion according to claim 1, characterized in that, Before determining the current crop health index and stress level based on the fused feature map using the plant stress status assessment module, the following steps are also included: Morphological processing was performed on the fused feature map to separate the soil background from the weed area; Based on the pixel classification results in the fused feature map, the canopy coverage and leaf area index of crop plants are determined. Canopy coverage and leaf area index are used as auxiliary features and spliced ​​with the fused feature map to generate an enhanced feature map; Health indices and stress levels are determined based on enhanced feature maps.

9. A method for early warning of crop diseases and pests based on multispectral fusion according to claim 1, characterized in that, After predicting the evolution trend of pests and diseases, it also includes: When the prediction results show that the spread of pests and diseases is not less than the preset safety threshold, an emergency warning signal is generated. The emergency warning signal triggered the agricultural management platform to activate its emergency response strategy, which included sending warning notices to surrounding farmers and coordinating prevention and control resources.

10. A method for early warning of crop diseases and pests based on multispectral fusion according to claim 6, characterized in that, Before being applied to farmland monitoring equipment, the pest and disease characteristic sensing network also included: Obtain the trained pest and disease feature perception network obtained after iterative training according to the joint loss function; Channel pruning is performed on the trained pest and disease feature perception network to remove redundant channels whose contribution to the fused feature map is not higher than a preset contribution threshold. Knowledge distillation is performed on the pruned network. The original, unpruned network is used as the teacher model, and the pruned network is used as the student model for fine-tuning and training. The fine-tuned network is then subjected to fixed-point quantization to generate a lightweight inference model.

Images