Real-time identification and prevention decision-making method and system for crop diseases and insect pests based on multimodal edge computing

Abstract

The present invention relates to the field of computer vision, specifically a method and system for real-time identification and control of crop pests and diseases based on multimodal edge computing. The system comprises: acquiring drone multispectral imagery, infrared thermal imaging, and the NDVI index, preprocessing them in combination with field IoT sensor data to generate a multimodal feature input sequence; constructing a spectral-environmental fusion feature matrix based on the Transformer attention mechanism, and utilizing the YOLOv7-Spectral model for pest and disease detection to generate a pest and disease distribution heat map; calculating the pest risk index (BRI) and pesticide application priority map based on historical data; employing an ant colony optimization algorithm to plan drone application paths and optimizing the application plan based on wind speed and humidity parameters; accurately applying pesticides along the optimized paths by the drones and monitoring disease trends; adjusting the model based on application feedback data, and optimizing pest and disease control strategies using federated learning. The present invention improves recognition accuracy, reduces pesticide use, and achieves precise, efficient, and intelligent control.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and system for real-time identification, prevention and control of crop diseases and insect pests based on multimodal edge computing. Background Art

[0002] Agricultural pest control is a crucial component of ensuring food security. Traditional crop pest monitoring relies primarily on manual inspections and single-image recognition methods, resulting in low monitoring efficiency, significant environmental impact, and excessive pesticide use. Furthermore, cloud computing-based remote pest identification methods struggle to meet the real-time control needs of large-scale farmland due to data transmission delays. Therefore, efficiently integrating multimodal data on edge computing devices, improving real-time pest identification accuracy, and optimizing drone-based precision pesticide application strategies are key technical challenges in smart agricultural prevention and control.

[0003] Existing pest and disease monitoring methods are mostly based on single RGB images or hyperspectral data, which are significantly affected by environmental factors such as lighting and occlusion, making it difficult to ensure stable pest and disease identification in complex farmland environments. Furthermore, traditional pesticide application methods rely on fixed or empirical drone spraying paths, failing to integrate disease risk assessment with meteorological parameters for optimization. This results in pesticide waste, environmental pollution, and reduced control efficiency.

[0004] To address these challenges, this invention integrates drone multispectral imagery, infrared thermal imaging, and field IoT sensor data to extract spectral reflectance, leaf temperature, NDVI index, and environmental factors. It then utilizes the Transformer attention mechanism to construct a spectral-environmental fusion feature matrix. This system, based on a lightweight YOLOv7-Spectral model, enables efficient pest and disease target detection on edge computing devices. Furthermore, this invention assesses disease severity using the Disease Risk Index (BRI) and uses the Ant Colony Optimization (ACO) algorithm to generate precise drone application paths. Furthermore, it employs a federated learning mechanism to optimize pest and disease identification models, enabling cross-farm data sharing and intelligent upgrades to disease prevention and control strategies. Summary of the Invention

[0005] The present invention provides a real-time identification and control decision-making method and system for crop diseases and insect pests based on multimodal edge computing. The method aims to solve the problem of how to build a lightweight YOLOv7-Spectral model based on drone multispectral imagery, infrared thermal imaging data, and field Internet of Things sensor data, integrating spectral reflectance, leaf temperature, NDVI index, and environmental factors. The model can realize real-time identification of pest targets on edge computing devices, and plan the precise spraying path of drones in combination with the ant colony optimization algorithm to improve the accuracy of pest identification and reduce the amount of pesticide used. The method also optimizes the control model through federated learning to achieve cross-farm data sharing and intelligent upgrading of disease control strategies.

[0006] In order to solve the above technical problems, the present invention provides a real-time identification and prevention and control decision-making method for crop diseases and insect pests based on multimodal edge computing, comprising:

[0007] A method for real-time identification and prevention of crop pests and diseases based on multimodal edge computing, characterized in that the method comprises the following steps:

[0008] Acquire multispectral image data, infrared thermal imaging data, and NDVI index collected by drones, and obtain temperature, humidity, wind speed, and soil moisture data from field IoT sensors, and synchronously preprocess the data to generate a multimodal feature input sequence;

[0009] Normalizing the multimodal feature input sequence, and performing multidimensional feature mapping based on the Transformer attention mechanism to generate a spectral-environment fusion feature matrix;

[0010] Based on the spectral-environmental fusion feature matrix and the pre-trained YOLOv7-Spectral model, pest and disease target detection is performed, pest and disease distribution heat maps are generated, and confidence screening is performed on the recognition results to extract pest and disease characteristic areas;

[0011] Extracting spatial location features from the pest and disease distribution heat map and data obtained from field IoT sensors, and combining them with historical disease trend data to assess disease severity and generate a disease risk index (BRI) and a pesticide application priority map;

[0012] Based on the pesticide application priority map and the drone operation parameters, an ant colony optimization path planning model is constructed, and combined with wind speed and humidity parameters, an optimized path sequence for drone pesticide application is generated;

[0013] Obtaining the optimized path sequence for drone spraying, and controlling the drone to perform precise spraying according to the optimized path, while monitoring meteorological data and real-time images of the spraying area, and updating the disease change trend sequence;

[0014] Based on the disease change trend sequence and post-application feedback data, the parameter weights of the YOLOv7-Spectral model are adjusted, and the disease identification and application path planning models are optimized through federated learning to generate an updated intelligent prevention and control model;

[0015] The disease risk index (BRI) is calculated by the following formula: ;

[0016] in, For Grid Disease risk index at the location;

[0017] For Grid Disease severity index at the site;

[0018] For Grid Environmental data of the location;

[0019] is the weight of disease and environmental factors.

[0020] Furthermore, the preprocessing of the multimodal feature input sequence includes the following steps:

[0021] interpolation and noise removal are performed on the multispectral image data, infrared thermal imaging data, and NDVI index to generate standardized multispectral image sequences and infrared thermal imaging sequences;

[0022] Perform time alignment and spatial registration on field IoT sensor data, and perform data normalization to generate standardized environmental data sequences;

[0023] Data fusion is performed on the standardized multispectral image sequence, infrared thermal imaging sequence and environmental data sequence to generate a preliminarily processed multimodal feature input sequence.

[0024] Furthermore, the construction of the multidimensional feature map includes the following steps:

[0025] Performing feature decomposition on the multimodal feature input sequence to extract spectral reflectance, leaf surface temperature, NDVI index and environmental factors;

[0026] The Transformer attention mechanism is used to perform feature mapping on the spectral reflectance, leaf surface temperature, NDVI index and environmental factors to generate a spectral-environment fusion feature matrix.

[0027] Furthermore, the target detection of pests and diseases includes the following steps:

[0028] Inputting the spectrum-environment fusion feature matrix into the pre-trained YOLOv7-Spectral model to perform pest and disease target detection and generate pest and disease target detection results;

[0029] Based on the pest and disease target detection results, distribution mapping of the detection targets is performed according to spatial positions to generate a pest and disease distribution heat map;

[0030] The pest and disease distribution heat map is screened for confidence, low-confidence targets are eliminated, and pest and disease characteristic area data are extracted.

[0031] Furthermore, the disease severity assessment comprises the following steps:

[0032] Based on the regional data of pest and disease characteristics and the data obtained by field IoT sensors, the spatial location characteristics of the target crop area are extracted to obtain the crop spatial distribution sequence;

[0033] Based on the crop spatial distribution sequence and combined with historical disease trend data, the disease severity index is calculated and a disease risk index (BRI) is generated.

[0034] Furthermore, the optimized path sequence for drone pesticide application includes the following steps:

[0035] Based on the pesticide application priority map and the UAV operation parameters, an ant colony optimization path planning model is constructed to generate an initial UAV pesticide application path plan;

[0036] The initial drone spraying path plan is optimized by combining wind speed and humidity parameters, and the spraying angle is adjusted to generate an optimized drone spraying path sequence;

[0037] The optimized UAV spraying path sequence is smoothed, and the spraying height is adjusted to generate a final precise spraying path.

[0038] Furthermore, controlling the drone to perform precise spraying according to the optimized path includes the following steps:

[0039] Based on the UAV operation parameters, the UAV is controlled to perform precise spraying according to the UAV optimized spraying path, while recording the spraying location and pesticide dosage data;

[0040] During the application process, meteorological data and real-time images of the application area are obtained, and the data are compared with historical data to obtain a sequence of disease change trends;

[0041] Based on the disease change trend sequence, disease changes are predicted and disease severity assessment data is updated.

[0042] Furthermore, adjusting the parameter weights of the YOLOv7-Spectral model includes the following steps:

[0043] Obtain post-application feedback data, including drone operation data, crop health status, and farmland environment data, to form a feedback sequence on the application effect;

[0044] Based on the pesticide application effect feedback sequence, the parameter weights of the multimodal feature mapping model and the YOLOv7-Spectral model are adjusted to generate an optimized pest and disease recognition model.

[0045] Furthermore, the federated learning optimization prevention and control model includes the following steps:

[0046] Based on pest and disease data from multiple farms, the local update parameters of each sub-model are calculated and the model is trained independently on each terminal device;

[0047] Based on the local update parameters, the global model parameters are calculated using a federated averaging algorithm, model aggregation is performed, and an updated intelligent prevention and control model is generated.

[0048] A real-time identification and prevention decision-making system for crop diseases and insect pests based on multimodal edge computing, applied to any of the above-mentioned real-time identification and prevention decision-making methods for crop diseases and insect pests based on multimodal edge computing, comprising:

[0049] Data acquisition module, used to obtain monitoring data related to crop growth and pests and diseases from multimodal data sources;

[0050] The data processing and fusion module is used to pre-process the collected multimodal data and perform feature mapping on the multimodal data using the Transformer attention mechanism to generate a spectral-environment fusion feature matrix;

[0051] Pest and disease recognition module, which is used for real-time target detection of crop pests and diseases based on the YOLOv7-Spectral deep learning model;

[0052] The disease risk assessment module is used to calculate the disease severity index based on the pest distribution heat map, field environmental sensor data and historical disease trend data, and further generate the disease risk index (BRI);

[0053] The drone spraying path optimization module is used to plan the optimal spraying path based on the disease risk index and drone operation parameters using the ant colony optimization algorithm;

[0054] The drone spraying execution and data monitoring module is used to control the drone to carry out precise spraying according to the optimized path, and record the spraying location and pesticide dosage in real time;

[0055] The federated learning optimization and model update module is used to optimize and adjust the multimodal feature mapping model and YOLOv7-Spectral pest and disease identification model based on pesticide application feedback data, and integrate cross-farm data through the federated learning mechanism to improve the model's generalization ability.

[0056] The following are its main beneficial effects:

[0057] (1) Improve the accuracy of pest and disease identification. Using multimodal data fusion technology, we perform Transformer feature mapping on drone multispectral images, infrared thermal imaging, NDVI index, and field environmental data to construct a spectral-environmental fusion feature matrix. This effectively reduces the impact of single spectral data on factors such as illumination changes and background interference, thereby improving the accuracy of pest and disease identification in complex farmland environments. Compared with traditional RGB image recognition methods, this method achieves more stable and higher-precision pest and disease target detection on edge computing devices.

[0058] (2) Reduce pesticide use and improve precision spraying. By calculating the Blight Risk Index (BRI), combining historical disease trend data and field environmental data, the severity of the disease is quantitatively assessed, and a pesticide application priority map is generated to ensure precise drone application. The ant colony optimization algorithm (ACO) is used to plan the drone's spraying path, and the spraying angle and flow rate are adjusted based on meteorological parameters such as wind speed and humidity to reduce unnecessary pesticide use. Compared with traditional uniform spraying methods, pesticide consumption is reduced by 30%, effectively reducing agricultural pollution and improving the precision of pesticide application.

[0059] (3) Improve the intelligence level of disease prevention and control, realize cross-farm data sharing, use Federated Learning (FL) to optimize pest and disease identification and control models, and use data from different farms to update cross-regional models to improve the generalization ability of disease identification. This invention can be deployed in a distributed manner across multiple farms, eliminating the need to centrally upload data to the cloud, preventing data privacy leaks, and can adapt to different climates and soil environments, improving the intelligence level of large-scale agricultural prevention and control.

[0060] (4) Edge computing enables millisecond-level real-time identification. Traditional cloud-based pest and disease identification methods suffer from data upload delays and cannot meet the needs of large-scale real-time monitoring of farmland. This paper adopts a lightweight YOLOv7-Spectral model and optimizes neural network parameters to make it suitable for edge computing devices such as Jetson AGX Xavier. This achieves millisecond-level pest and disease detection, improves the response speed of real-time farmland prevention and control, and avoids the spread of diseases due to delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A flowchart of a method for real-time identification and prevention of crop pests and diseases based on multimodal edge computing provided in an embodiment of the present application;

[0062] Figure 2 This is a structural block diagram of the real-time identification and prevention and control decision-making system for crop diseases and pests based on multimodal edge computing provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0064] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0065] Example 1: Reference Figure 1 , is a flow chart of a method for real-time identification and prevention of crop diseases and insect pests based on multimodal edge computing provided by an embodiment of the present invention. The process may include at least steps S100-S700:

[0066] S100, obtaining multispectral image data, infrared thermal imaging data, and NDVI index collected by drones, and obtaining temperature, humidity, wind speed, and soil moisture data from field IoT sensors, synchronously preprocessing the data, and generating a multimodal feature input sequence;

[0067] S200, normalizing the multimodal feature input sequence, and performing multidimensional feature mapping based on the Transformer attention mechanism to generate a spectrum-environment fusion feature matrix;

[0068] S300: Based on the spectrum-environment fusion feature matrix and the pre-trained YOLOv7-Spectral model, perform pest and disease target detection, generate a pest and disease distribution heat map, perform confidence screening on the recognition results, and extract pest and disease characteristic areas;

[0069] S400, extracting spatial location features from the pest distribution heat map and data obtained by field IoT sensors, and combining historical disease trend data to assess disease severity, thereby generating a disease risk index (BRI) and a pesticide application priority map;

[0070] S500: Based on the pesticide application priority map and the drone operation parameters, an ant colony optimization path planning model is constructed, and combined with wind speed and humidity parameters, an optimized path sequence for drone pesticide application is generated;

[0071] S600: Obtain the optimized path sequence for drone spraying, control the drone to perform precise spraying according to the optimized path, and simultaneously monitor meteorological data and real-time images of the spraying area to update the disease change trend sequence;

[0072] S700. Based on the disease change trend sequence and the feedback data after pesticide application, adjust the parameter weights of the YOLOv7-Spectral model, and optimize the disease identification and pesticide application path planning models through federated learning to generate an updated intelligent prevention and control model.

[0073] Step S100 at least includes steps S110-S130:

[0074] S110, UAV multispectral and infrared thermal imaging data acquisition;

[0075] Specifically, the drone-mounted hyperspectral imaging device is called to obtain crop canopy spectral reflectance data, which covers the 400-1000nm band, and multispectral images are collected according to the set spatial resolution to generate a multispectral image dataset. At the same time, the infrared thermal imaging equipment carried by the drone is used to measure the surface temperature of crop leaves and generate infrared thermal imaging data sets. .

[0076] The multispectral image dataset Spectral reflectance The calculation is as follows: ;in:

[0077] For crop canopy at wavelength The radiance at

[0078] Blackboard reference brightness (used to eliminate device noise);

[0079] The reference brightness of the whiteboard (used for lighting normalization).

[0080] The infrared thermal imaging dataset The surface temperature of crop leaves is measured by the drone thermal imager to calculate the crop leaf temperature. , which is calculated as follows: ;

[0081] in: is infrared thermal radiation energy;

[0082] is the Stefan-Boltzmann constant ( ).

[0083] Furthermore, the and Generate spatially aligned crop multispectral-thermal imaging datasets through geo-referenced , as the input for subsequent data fusion.

[0084] S120, field IoT sensor data acquisition;

[0085] Specifically, environmental data including air temperature is collected from IoT sensors deployed in the field. , air humidity , soil moisture and wind speed , generate environmental sensing datasets .

[0086] The environmental data is transmitted in real time via LoRa or 5G network and timestamped. Synchronous processing is performed to ensure that all data are calculated within the same time window.

[0087] The environmental data is normalized as follows: ;

[0088] in: is the normalized environmental sensing data;

[0089] and are the minimum and maximum values ​​of the environmental data respectively.

[0090] Furthermore, the environmental data With the Perform time synchronization and register according to crop spatial coordinates to generate a fused input dataset , used for subsequent multimodal feature extraction.

[0091] S130, data preprocessing and feature extraction;

[0092] Specifically, the fusion input data set Perform interpolation, noise removal, and normalization to generate a preliminarily processed multimodal feature input sequence .

[0093] For missing data caused by collection errors, bilinear interpolation is used to fill in the gaps so that all data remain consistent in time and space dimensions.

[0094] The interpolation completion is calculated as follows: ;

[0095] in: is the interpolation value of the current pixel;

[0096] 、 、 、 are the values ​​of adjacent pixels respectively.

[0097] The interpolated and completed dataset To remove noise, wavelet transform is used to remove high-frequency noise and reduce data acquisition errors. The wavelet transform denoising process is as follows: ;

[0098] in: is the denoised data;

[0099] is the wavelet coefficient;

[0100] is the wavelet basis function.

[0101] For the denoised dataset Normalization is performed to ensure that different modal data are within the same numerical range and to generate standardized feature data sets. , calculated as follows: ;

[0102] in: is the mean of the data set;

[0103] is the standard deviation of the data set.

[0104] Furthermore, the As input data, it is transmitted to the S200 multimodal data fusion module and used for feature correlation modeling based on the S220 Transformer's multidimensional feature mapping.

[0105] Step S200 at least includes steps S210-S230:

[0106] S210, multimodal feature normalization;

[0107] Specifically, call the multimodal feature input sequence generated in S130 The data is normalized to eliminate differences in numerical scales between different data modalities. The normalization includes global normalization and crop category adaptive weighted normalization.

[0108] The spectral reflectance calculated in S130 , leaf surface temperature , NDVI index and environmental factors Standardization is performed and the normalization calculation is as follows: ;

[0109] in: is the normalized feature;

[0110] is the original eigenvalue;

[0111] is the mean value of the feature;

[0112] is the standard deviation of the feature.

[0113] The normalization process ensures that different data modalities are in the same numerical range, thereby improving the stability of subsequent model learning.

[0114] Furthermore, the normalized data is weighted according to the crop type and growth stage to generate the normalized features of the crop growth stage: ;

[0115] in: is the eigenvalue after adaptive adjustment;

[0116] is the normalized weight corresponding to the crop type;

[0117] is the bias parameter corresponding to the crop growth stage.

[0118] The adaptive weighted normalization is dynamically adjusted according to the spectral feature differences of different crop varieties to improve the accuracy of feature fusion.

[0119] Based on the above normalization steps, generate a standardized feature data sequence The data is used as input and transmitted to S220 for transformer-based multidimensional feature mapping.

[0120] S220, Transformer-based multidimensional feature mapping;

[0121] Specifically, the standardized feature data sequence generated from S210 Spectral reflectance, leaf surface temperature, NDVI index and environmental factors are extracted, a multi-dimensional feature mapping relationship is constructed, and a spectral-environment fusion feature matrix is ​​generated based on the Transformer attention mechanism.

[0122] Standardized feature data sequence Perform feature decomposition to form feature subsets: ;

[0123] in: is the spectral reflectance matrix;

[0124] is the leaf surface temperature matrix;

[0125] is the normalized NDVI index matrix;

[0126] is the environmental factor matrix.

[0127] Based on the feature subset, the Transformer self-attention mechanism is used to calculate the weight distribution of multimodal features. The calculation formula is as follows: ;

[0128] in: It is the fusion feature after attention weighting;

[0129] are query, key, and value matrices respectively, and are calculated as follows: ;

[0130] in is the weight matrix of query, key and value;

[0131] is the feature dimension normalization factor.

[0132] The Transformer calculation process can automatically adjust the influence weights of different data modalities, so that pest-related features receive higher attention and enhance the recognition ability of the model.

[0133] The fusion feature matrix after the weighted attention mechanism Perform transformation to generate spectrum-environment fusion feature matrix : ;

[0134] in: is the linear transformation weight matrix;

[0135] is the bias parameter.

[0136] Furthermore, the S220 calculated As input, it is transmitted to S230 for dimensionality reduction and feature extraction.

[0137] S230, multimodal data fusion output;

[0138] Specifically, call the spectrum-environment fusion feature matrix generated by S220 , dimensionality reduction is performed on the data, and core features are extracted through a feature clustering algorithm to generate fused feature data for pest and disease identification.

[0139] Spectral-environment fusion feature matrix calculated for S220 Perform principal component analysis (PCA) to reduce feature dimensions and improve computational efficiency. The calculation formula is as follows: ;

[0140] in: is the feature matrix after dimensionality reduction;

[0141] is the principal component transformation matrix of the feature matrix, which is calculated by PCA.

[0142] The feature dimension reduction reduces redundant information and improves the computational efficiency of subsequent pest and disease detection.

[0143] Furthermore, the feature matrix after dimension reduction is Perform feature clustering analysis and calculate the similarity of different feature subsets. The calculation is as follows: ;

[0144] in: For the Hedi The similarity of samples;

[0145] is the similarity measurement parameter.

[0146] The feature cluster analysis can effectively identify key features related to pests and diseases, eliminate irrelevant information, and improve detection accuracy.

[0147] Finally, the data matrix after feature cluster analysis As output, it is transmitted to the S300 pest and disease target detection module for the pest and disease detection task of the YOLOv7-Spectral model.

[0148] Step S300 at least includes steps S310-S330:

[0149] S310, input feature matrix to pre-trained YOLOv7-Spectral model;

[0150] Specifically, call the fusion feature data generated in S230 , the data is input into the pre-trained YOLOv7-Spectral model to perform target detection to identify pest targets on crops and generate pest target detection results .

[0151] The fusion feature data calculated in S230 Perform format conversion to ensure that it meets the input requirements of the YOLOv7-Spectral target detection model. The calculation is as follows: ;

[0152] in: This is the input data before YOLOv7-Spectral processing;

[0153] The input image size of the object detection network.

[0154] The data after format conversion Input the pre-trained YOLOv7-Spectral model, perform forward reasoning, and generate the target bounding box and its confidence score , calculated as follows: ;

[0155] in:

[0156] is the bounding box coordinates of the pest target;

[0157] is the confidence score of the corresponding target;

[0158] Represents the detected target index.

[0159] Furthermore, based on the inference results of the YOLOv7-Spectral model, the pest and disease target detection result data is generated , which includes the location, category and confidence score of all detected targets, and transmits the data to S320 to generate a pest distribution heat map.

[0160] S320, generation of pest and disease distribution heat map;

[0161] Specifically, call the pest target detection result calculated in S310 , perform distribution mapping according to the target spatial location and generate a heat map of pest and disease distribution .

[0162] Detection results of the pests and diseases targets The target bounding box in Normalization is performed to adapt to the pixel coordinate system generated by the heat map, and the calculation is as follows: ;

[0163] in: is the normalized target bounding box;

[0164] is the width and height of the input image.

[0165] Based on the normalized target coordinates , calculate the spatial density distribution of pest and disease targets , calculated as follows: ;

[0166] in: Represented in pixel coordinates Pest and disease density at the site;

[0167] is the confidence score of the pest and disease target;

[0168] is the density distribution smoothing parameter.

[0169] Furthermore, the pest density distribution data Perform visualization to generate a heat map of pest and disease distribution and transmits the data to S330 for confidence screening and feature extraction.

[0170] S330, confidence screening and feature extraction;

[0171] Specifically, call the pest distribution heat map calculated by S320 , perform confidence screening on the data, and extract characteristic areas of pests and diseases to obtain characteristic area data of pests and diseases .

[0172] Heat map of the distribution of pests and diseases Perform confidence screening to eliminate low-confidence targets. The calculation is as follows: ;

[0173] in: This is the heat map of pest and disease distribution after screening;

[0174] is the confidence threshold.

[0175] The heat map of the distribution of the selected pests and diseases calculated in S330 , conduct connected region analysis and extract pest and disease characteristic regions, and calculate as follows: ;

[0176] in: Regional data on pest and disease characteristics;

[0177] Extract functions for connected components.

[0178] Finally, the pest and disease characteristic area data As output, it is transmitted to the S400 pest and disease risk assessment module for subsequent disease severity assessment and pesticide application optimization calculation.

[0179] Step S400 at least includes steps S410-S430:

[0180] S410, extracting spatial position features;

[0181] Specifically, call the pest and disease characteristic area data calculated in S330 , and combined with the environmental sensing data sequence generated by S120 , extract the spatial location characteristics of the target crop area and obtain the crop spatial distribution sequence .

[0182] Regional data on pest characteristics Perform spatial coordinate transformation to ensure that the target location matches the farmland grid area. The calculation is as follows: ;

[0183] in:

[0184] is the target space coordinate after transformation;

[0185] is the original target coordinate;

[0186] is the geospatial coordinate transformation matrix.

[0187] Field IoT sensor data calculated based on S120 , perform crop area index matching to obtain the geographic grid number corresponding to the target crop , calculated as follows: ;

[0188] in: The geographical grid number corresponding to the target crop;

[0189] is the grid index function.

[0190] Generate crop spatial distribution sequence based on the target position mapping and crop area index matching and transmit the data to S420 for disease severity assessment.

[0191] S420, disease severity assessment;

[0192] Specifically, the crop spatial distribution sequence calculated in S410 is called , combined with historical disease trend data , calculate the disease severity index , and further generate a disease risk index (BRI).

[0193] Based on the crop spatial distribution sequence and historical disease trend data , calculate the disease severity index , calculated as follows: ;

[0194] in: For Grid The corresponding disease severity;

[0195] For Grid Historical disease index;

[0196] is the confidence score of the pest and disease target;

[0197] Target To Grid The Euclidean distance of

[0198] is the spatial impact scale parameter; is the weight parameter.

[0199] Furthermore, based on the disease severity index Calculate the disease risk index (BRI) as follows: ;

[0200] in: For Grid Disease risk index;

[0201] is the environmental factor data;

[0202] is the weight of disease and environmental factors.

[0203] Finally, disease risk index (BRI) data is generated and transmitted to S430 for generating a pesticide application priority map.

[0204] S430, generating a pesticide application priority map;

[0205] Specifically, the disease risk index (BRI) data calculated in S420 is called, the data is divided into regions, and a pesticide application priority map is generated based on the disease distribution trend. .

[0206] Based on the disease risk index (BRI) calculated by S420, the farmland areas are graded to generate areas of different risk levels. The calculation is as follows: ;

[0207] in: For Grid Corresponding disease risk level;

[0208] is the risk index threshold.

[0209] Calculate the priority of pesticide application based on the disease risk area division results , calculated as follows: ;

[0210] in: For Grid Priority of pesticide application;

[0211] is the weighting parameter.

[0212] Finally, a pesticide application priority map is generated based on the pesticide application priority calculation results. The data is then transmitted to the S500 UAV spraying path optimization module for optimizing the UAV spraying operation.

[0213] Step S500 at least includes steps S510-S530:

[0214] S510, constructing an ant colony optimization path planning model; specifically, calling the pesticide application priority map calculated in S430 , and combined with the UAV operation parameters , build an ant colony optimization path planning model to generate the initial drone spraying path plan .

[0215] Based on the pesticide application priority map , building a farmland grid path network ,in: Represents a grid collection, each grid represents an area of ​​farmland; Represents the connected edges between adjacent grids; each edge The path cost is calculated as follows: ;

[0216] in: From the grid To Grid The path cost;

[0217] is the Euclidean distance between two grids;

[0218] For Grid Disease risk index;

[0219] is the path cost weight parameter.

[0220] Initialize the Ant Colony Optimization (ACO) model and set the number of ants and the maximum number of iterations , define the pheromone matrix and heuristic information : ;

[0221] in: For path Pheromone concentration on;

[0222] For heuristic information;

[0223] A very small positive number to prevent division by zero.

[0224] Calculate the initial spraying path plan of the drone based on the ant colony optimization model , the path selection probability is calculated as follows: ;

[0225] in: For Ant slave nodes Select to Node probability;

[0226] are the weights of pheromone and heuristic information.

[0227] After the calculation is completed, the initial UAV spraying path plan Transmit to S520 for path optimization.

[0228] S520, performing path optimization based on wind speed and humidity parameters;

[0229] Specifically, the initial drone spraying path plan calculated in S510 is called , and combined with the wind speed and humidity parameters generated by S120 , optimize the spraying path and generate the optimized drone spraying path sequence .

[0230] Based on wind speed parameters , adjust the drone in the grid Spray angle at , calculated as follows: ;

[0231] in: For drones in the grid The spray angle at

[0232] The wind speed is and Directional weight.

[0233] Based on humidity parameters , adjust the spraying flow , calculated as follows: ;

[0234] in: For Grid The flow of drone spraying at the site;

[0235] It is the benchmark spraying flow rate.

[0236] Based on the corrected spray angle and spraying flow , update the drone spraying path and generate the optimized drone spraying path sequence , and transmitted to S530 for final path optimization.

[0237] S530, outputting the final pesticide application path;

[0238] Specifically, the optimized drone spraying path sequence calculated in S520 is called , smooth the path and adjust the spraying height to generate the final precise spraying path .

[0239] Optimized UAV spraying path sequence Bezier curve fitting is performed to ensure the smoothness of the flight trajectory. The calculation is as follows: ;

[0240] in: is the smoothed path point;

[0241] are three adjacent points on the path;

[0242] is the interpolation parameter, and its value range is .

[0243] Based on crop height , adjust the drone in the grid Flight altitude , calculated as follows: ;

[0244] in: is the spraying height of the drone;

[0245] is the reference spraying height;

[0246] is the crop height adjustment factor.

[0247] Finally, the final precise spraying path is generated The data is then transmitted to the S600 UAV spraying execution and data monitoring module for precise spraying tasks.

[0248] Step S600 at least includes steps S610-S630:

[0249] S610, implement precision pesticide application;

[0250] Specifically, the final precise spraying path calculated in S530 is called , and based on the UAV operation parameters , control the drone to perform precise spraying according to the path, and record the spraying location data at the same time and pesticide usage data .

[0251] Based on the final precise spraying path , control the drone to fly to each target grid in turn , perform precise spraying in the corresponding grid and calculate the current position of the drone ,as follows: ;

[0252] in: For time The actual position of the drone at any moment;

[0253] is the target position in the path sequence;

[0254] is the path interpolation coefficient.

[0255] Spraying height calculated based on S530 and wind speed adjustment parameters , dynamically adjust pesticide flow , calculated as follows: ;

[0256] in: For Grid Pesticide spraying flow rate at the site;

[0257] is the benchmark spraying flow rate;

[0258] The maximum spraying height of the drone.

[0259] Furthermore, the GPS location information of the drone, real-time pesticide consumption and duration of the application process are recorded to generate application location data. and pesticide use data The data is transmitted to S620 for monitoring the spraying area data.

[0260] S620, data monitoring of the application area;

[0261] Specifically, call the spraying location data generated in S610 and pesticide use data , combined with the meteorological data of the spraying area collected by S120 and real-time image data obtained by drone cameras , monitor the spraying area and calculate the disease change trend sequence .

[0262] Based on application location data and meteorological data , matching the wind speed, humidity and temperature parameters at the time of application, the calculation is as follows: ;

[0263] in: Grid for spraying time Wind speed at

[0264] The air humidity at the time of application;

[0265] The air temperature at the time of application.

[0266] Disease change trend sequence calculated based on S620 and real-time drone image data , calculate the changes in crop health status in the target area after pesticide application, as follows: ;

[0267] in: For Grid Crop health recovery rate at the site;

[0268] This is a heat map of disease distribution before pesticide application.

[0269] Finally, the pesticide application effect feedback sequence is generated and transmit the data to S720 for model optimization and weight adjustment.

[0270] S720, model optimization and weight adjustment;

[0271] Specifically, the pesticide application effect feedback sequence calculated in S710 is called , and combined with the multimodal feature mapping model generated by S220 and S310 pre-trained YOLOv7-Spectral model , optimize and adjust the model parameters to generate an optimized pest and disease identification model .

[0272] Based on the feedback sequence of pesticide application effect , calculate the loss function of the model , calculated as follows: ;

[0273] in: is the error loss of the model;

[0274] For actual disease labels;

[0275] The disease label predicted by the model.

[0276] Based on the loss function , the Adam optimization algorithm is used to update the model parameters, and the calculation is as follows: ;

[0277] in: For the model at time Parameters at

[0278] is the learning rate.

[0279] Finally, an optimized pest and disease identification model is generated and transmit the model to S730 for federated learning training.

[0280] S730, federated learning optimization prevention and control model;

[0281] Specifically, the optimized pest and disease identification model calculated in S720 is called , and combined with cross-farm data , conduct federated learning training, achieve model generalization optimization, and generate an updated intelligent prevention and control model .

[0282] Based on cross-farm data , calculate the parameter update of each farm sub-model , calculated as follows: ;

[0283] in: For the farm Model update parameters at ;

[0284] For the farm The model loss at .

[0285] Sub-model updates based on all farms , the FedAvg algorithm is used to calculate the global model parameters , calculated as follows: ;

[0286] in: For the farm The data weight at .

[0287] Finally, an updated intelligent prevention and control model is generated and transmit the model to S310 to perform the pest and disease target detection task.

[0288] Example 2: Figure 2 The following is a structural block diagram of a real-time identification and prevention decision-making system for crop pests and diseases based on multimodal edge computing according to an embodiment of the present invention. Figure 2 As shown, the structure may include:

[0289] Data acquisition module 10 is used to acquire crop growth and pest monitoring data from multimodal data sources, including multispectral imaging data from drones, infrared thermal imaging data, and the NDVI index, as well as environmental data (temperature, humidity, wind speed, soil moisture, etc.) collected by field IoT sensors. This module ensures the diversity and high temporal and spatial resolution of the collected data, providing accurate raw input for subsequent pest and disease identification.

[0290] The data processing and fusion module 20 preprocesses the collected multimodal data, including data denoising, missing value filling, and normalization. It also uses the Transformer attention mechanism to perform feature mapping on the multimodal data and generate a spectral-environment fusion feature matrix. This module ensures data alignment across different data sources, improving the model's generalization capabilities and data fusion accuracy.

[0291] The pest and disease identification module 30 performs real-time target detection of crop pests and diseases based on the YOLOv7-Spectral deep learning model. This module uses a spectral-environmental fusion feature matrix as input and uses the target detection model to extract the bounding boxes and confidence levels of pest and disease targets. It then generates a pest and disease distribution heat map and removes low-confidence targets to improve recognition accuracy and robustness.

[0292] The disease risk assessment module 40 calculates the disease severity index based on pest distribution heat maps, field environmental sensor data, and historical disease trend data, and further generates a disease risk index (BRI). This module uses the disease risk index to rank disease-affected areas, providing data support for precise pesticide application.

[0293] The drone application path optimization module 50 uses an ant colony optimization algorithm to plan the optimal application path based on the disease risk index and drone operating parameters. This module optimizes the drone's application angle and flight trajectory by combining wind speed and humidity parameters, reducing pesticide usage, improving drone operating efficiency, and generating a precise application path.

[0294] The drone application execution and data monitoring module 60 controls the drone to precisely apply pesticides along an optimized path and records application locations and pesticide dosages in real time. This module also acquires meteorological data and real-time drone imagery of the application area. Based on disease trend sequences, it dynamically monitors application effectiveness, providing feedback for subsequent model optimization.

[0295] The Federated Learning Optimization and Model Update Module 70 optimizes the multimodal feature mapping model and the YOLOv7-Spectral pest and disease identification model based on pesticide application feedback data. It also integrates cross-farm data through a federated learning mechanism to improve the model's generalization capabilities. This module uses the Federated Averaging Algorithm (FedAvg) to update global model parameters, ensuring the pest and disease identification and pesticide application path optimization models remain efficient and adaptable over the long term.

[0296] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A real-time identification and prevention decision-making method for crop diseases and insect pests based on multimodal edge computing, characterized in that: The method comprises the following steps: Use the drone-mounted hyperspectral imaging device to obtain crop canopy spectral reflectance data, which covers the 400-1000nm band. Collect multispectral images at the set spatial resolution to generate a multispectral image dataset. ;Use the infrared thermal imaging equipment carried by the drone to measure the surface temperature of crop leaves and generate infrared thermal imaging data sets ; Obtain temperature, humidity, wind speed, and soil moisture data from field IoT sensors, perform synchronous preprocessing on the data, and generate a multimodal feature input sequence; The multimodal feature input sequence is normalized, and multidimensional feature mapping is performed based on the Transformer attention mechanism to generate a spectrum-environment fusion feature matrix, specifically including: Standardized feature data sequence Perform feature decomposition to form feature subsets: ; in, is the spectral reflectance matrix, is the leaf surface temperature matrix, is the normalized NDVI index matrix, is the environmental factor matrix; Based on the feature subset, the Transformer self-attention mechanism is used to calculate the weight distribution of multimodal features: ; ; in, It is the fusion feature after attention weighting; are query, key, and value matrices respectively; is the weight matrix of query, key and value, is the feature dimension normalization factor; The fusion feature matrix after the attention mechanism weighting Perform transformation to generate spectrum-environment fusion feature matrix : ; in, is the linear transformation weight matrix, is the bias parameter; Based on the spectral-environmental fusion feature matrix and the pre-trained YOLOv7-Spectral model, pest and disease target detection is performed on the Jetson AGXXavier edge computing device to generate a pest and disease distribution heat map. The recognition results are then subjected to confidence screening and feature extraction to extract pest and disease characteristic areas. Spatial location features are extracted from the pest distribution heat map and the data obtained by the field IoT sensors, and combined with historical disease trend data, the disease severity index is calculated to generate the disease risk index (BRI) and the pesticide application priority map, wherein the calculation formula of the disease risk index (BRI) is: ; in, For Grid Disease risk index; For Grid The disease severity index at the location is calculated as: ;in, For Grid Historical disease index; is the confidence score of the pest and disease target; Target To Grid The Euclidean distance of is the spatial impact scale parameter; is the weight parameter; For Grid Environmental factor data; is the weight of disease and environmental factors; Based on the pesticide application priority map and the drone operation parameters, an ant colony optimization path planning model is constructed, and combined with wind speed and humidity parameters, an optimized path sequence for drone pesticide application is generated; Obtaining the optimized path sequence for drone spraying, and controlling the drone to perform precise spraying according to the optimized path, while monitoring meteorological data and real-time images of the spraying area, and updating the disease change trend sequence; Based on the disease trend sequence and post-application feedback data, the parameter weights of the YOLOv7-Spectral model were adjusted, and the disease identification and application path planning models were optimized through federated learning to generate an updated intelligent prevention and control model. Specifically, the model includes: Based on the feedback sequence of pesticide application effect , calculate the model loss function : ;in, is the actual disease label, The disease label predicted by the model; Use the Adam optimization algorithm to update the model parameters: in For the model at time Parameters, is the learning rate; Based on cross-farm data , calculate the parameter updates for each farm sub-model: in For the farm Model loss at ; The federated averaging algorithm FedAvg is used to calculate the global model parameters: in For the farm The data weight at .

2. The method according to claim 1, characterized in that The preprocessing of the multimodal feature input sequence includes the following steps: interpolation and noise removal are performed on the multispectral image data, infrared thermal imaging data, and NDVI index to generate standardized multispectral image sequences and infrared thermal imaging sequences; Perform time alignment and spatial registration on field IoT sensor data, and perform data normalization to generate standardized environmental data sequences; Data fusion is performed on the standardized multispectral image sequence, infrared thermal imaging sequence and environmental data sequence to generate a preliminarily processed multimodal feature input sequence.

3. The method according to claim 1, characterized in that The construction of the multidimensional feature map comprises the following steps: Performing feature decomposition on the multimodal feature input sequence to extract spectral reflectance, leaf surface temperature, NDVI index and environmental factors; The Transformer attention mechanism is used to perform feature mapping on the spectral reflectance, leaf surface temperature, NDVI index and environmental factors to generate a spectral-environment fusion feature matrix.

4. The method according to claim 1, wherein The target detection of pests and diseases includes the following steps: Inputting the spectrum-environment fusion feature matrix into the pre-trained YOLOv7-Spectral model to perform pest and disease target detection and generate pest and disease target detection results; Based on the pest and disease target detection results, distribution mapping of the detection targets is performed according to spatial positions to generate a pest and disease distribution heat map; The pest and disease distribution heat map is screened for confidence, low-confidence targets are eliminated, and pest and disease characteristic area data are extracted.

5. The method according to claim 1, wherein The process of generating the disease risk index BRI comprises the following steps: Generate crop spatial distribution sequence based on target location mapping and crop area index matching; Based on the crop spatial distribution sequence and historical disease trend data, a disease severity index is calculated and a disease risk index BRI is generated.

6. The method according to claim 1, characterized in that The optimized path sequence for drone spraying includes the following steps: Based on the pesticide application priority map and the UAV operation parameters, an ant colony optimization path planning model is constructed to generate an initial UAV pesticide application path plan; The initial drone spraying path plan is optimized by combining wind speed and humidity parameters, and the spraying angle is adjusted to generate an optimized drone spraying path sequence; The optimized UAV spraying path sequence is smoothed, and the spraying height is adjusted to generate a final precise spraying path.

7. The method according to claim 1, characterized in that Controlling the drone to perform precise spraying according to the optimized path includes the following steps: Based on the UAV operation parameters, the drone is controlled to perform precise spraying according to the optimized spraying path, while recording the spraying location and pesticide dosage data; During the application process, meteorological data and real-time images of the application area are obtained, and the data are compared with historical data to obtain a sequence of disease change trends; Based on the disease change trend sequence, disease changes are predicted and disease severity assessment data is updated.

8. The method according to claim 1, characterized in that Adjusting the parameter weights of the YOLOv7-Spectral model includes the following steps: Obtain post-application feedback data, including drone operation data, crop health status, and farmland environment data, to form a feedback sequence on the application effect; Based on the pesticide application effect feedback sequence, the parameter weights of the multimodal feature mapping model and the YOLOv7-Spectral model are adjusted to generate an optimized pest and disease recognition model.

9. The method according to claim 1, characterized in that The federated learning optimization prevention and control model includes the following steps: Based on pest and disease data from multiple farms, the local update parameters of each sub-model are calculated and the model is trained independently on each terminal device; Based on the local update parameters, the global model parameters are calculated using a federated averaging algorithm, model aggregation is performed, and an updated intelligent prevention and control model is generated.

10. A real-time identification and prevention and control decision-making system for crop diseases and insect pests based on multimodal edge computing, applied to the real-time identification and prevention and control decision-making method for crop diseases and insect pests based on multimodal edge computing according to any one of claims 1 to 9, characterized in that: include: Data acquisition module, used to obtain monitoring data related to crop growth and pests and diseases from multimodal data sources; The data processing and fusion module is used to pre-process the collected multimodal data and perform feature mapping on the multimodal data using the Transformer attention mechanism to generate a spectral-environment fusion feature matrix; Pest and disease recognition module, which is used for real-time target detection of crop pests and diseases based on the YOLOv7-Spectral deep learning model; The disease risk assessment module is used to calculate the disease severity index based on the pest and disease distribution heat map, field environmental sensor data and historical disease trend data, and further generate the disease risk index (BRI); The drone spraying path optimization module is used to plan the optimal spraying path based on the disease risk index and drone operation parameters using the ant colony optimization algorithm; The drone spraying execution and data monitoring module is used to control the drone to carry out precise spraying according to the optimized path, and record the spraying location and pesticide dosage in real time; The federated learning optimization and model update module is used to optimize and adjust the multimodal feature mapping model and YOLOv7-Spectral pest and disease identification model based on pesticide application feedback data, and integrate cross-farm data through the federated learning mechanism to improve the model's generalization ability.

Images