An agricultural pest and disease prediction system and method based on multi-modal data fusion and edge AI

Abstract

This invention relates to the field of smart agriculture information technology, and discloses an agricultural pest and disease prediction system and method based on multimodal data fusion and edge AI, comprising: a data acquisition module, an edge AI analysis module, and a communication interaction module; the data acquisition module is used to collect agricultural monitoring data containing agricultural environmental information and crop status information; the edge AI analysis module is used to call a pest and disease prediction model that integrates multimodal agricultural feature extraction to perform multimodal joint feature extraction on the agricultural monitoring data containing agricultural environmental information and crop status information, and infer pest and disease prediction results based on the multimodal joint features, and generate pest and disease early warning information based on the pest and disease prediction results; the communication interaction module is used to upload the agricultural monitoring data collected by the data acquisition module and the pest and disease prediction results and pest and disease early warning information output by the edge AI analysis module to a cloud platform. This invention enables agricultural pest and disease prediction on resource-constrained edge devices.

Description

Technical Field

[0001] This invention relates to the field of smart agriculture information technology, specifically to an agricultural pest and disease prediction system and method based on multimodal data fusion and edge AI. Background Technology

[0002] Existing agricultural pest and disease monitoring and early warning systems typically deploy temperature sensors, humidity sensors, soil chemical sensors, light sensors, and image acquisition equipment in agricultural settings such as farmland and greenhouses to collect agricultural environmental and crop status information. The collected data is then transmitted to a host computer or cloud platform for storage, display, and analysis to assist in agricultural production management. However, most existing technologies focus on single-type data analysis or simply aggregate multiple data sets, lacking effective techniques for correlation analysis and fusion processing of different types of agricultural data. The occurrence of agricultural pests and diseases is usually influenced by both environmental condition evolution and crop phenotypic changes. Agricultural environmental information reflects the environmental triggers for pest and disease occurrence, while crop status information reflects phenotypic characteristics such as leaf spots, chlorosis, and pest edges. Analyzing only single-modal data makes it difficult to consider both environmental evolution information and crop phenotypic information; using only independent analysis or simple overlay methods also fails to fully explore the correlations between different data, thus affecting the accuracy and reliability of agricultural pest and disease prediction results.

[0003] Furthermore, many existing intelligent agricultural monitoring solutions rely heavily on uploading data to the cloud for analysis and processing, or only perform data collection and preprocessing at the edge, while dedicating the main intelligent analysis tasks to the cloud platform. Systems based on cloud-based AI processing are highly dependent on network connectivity and cloud computing resources. In situations where network conditions are unstable or communication bandwidth is limited in agricultural settings, this can easily lead to increased data transmission burden and decreased analysis timeliness, failing to meet the real-time, continuous, low-bandwidth, and low-power consumption requirements of agricultural monitoring scenarios. Additionally, since all cloud-based AI computation is performed on servers or public clouds, with devices only transmitting data and inference and computation completed entirely in the cloud, costs are significantly increased. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides an agricultural pest and disease prediction system and method based on multimodal data fusion and edge AI, which can perform multimodal data fusion analysis on agricultural environmental information and crop status information, and realize agricultural pest and disease prediction on the resource-constrained edge side.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0006] An agricultural pest and disease prediction system based on multimodal data fusion and edge AI includes: a data acquisition module, an edge AI analysis module, and a communication interaction module; The data acquisition module is used to collect agricultural monitoring data that includes agricultural environmental information and crop status information; The edge AI analysis module is used to call the pest and disease prediction model that integrates multimodal agricultural feature extraction to perform multimodal joint feature extraction on agricultural monitoring data containing agricultural environmental information and crop status information, and to infer pest and disease prediction results based on the multimodal joint features, and generate pest and disease early warning information based on the pest and disease prediction results. The communication interaction module is used to upload agricultural monitoring data collected by the data acquisition module and pest and disease prediction results and pest and disease early warning information output by the edge AI analysis module to the cloud platform.

[0007] Furthermore, the pest and disease prediction model that integrates multimodal agricultural feature extraction includes: an agricultural environmental information feature extraction branch, a crop state information feature extraction branch, an attention alignment fusion module, and a pest and disease prediction output module; The agricultural environmental information feature extraction branch is used to extract time-series features from agricultural environmental information to obtain a global time-series feature vector of the agricultural environment. The crop state information feature extraction branch is used to extract spatial features from crop state information to obtain crop state spatial features; The attention alignment and fusion module is used to align and fuse the agricultural environment temporal global feature vector with the crop state space features to generate a unified multimodal joint feature. The pest and disease prediction output module is used to output pest and disease prediction results based on unified multimodal joint features.

[0008] Furthermore, the agricultural environmental information feature extraction branch adopts a lightweight temporal feature extraction structure of multi-scale dilated temporal convolution, including: a first 1×1 convolutional layer, a multi-scale temporal convolutional unit, a residual dilated temporal feature enhancement block, and a temporal global aggregation unit; The first 1×1 convolutional layer is used to map agricultural environmental information into environmental temporal embedding features; The multi-scale temporal convolutional unit is used to extract multi-scale temporal features from each environment temporal embedding feature in the temporal sequence, concatenate the extracted multi-scale temporal features in the channel dimension, and perform channel fusion and dimensional compression on the concatenated features to obtain multi-scale temporal environment features in the temporal sequence. The residual expansion temporal feature enhancement block is used to perform nonlinear feature enhancement and channel fusion on each multi-scale temporal environment feature under the time series, so as to obtain enhanced temporal environment features; The temporal global aggregation unit is used to aggregate the enhanced temporal environmental features in a temporal order to obtain the agricultural environment temporal global feature vector.

[0009] Furthermore, the residual dilation temporal feature enhancement block includes: a dilated one-dimensional convolutional layer, a batch normalization layer, a ReLU nonlinear activation layer, a pointwise convolutional layer, and an identity residual block; The dilated one-dimensional convolutional layer is used to capture temporal context features containing contextual information from each multi-scale temporal context feature; The batch normalization layer is used to normalize the captured temporal environmental features containing contextual information; The ReLU nonlinear activation layer is used to perform nonlinear transformation on the normalized temporal environmental features; The pointwise convolutional layer is used to perform pointwise convolution on the temporal environmental features of the nonlinear transformation; The identity residual block is used to add the pointwise convolutional temporal environmental features to the multi-scale temporal environmental features element-wise to obtain enhanced temporal environmental features.

[0010] Furthermore, the agricultural environmental information feature extraction branch adopts a lightweight spatial feature extraction structure based on multi-scale inverse residual convolution, including: overlapping convolution embedding unit, several multi-scale inverse residual units, multi-scale feature alignment and fusion unit, and tensor renormalization unit; The overlapping convolutional embedding unit is used to perform shallow feature extraction and spatial downsampling of crop state information based on adjacent convolutional windows with overlapping regions to obtain the initial spatial features of the crop. The aforementioned multi-scale inverse residual units extract spatial features of different spatial scales from the initial spatial features of the crop; The multi-scale feature alignment and fusion unit is used to align spatial features of different scales in terms of spatial size and unify channel dimensions to obtain a crop fusion spatial feature map. The tensor remodeling unit is used to perform tensor remodeling operations on the crop fusion spatial feature map to convert it into a crop spatial tensor.

[0011] Furthermore, each multi-scale inverse residual unit includes: a channel-expanding convolutional layer, a multi-scale depth convolutional layer, a feature splicing layer, a channel-fusion compression convolutional layer, and a residual connection block; The channel-expanded convolutional layer is used to map the initial spatial features of the crop from low-dimensional channels to high-dimensional channel feature space; The multi-scale deep convolutional layer is used to extract spatial features of different scales from the high-dimensional channel feature space; The feature splicing layer is used to splice spatial features of different scales along the channel dimension; The channel fusion compression convolutional layer is used to perform channel fusion on the spliced ​​multi-scale spatial features, mapping them to a preset number of channels to obtain spatial features of the corresponding spatial scale.

[0012] Furthermore, the attention alignment fusion module transforms the agricultural environment temporal global feature vector into a query vector through a linear mapping. The crop state space features are transformed into key matrices through linear mapping. Sum matrix According to the query vector of the linear mapping Key matrix Sum matrix Generate unified multimodal joint features:

[0013] in, Represents the query vector AND key matrix The length of the vector corresponding to the dot product calculation. express function, This indicates the transpose operation.

[0014] Furthermore, the pest and disease prediction model integrating multimodal agricultural feature extraction is trained on a cloud platform. After training, the agricultural environmental information feature extraction branch and crop state information feature extraction branch in the pest and disease prediction model are pruned, and the parameters of the pruned pest and disease prediction model integrating multimodal agricultural feature extraction are sent to the edge AI analysis module through the communication interaction module.

[0015] Furthermore, the importance of each channel in the convolutional layer of the agricultural environmental information feature extraction branch and the crop state information feature extraction branch is evaluated separately, and channels whose importance is lower than the set channel importance threshold are pruned. The process for evaluating the importance of each channel in the convolutional layer is as follows:

[0016] in, This represents the first feature extraction branch for agricultural environmental information or the feature extraction branch for crop status information. The first convolutional layer The importance score of each channel This represents the first feature extraction branch for agricultural environmental information or the feature extraction branch for crop status information. The first convolutional layer Each channel corresponds to a scaling factor for the batch normalization layer. This represents the number of training samples used to evaluate the channel importance in the convolutional layer. express index, Indicates the first The training sample is processed by the agricultural environmental information feature extraction branch or the crop status information feature extraction branch. The first convolutional layer Channel response for each channel, This indicates the calculation of the mean.

[0017] Furthermore, the present invention also provides a method for predicting agricultural pests and diseases based on multimodal data fusion and edge AI, comprising the following steps: Step S1: Collect agricultural monitoring data containing agricultural environmental information and crop status information; Step S2: Construct an edge AI analysis module and deploy a pest and disease prediction model that integrates multimodal agricultural feature extraction in the AI ​​analysis module. This model is used to perform multimodal joint feature extraction on agricultural monitoring data containing agricultural environmental information and crop status information, and to infer pest and disease prediction results based on the multimodal joint features. Finally, pest and disease early warning information is generated based on the pest and disease prediction results. Step S3: Upload the collected agricultural monitoring data, pest and disease prediction results, and pest and disease early warning information to the cloud platform.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) The agricultural pest and disease prediction system and method based on multimodal data fusion and edge AI of the present invention deploys a pest and disease prediction model that integrates multimodal agricultural feature extraction on an edge embedded microprocessor. It extracts the global features of agricultural environment time series in agricultural environmental information and the spatial features in crop state information respectively. The global features of agricultural environment time series are mapped into query vectors through attention mechanism to guide the weighted fusion of crop spatial features. This enables the pest and disease prediction model to dynamically adjust the contribution of different leaf local spatial features in the prediction results according to the current environmental state, thereby improving the accuracy and reliability of pest and disease prediction results. At the same time, it realizes the local intelligent analysis of pest and disease prediction tasks, reduces the dependence on cloud computing resources and network transmission conditions, and improves the real-time performance and continuity of the pest and disease prediction system in weak network environment. (2) The agricultural environmental information feature extraction branch in this invention adopts a lightweight temporal feature extraction structure based on multi-scale dilated temporal convolution. It extracts agricultural environmental change features at different time scales through multi-scale temporal convolution units, including short-term fluctuations, continuous time period change trends and cumulative change information over a longer time range. Then, it enhances the agricultural environmental change features through residual dilated temporal feature enhancement blocks, so that the extracted agricultural environmental temporal global feature vector can simultaneously represent the overall environmental change trend and peak change information, thereby improving the ability of agricultural environmental temporal data to represent environmental factors related to the risk of pests and diseases. (3) The crop state information feature extraction branch in this invention adopts a lightweight spatial feature extraction structure based on multi-scale inverse residual convolution. First, shallow feature extraction and spatial downsampling are performed on the leaf image through overlapping convolution embedding units. While reducing the spatial resolution of the image, the details of leaf edge, texture and early lesion are preserved. Then, through multi-scale inverse residual units, the spatial features of small lesion edges, local chlorotic texture and large-scale insect phenotypic changes can be characterized simultaneously, thereby improving the feature expression ability of key phenotypic regions such as lesion, chlorosis and insect edge. (4) This invention reduces the model parameter size, storage occupation, running memory occupation and inference computation overhead by performing structured pruning on the pest and disease prediction model on the cloud platform and pre-allocating a fixed-size continuous tensor storage area in the edge embedded microprocessor, making the pest and disease prediction model suitable for deployment of resource-constrained, low-power agricultural field edge devices, and has good engineering feasibility. Attached Figure Description

[0019] Figure 1 This is a framework diagram of the agricultural pest and disease prediction system based on multimodal data fusion and edge AI of the present invention. Figure 2 This is a schematic diagram of the pest and disease prediction model in this invention; Figure 3 This is a schematic diagram of the agricultural environmental information feature extraction branch in this invention; Figure 4 This is a schematic diagram of the crop state information feature extraction branch in this invention; Figure 5 This is a schematic diagram of the attention alignment fusion module in this invention; Figure 6 This is a flowchart of the agricultural pest and disease prediction method based on multimodal data fusion and edge AI of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.

[0021] like Figure 1This is a framework diagram of the agricultural pest and disease prediction system based on multimodal data fusion and edge AI of the present invention. This system is deployed in greenhouse or small-scale farmland scenarios to collaboratively collect, fuse, analyze, and predict agricultural environmental and crop status information. Specifically, it includes: a data acquisition module, an edge AI analysis module, and a communication interaction module. The data acquisition module collects agricultural monitoring data containing agricultural environmental and crop status information. The edge AI analysis module calls a pest and disease prediction model that integrates multimodal agricultural feature extraction to perform multimodal joint feature extraction on the agricultural monitoring data containing agricultural environmental and crop status information, infers pest and disease prediction results based on the multimodal joint features, and generates pest and disease early warning information based on the prediction results. The communication interaction module uploads the agricultural monitoring data collected by the data acquisition module, the pest and disease prediction results output by the edge AI analysis module, and the pest and disease early warning information to a cloud platform.

[0022] In this invention, agricultural environmental information is selected from time-series parameters such as air temperature, air humidity, soil chemical composition, and light intensity, which are related to the occurrence of pests and diseases, to characterize the environmental factors inducing their occurrence. Crop status information is selected from leaf images acquired by image acquisition equipment to characterize phenotypic changes in crop leaves, such as lesions, chlorosis, and pest edges. The agricultural environmental information and crop status information together constitute heterogeneous multi-source input data for pest and disease prediction, which is sent to the edge AI analysis module for pest and disease inference analysis. On the one hand, the agricultural environmental time-series information reflects the environmental factors before the occurrence of pests and diseases; on the other hand, the leaf image information reflects the phenotypic characteristics of crops after damage, thereby achieving joint analysis of pest and disease occurrence trends and phenotypic changes, improving the accuracy and stability of pest and disease prediction.

[0023] When this agricultural pest and disease prediction system is running, it samples agricultural environmental information at a fixed cycle, preferably 1 minute; the image acquisition device acquires leaf images at a fixed shooting cycle, preferably 10 minutes. Each time a pest and disease prediction is performed, the acquisition time of the current leaf image is used as the time alignment benchmark, and the agricultural environmental information sequence preceding that time is extracted as the time-series input window. By using the leaf image acquisition time as the time anchor point and selecting agricultural environmental time-series data within the preset time window, a correspondence can be established between the environmental evolution process and leaf phenotypic changes in the time dimension. This overcomes the shortcomings of single data analysis or simple data aggregation processing in existing pest and disease prediction schemes, thereby improving the effectiveness of multimodal data fusion analysis.

[0024] In this embodiment, the image acquisition device is preferably an RGB image acquisition device, used to acquire two-dimensional color image information of the leaf. Compared with hyperspectral image sensors, it has the advantages of lower data dimensionality, simpler acquisition link, lower computational overhead, and easier local processing and model inference on resource-constrained edge-side embedded microprocessors.

[0025] In one technical solution of the present invention, the edge AI analysis module uses an embedded microprocessor as the core processing unit. In this embodiment, the ESP32-S3 microcontroller is preferred. After the agricultural environment information and crop status information collected by the data acquisition module are sent to the edge AI analysis module, the edge AI analysis module performs time alignment, preprocessing, model inference and output of pest and disease prediction results.

[0026] Before agricultural environmental time-series data are input into the pest and disease prediction model, filtering, outlier removal, and normalization are performed. Minimum-maximum normalization is then applied based on the value range of each environmental parameter in the training dataset to reduce the impact of different dimensions and numerical ranges on subsequent inference. Before crop status information is input into the pest and disease prediction model, the rectangular region containing the main leaf body in the original image acquired by the image acquisition device is cropped. The cropped image is then scaled to a 64×64 pixel RGB image using bilinear interpolation, and the pixel values ​​are linearly mapped from 0 to 255 to the 0 to 1 range as input to the pest and disease prediction model.

[0027] like Figure 2 The pest and disease prediction model integrating multimodal agricultural feature extraction in this invention includes: an agricultural environmental information feature extraction branch, a crop state information feature extraction branch, an attention alignment fusion module, and a pest and disease prediction output module. The agricultural environmental information feature extraction branch is used to extract temporal features from agricultural environmental information to obtain a global temporal feature vector of the agricultural environment. The crop state information feature extraction branch is used to extract spatial features from crop state information to obtain spatial features of the crop state. The attention alignment fusion module is used to align and fuse the global temporal feature vector of the agricultural environment with the spatial features of the crop state to generate a unified multimodal joint feature. The pest and disease prediction output module is used to output the pest and disease prediction result based on the unified multimodal joint feature.

[0028] This invention deploys a pest and disease prediction model that integrates multimodal agricultural feature extraction on an edge-side embedded microprocessor. It extracts temporal global features of the agricultural environment from agricultural environmental information and spatial features from crop state information. An attention mechanism maps the temporal global features of the agricultural environment into query vectors, guiding the weighted fusion of crop spatial features. This allows the pest and disease prediction model to dynamically adjust the contribution of different leaf local spatial features to the prediction results based on the current environmental state, thereby improving the accuracy and reliability of pest and disease prediction results. Simultaneously, it achieves localized intelligent analysis of pest and disease prediction tasks, reducing reliance on cloud computing resources and network transmission conditions, and improving the real-time performance and continuity of the pest and disease prediction system in weak network environments.

[0029] like Figure 3 In this invention, the agricultural environmental information feature extraction branch adopts a lightweight temporal feature extraction structure of multi-scale dilated temporal convolution, including: a first 1×1 convolutional layer, a multi-scale temporal convolutional unit, a residual dilated temporal feature enhancement block, and a temporal global aggregation unit. This enables the extracted agricultural environmental temporal global feature vector to simultaneously characterize the overall trend of environmental change and peak change information, thereby improving the ability of agricultural environmental temporal data to characterize environmental factors related to the risk of pest and disease occurrence.

[0030] The first 1×1 convolutional layer is used to convert the dimension to... Agricultural environmental information is channel-mapped and feature-fused, mapped to a dimension of The environmental temporal embedding features are used to achieve preliminary feature combinations between different agricultural environmental parameters.

[0031] Multi-scale temporal convolutional units are used to extract multi-scale temporal features from each environmental temporal embedding feature. These include short-term variation convolutional branches, medium-term variation convolutional branches, and long-term dependency convolutional branches, which extract agricultural environmental temporal variation features at different time scales. The extracted multi-scale temporal features are concatenated along the channel dimension, and the concatenated features are then fused and dimensionally compressed to obtain a 60×32 multi-scale temporal environmental feature. This allows for the extraction of information including short-term fluctuations, trends over continuous time periods, and cumulative changes over longer time periods. Specifically, the short-term variation convolutional branch uses a one-dimensional depthwise convolution with a kernel size of 3 to extract short-term fluctuation features of environmental data from the environmental temporal embedding features; the medium-term variation convolutional branch uses a one-dimensional depthwise convolution with a kernel size of 5 to extract environmental change trends over continuous time periods from the environmental temporal embedding features; and the long-term dependency convolutional branch uses a one-dimensional convolution with dilation to expand the temporal receptive field and extract cumulative temperature and humidity changes over longer time periods from the environmental temporal embedding features.

[0032] The residual expansion temporal feature enhancement block is used to perform nonlinear feature enhancement and channel fusion on each multi-scale temporal environmental feature under the time series, so as to obtain enhanced temporal environmental features and thus improve the expressive power of agricultural environmental temporal features. The residual dilation temporal feature enhancement block in this invention includes: a dilated one-dimensional convolutional layer, a batch normalization layer, a ReLU nonlinear activation layer, a pointwise convolutional layer, and an identity residual block. The dilated one-dimensional convolutional layer has a kernel size of 3 and a dilation rate of 2, used to capture temporal environmental features containing contextual information from each multi-scale temporal environmental feature. The batch normalization layer normalizes the captured temporal environmental features containing contextual information. The ReLU nonlinear activation layer performs a nonlinear transformation on the normalized temporal environmental features. The pointwise convolutional layer performs pointwise convolution on the nonlinearly transformed temporal environmental features. The identity residual block adds the pointwise convolutional temporal environmental features element-wise to the multi-scale temporal environmental features to preserve the original multi-scale temporal features and alleviate feature degradation during convolution, resulting in enhanced temporal environmental features.

[0033] The temporal global aggregation unit is used to aggregate the enhanced temporal environmental features in a temporal order to obtain the agricultural environment temporal global feature vector. Specifically, the temporal global aggregation unit includes global average pooling, global max pooling, and linear mapping. Global average pooling is used to extract the overall change trend information within the entire time window from the enhanced temporal environmental features. Global max pooling is used to extract the peak change information from the enhanced temporal environmental features. Linear mapping is used to concatenate the outputs of global average pooling and global max pooling to map a 32-dimensional agricultural environment temporal global feature vector.

[0034] like Figure 4 In this invention, the agricultural environmental information feature extraction branch adopts a lightweight spatial feature extraction structure based on multi-scale inverse residual convolution, including: overlapping convolution embedding unit, several multi-scale inverse residual units, multi-scale feature alignment and fusion unit, and tensor renormalization unit.

[0035] The overlapping convolutional embedding unit is used to perform shallow feature extraction and spatial downsampling of crop state information based on adjacent convolutional windows with overlapping regions, obtaining the initial spatial features of the crop. This reduces the spatial resolution of the image while preserving details of leaf edges, texture, and early lesions. The overlapping convolutional embedding unit uses a 2D convolutional layer with a kernel size of 3×3, a stride of 2, padding of 1, and 16 output channels to scan local regions of the input leaf image, ensuring that adjacent convolutional windows overlap in both the height and width directions. This reduces the spatial resolution of the image while preserving details of leaf edges, texture, and early lesions. After processing by the overlapping convolutional embedding unit, the input leaf image is mapped from a 64×64×3 RGB image to an initial spatial feature map of size 32×32×16.

[0036] Several multi-scale inverted residual units extract spatial features at different spatial scales from the initial spatial features of the crop. Specifically, they include a first multi-scale inverted residual unit, a second multi-scale inverted residual unit, and a third multi-scale inverted residual unit to extract local leaf texture features, mesoscale lesion features, and high-level semantic features, respectively. This allows for the simultaneous characterization of spatial features of small lesion edges, local chlorotic texture, and large-scale insect phenotypic changes, thereby improving the ability to express features of key phenotypic regions such as lesions, chlorosis, and insect edges.

[0037] Each multi-scale inverse residual unit includes: a channel-expanding convolutional layer, a multi-scale deep convolutional layer, a feature concatenation layer, a channel fusion compression convolutional layer, and a residual connection block. The channel-expanding convolutional layer maps the initial spatial features of the crop from low-dimensional channels to a high-dimensional channel feature space, increasing the channel representation dimension of the input features. The multi-scale deep convolutional layer extracts spatial features of different scales from the high-dimensional channel feature space. The multi-scale deep convolutional layer includes: Depth convolution branches, The deep convolution branch and the two-dimensional holed deep convolution branch with dilation rate are used to extract the edges of small lesions, local chlorotic textures and large-scale insect phenotypic changes in leaf images, respectively; the feature splicing layer is used to splice spatial features of different scales in the channel dimension; the channel fusion compression convolution layer is used to perform channel fusion on the spliced ​​multi-scale spatial features, map them to a preset number of channels, and obtain spatial features of the corresponding spatial scale.

[0038] The multi-scale feature alignment and fusion unit is used to align spatial features at different scales and unify their channel dimensions to obtain a crop fused spatial feature map. Specifically, a 32×32×16 local texture feature map is aligned to 8×8 through two downsampling operations, and its channel number is mapped to 32 through a 1×1 convolution, resulting in an 8×8×32 first aligned feature map. A 16×16×24 mesoscale lesion feature map is aligned to 8×8 through one downsampling operation, and its channel number is mapped to 32 through a 1×1 convolution, resulting in an 8×8×32 second aligned feature map. The 8×8×32 high-level semantic feature map is used as the third aligned feature map. Subsequently, the first, second, and third aligned feature maps are concatenated along the channel dimension to obtain an 8×8×96 multi-scale concatenated feature map. Then, a 1×1 convolution is used to perform channel fusion and dimensional compression on the multi-scale concatenated feature map to obtain an 8×8×32 fused spatial feature map.

[0039] Tensor remodeling units are used to perform tensor remodeling operations on crop fusion spatial feature maps, transforming the fusion spatial feature maps into... The crop space tensor.

[0040] like Figure 5 In this invention, the attention alignment fusion module integrates the temporal global feature vector of the agricultural environment. Transform into a query vector through linear mapping crop state-space features Converted into a bond matrix via linear mapping respectively Sum matrix According to the query vector of the linear mapping Key matrix Sum matrix Generate unified multimodal joint features:

[0041] in, , and Let each represent a learnable linear mapping matrix. Represents the query vector AND key matrix The length of the vector corresponding to the dot product calculation. express function, This indicates the transpose operation.

[0042] Through the attention alignment process described above, agricultural environmental information is no longer merely an additional input, but rather a query vector generated from the agricultural environmental time-series global feature vector. The weights involved in the crop state space feature calculation are determined based on the different temporal characteristics of the agricultural environment corresponding to different environmental conditions. Therefore, the generated query vector... The attention weights change with the input environment, leading to different distributions and dynamically adjusting the contribution of local spatial features in the multimodal joint feature representation. As a result, the pest and disease prediction model can highlight the spatial features corresponding to local leaf phenotypic features highly correlated with the current environmental state, such as the spatial features of lesions, chlorosis, or pest-related areas, thereby improving the accuracy of pest and disease prediction.

[0043] The pest and disease prediction output module employs a two-layer fully connected network structure to generate the final pest and disease prediction results. Specifically, the first fully connected layer receives data of the aforementioned size... The multimodal joint features are mapped to a 16-dimensional hidden feature vector; subsequently, the second fully connected layer combines this 16-dimensional hidden feature vector with... The activation function calculates and outputs a pest and disease risk probability value between 0 and 1. The calculation process is as follows:

[0044]

[0045] in, and These represent the weights and biases of the first fully connected layer, respectively. and These represent the weights and biases of the second fully connected layer, respectively. This represents a 16-dimensional hidden feature vector. This represents the probability value of pest and disease risk, and .

[0046] In one technical solution of the present invention, a pest and disease prediction model that integrates multimodal agricultural feature extraction is trained on a cloud platform and deployed at the edge. That is, it is trained using historical agricultural monitoring data in a host computer or cloud environment. After training, the pest and disease prediction model is lightweighted and then deployed to the edge AI analysis module for operation.

[0047] In the training process of the pest and disease prediction model, a classification loss function is used to optimize the model. The classification loss function is used to characterize the error between the pest and disease prediction results output by the model and the corresponding pest and disease labels. The classification loss function adopts the binary cross-entropy loss function, and its calculation process is expressed as follows:

[0048] in, Represents the classification loss function. Indicates the number of training samples. Indicates the first The disease and pest labels corresponding to each training sample. This indicates that the pest and disease prediction model is effective for the first... The model outputs the probability values ​​of pest and disease risk for each training sample. By continuously adjusting the network parameters of the pest and disease prediction model, the value of the classification loss function is gradually reduced, thereby making the probability values ​​of pest and disease risk output by the model gradually approach the corresponding pest and disease labels.

[0049] In one technical solution of the present invention, the lightweighting process includes: model pruning and model quantization; in: The model pruning employs a structured pruning approach, comprehensively removing redundant convolutional kernels and feature channels from the network. In this invention, the agricultural environmental information feature extraction branch and the crop state information feature extraction branch in the trained pest and disease prediction model are pruned to reduce the model parameter size and computational load. Specifically: The importance of each channel in the convolutional layer of the agricultural environmental information feature extraction branch and the crop state information feature extraction branch is evaluated separately, and channels whose importance is lower than the set channel importance threshold are pruned. The process for evaluating the importance of each channel in a convolutional layer is as follows:

[0050] in, This represents the first feature extraction branch for agricultural environmental information or the feature extraction branch for crop status information. The first convolutional layer The importance score of each channel This represents the first feature extraction branch for agricultural environmental information or the feature extraction branch for crop status information. The first convolutional layer Each channel corresponds to a scaling factor for the batch normalization layer. This represents the number of training samples used to evaluate the channel importance in the convolutional layer. express index, Indicates the first The training sample is processed by the agricultural environmental information feature extraction branch or the crop status information feature extraction branch. The first convolutional layer Channel response for each channel, This indicates the calculation of the mean.

[0051] By combining the channel importance evaluation of the agricultural environmental information feature extraction branch with the agricultural environmental time-series input data, and the channel importance evaluation of the crop state information feature extraction branch with the leaf image input data, the problem of insufficient consideration of input modal differences caused by pruning based solely on a single scaling factor is avoided. For each output channel within the same convolutional layer to be pruned, the lower the channel importance score, the weaker the scaling contribution and actual activation response of that channel under the corresponding input modality, and the more likely it is to be pruned. After pruning, the pruned model is fine-tuned using historical agricultural monitoring data to recover some of the accuracy loss caused by channel pruning.

[0052] Model quantization is used to quantize the weight parameters and activation values ​​in the trained FP32 floating-point pest and disease prediction model into INT8 format. This reduces model storage size and inference computation overhead, making the pest and disease prediction model suitable for execution on resource-constrained edge-side embedded microprocessors. Model quantization employs a full-integer post-training quantization method. Specifically, FP32 floating-point models are first trained using historical agricultural monitoring data. Then, a calibration dataset matching the actual deployment scenario is extracted from the historical agricultural monitoring data. This calibration dataset is used to perform forward inference on the pest and disease prediction model to statistically analyze the numerical ranges of the input, output, and intermediate activation tensors at each layer. Based on these numerical ranges, the quantization parameters of each layer's tensors are determined, and the weight parameters and activation values ​​in the FP32 floating-point pest and disease prediction model are quantized into INT8 format to reduce model storage size and inference computation overhead. When the model accuracy decreases beyond a preset range after quantization, a quantization-aware training method is used to fine-tune the model parameters.

[0053] The parameters of the pruned and quantized pest and disease prediction model are sent to the edge AI analysis module via the communication interaction module. To reduce the overhead of dynamic memory allocation during operation, a fixed-size contiguous tensor storage area is pre-allocated in the on-chip static random access memory of the AI ​​analysis module's microcontroller to store model weights, intermediate activation tensors, and temporary data during inference. Through the aforementioned pruning and quantization processes, the model parameter file size is controlled within 600KB, and the peak memory usage during runtime is controlled within 350KB, enabling the pest and disease prediction model to complete pest and disease prediction inference on resource-constrained embedded microprocessors.

[0054] In one technical solution of the present invention, the initial value of the pest and disease risk threshold is set to 0.70, and can be adjusted by the cloud platform through the communication interaction module by issuing parameter configuration instructions; when the risk probability value output by the model is greater than or equal to the currently set pest and disease risk threshold, it is determined that the current state is high-risk, the system generates pest and disease early warning information, and uploads the early warning information, the corresponding pest and disease risk probability value, and the agricultural monitoring data summary corresponding to this prediction to the cloud platform; when the risk probability value output by the model is less than the currently set pest and disease risk threshold, it is determined that the current state is low-risk, the system records the prediction result at the local edge node, and uploads the most recent prediction result and the agricultural monitoring data summary to the cloud platform according to the preset data upload cycle.

[0055] In one technical solution of this invention, a communication interaction module is used to realize the remote transmission and parameter interaction of agricultural monitoring data, pest and disease prediction results, and early warning information. The system uploads pest and disease prediction results, early warning information, and agricultural monitoring data to a cloud platform via a wireless network for display, storage, and remote viewing. Simultaneously, the cloud platform can issue parameter configuration commands to the system to adjust pest and disease early warning thresholds, image acquisition cycles, and data upload cycles. The agricultural environmental data sampling cycle remains unchanged according to pre-set model input rules. Even in cases of unstable or interrupted network connections, the system can still complete pest and disease monitoring and early warning tasks locally, thereby reducing reliance on cloud computing resources and continuous network connectivity.

[0056] In one technical solution of the present invention, such as Figure 6 Furthermore, a method for predicting agricultural pests and diseases based on multimodal data fusion and edge AI is provided, including the following steps: Step S1: Collect agricultural monitoring data containing agricultural environmental information and crop status information; Step S2: Construct an edge AI analysis module and deploy a pest and disease prediction model that integrates multimodal agricultural feature extraction in the AI ​​analysis module. This model is used to perform multimodal joint feature extraction on agricultural monitoring data containing agricultural environmental information and crop status information, and to infer pest and disease prediction results based on the multimodal joint features. Finally, pest and disease early warning information is generated based on the pest and disease prediction results. Step S3: Upload the collected agricultural monitoring data, pest and disease prediction results, and pest and disease early warning information to the cloud platform.

[0057] This invention, based on multimodal data fusion and edge AI, enables the fusion and analysis of agricultural environmental time-series data and crop status information, and completes pest and disease prediction inference on a resource-constrained edge-side embedded microprocessor. Compared with solutions that rely solely on cloud analysis or simply aggregate multi-source data, this invention can more fully explore the correlation between agricultural environmental information and crop status information, improve the accuracy of pest and disease prediction, and reduce the system's dependence on network conditions and cloud computing resources, thus demonstrating good engineering feasibility.

[0058] In one technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program that enables a computer to execute an agricultural pest and disease prediction method based on multimodal data fusion and edge AI.

[0059] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an agricultural pest and disease prediction method based on multimodal data fusion and edge AI.

[0060] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, and portable compact disc read-only memory (CD). ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0061] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0062] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An agricultural pest and disease prediction system based on multimodal data fusion and edge AI, characterized in that, include: Data acquisition module, edge AI analysis module, and communication interaction module; The data acquisition module is used to collect agricultural monitoring data that includes agricultural environmental information and crop status information; The edge AI analysis module is used to call the pest and disease prediction model that integrates multimodal agricultural feature extraction to perform multimodal joint feature extraction on agricultural monitoring data containing agricultural environmental information and crop status information, and to infer pest and disease prediction results based on the multimodal joint features, and generate pest and disease early warning information based on the pest and disease prediction results. The communication interaction module is used to upload agricultural monitoring data collected by the data acquisition module and pest and disease prediction results and pest and disease early warning information output by the edge AI analysis module to the cloud platform.

2. The agricultural pest and disease prediction system based on multimodal data fusion and edge AI according to claim 1, characterized in that, The pest and disease prediction model that integrates multimodal agricultural feature extraction includes: an agricultural environmental information feature extraction branch, a crop state information feature extraction branch, an attention alignment fusion module, and a pest and disease prediction output module; The agricultural environmental information feature extraction branch is used to extract time-series features from agricultural environmental information to obtain a global time-series feature vector of the agricultural environment. The crop state information feature extraction branch is used to extract spatial features from crop state information to obtain crop state spatial features; The attention alignment and fusion module is used to align and fuse the agricultural environment temporal global feature vector with the crop state space features to generate a unified multimodal joint feature. The pest and disease prediction output module is used to output pest and disease prediction results based on unified multimodal joint features.

3. The agricultural pest and disease prediction system based on multimodal data fusion and edge AI according to claim 2, characterized in that, The agricultural environmental information feature extraction branch adopts a lightweight temporal feature extraction structure of multi-scale dilated temporal convolution, including: a first 1×1 convolutional layer, a multi-scale temporal convolutional unit, a residual dilated temporal feature enhancement block, and a temporal global aggregation unit; The first 1×1 convolutional layer is used to map agricultural environmental information into environmental temporal embedding features; The multi-scale temporal convolutional unit is used to extract multi-scale temporal features from each environment temporal embedding feature in the temporal sequence, concatenate the extracted multi-scale temporal features in the channel dimension, and perform channel fusion and dimensional compression on the concatenated features to obtain multi-scale temporal environment features in the temporal sequence. The residual expansion temporal feature enhancement block is used to perform nonlinear feature enhancement and channel fusion on each multi-scale temporal environment feature under the time series, so as to obtain enhanced temporal environment features; The temporal global aggregation unit is used to aggregate the enhanced temporal environmental features in a temporal order to obtain the agricultural environment temporal global feature vector.

4. The agricultural pest and disease prediction system based on multimodal data fusion and edge AI according to claim 3, characterized in that, The residual dilation temporal feature enhancement block includes: a dilated one-dimensional convolutional layer, a batch normalization layer, a ReLU nonlinear activation layer, a pointwise convolutional layer, and an identity residual block; The dilated one-dimensional convolutional layer is used to capture temporal context features containing contextual information from each multi-scale temporal context feature; The batch normalization layer is used to normalize the captured temporal environmental features containing contextual information; The ReLU nonlinear activation layer is used to perform nonlinear transformation on the normalized temporal environmental features; The pointwise convolutional layer is used to perform pointwise convolution on the temporal environmental features of the nonlinear transformation; The identity residual block is used to add the pointwise convolutional temporal environmental features to the multi-scale temporal environmental features element-wise to obtain enhanced temporal environmental features.

5. The agricultural pest and disease prediction system based on multimodal data fusion and edge AI according to claim 2, characterized in that, The agricultural environmental information feature extraction branch adopts a lightweight spatial feature extraction structure based on multi-scale inverse residual convolution, including: overlapping convolution embedding unit, several multi-scale inverse residual units, multi-scale feature alignment and fusion unit, and tensor renormalization unit; The overlapping convolutional embedding unit is used to perform shallow feature extraction and spatial downsampling of crop state information based on adjacent convolutional windows with overlapping regions to obtain the initial spatial features of the crop. The aforementioned multi-scale inverse residual units extract spatial features of different spatial scales from the initial spatial features of the crop; The multi-scale feature alignment and fusion unit is used to align spatial features of different scales in terms of spatial size and unify channel dimensions to obtain a crop fusion spatial feature map. The tensor remodeling unit is used to perform tensor remodeling operations on the crop fusion spatial feature map to convert it into a crop spatial tensor.

6. The agricultural pest and disease prediction system based on multimodal data fusion and edge AI according to claim 5, characterized in that, Each multi-scale inverse residual unit includes: a channel-expanding convolutional layer, a multi-scale deep convolutional layer, a feature splicing layer, a channel-fusion compressive convolutional layer, and a residual connection block; The channel-expanded convolutional layer is used to map the initial spatial features of the crop from low-dimensional channels to high-dimensional channel feature space; The multi-scale deep convolutional layer is used to extract spatial features of different scales from the high-dimensional channel feature space; The feature splicing layer is used to splice spatial features of different scales along the channel dimension; The channel fusion compression convolutional layer is used to perform channel fusion on the spliced ​​multi-scale spatial features, mapping them to a preset number of channels to obtain spatial features of the corresponding spatial scale.

7. The agricultural pest and disease prediction system based on multimodal data fusion and edge AI according to claim 2, characterized in that, The attention alignment and fusion module converts the agricultural environment time-series global feature vector into a query vector through a linear mapping. The crop state space features are transformed into key matrices through linear mapping. Sum matrix According to the query vector of the linear mapping Key matrix Sum matrix Generate unified multimodal joint features: in, Represents the query vector AND key matrix The length of the vector corresponding to the dot product calculation. express function, This indicates the transpose operation.

8. The agricultural pest and disease prediction system based on multimodal data fusion and edge AI according to claim 2, characterized in that, The pest and disease prediction model integrating multimodal agricultural feature extraction is trained on a cloud platform. After training, the agricultural environment information feature extraction branch and crop status information feature extraction branch in the pest and disease prediction model are pruned. The parameters of the pruned pest and disease prediction model integrating multimodal agricultural feature extraction are then sent to the edge AI analysis module through the communication interaction module.

9. An agricultural pest and disease prediction system based on multimodal data fusion and edge AI according to claim 8, characterized in that, The importance of each channel in the convolutional layer of the agricultural environmental information feature extraction branch and the crop state information feature extraction branch is evaluated separately, and channels whose importance is lower than the set channel importance threshold are pruned. The process for evaluating the importance of each channel in the convolutional layer is as follows: in, This represents the first feature extraction branch for agricultural environmental information or the feature extraction branch for crop status information. The first convolutional layer The importance score of each channel This represents the first feature extraction branch for agricultural environmental information or the feature extraction branch for crop status information. The first convolutional layer Each channel corresponds to a scaling factor for the batch normalization layer. This represents the number of training samples used to evaluate the channel importance in the convolutional layer. express index, Indicates the first The training sample is processed by the agricultural environmental information feature extraction branch or the crop status information feature extraction branch. The first convolutional layer Channel response for each channel, This indicates the calculation of the mean.

10. A method for predicting agricultural pests and diseases based on multimodal data fusion and edge AI, characterized in that, Includes the following steps: Step S1: Collect agricultural monitoring data containing agricultural environmental information and crop status information; Step S2: Construct an edge AI analysis module and deploy a pest and disease prediction model that integrates multimodal agricultural feature extraction in the AI ​​analysis module. This model is used to perform multimodal joint feature extraction on agricultural monitoring data containing agricultural environmental information and crop status information, and to infer pest and disease prediction results based on the multimodal joint features. Finally, pest and disease early warning information is generated based on the pest and disease prediction results. Step S3: Upload the collected agricultural monitoring data, pest and disease prediction results, and pest and disease early warning information to the cloud platform.

Images