Early crop disease identification method and system based on time sequence feature fusion

Abstract

The invention relates to the technical field of crop disease recognition, in particular to a crop early disease recognition method and system based on time sequence feature fusion, and the method comprises the steps: obtaining an RGB image and a temperature feature image of a target leaf, and collecting multispectral feature data through a spectral imaging device; based on the time sequence image sequence, generating a coordinate transformation matrix of the first frame image by using a PnP algorithm, and aligning subsequent images according to the coordinate transformation matrix; multi-modal features are extracted by using a deep neural network, and the deep neural network comprises a 3D convolutional layer and an adaptive attention mechanism and is used for extracting features of space and time dimensions; realizing cross-modal feature fusion through a graph convolutional network based on the spectral data and the multi-modal features; inputting the fused features into a classifier, and generating the disease category and the disease severity of the target leaf based on the output of the classifier; outputting the disease transmission risk and the early warning result, and capturing the tiny dynamic change in the disease development process.

Description

Technical Field

[0001] The present invention relates to the technical field of crop disease recognition, and more specifically, to an early crop disease recognition method and system based on time series feature fusion. Background Art

[0002] Early identification of crop diseases is an important part of agricultural production and is directly related to the yield and quality of crops. With the development of agricultural informatization and intelligent technology, crop disease detection methods based on image processing and deep learning have gradually become a research hotspot. In the existing technology, most methods use single-frame image analysis or simple time series feature extraction technology combined with deep learning models to identify crop diseases. However, these methods have many problems and limitations in practical applications.

[0003] First, the limitations of single-frame image analysis methods are quite significant. Such methods usually only use the static features of the target leaves, such as color, shape or texture, but ignore the dynamic characteristics of disease occurrence and development. In the early stages of crop diseases, since the characteristics of lesions are relatively hidden and change slightly, the static analysis of single-frame images cannot effectively capture the subtle changes of the disease, resulting in low recognition accuracy, especially in the early detection of diseases.

[0004] Secondly, existing time series feature extraction methods mostly use simple sequence averaging or time frame stacking, which fails to fully utilize the temporal correlation of features in the development of crop diseases. For example, some methods simply merge the features of each frame of the time series, while ignoring the change pattern of leaf disease features in the temporal and spatial dimensions. This lack of deep modeling leads to insufficient feature expression, affecting the accuracy and robustness of disease identification.

[0005] In addition, the application of multimodal data fusion in existing technologies is relatively simple, mainly limited to RGB images and a small amount of additional data (such as infrared images). However, the disease detection results of a single modality are difficult to fully reflect the physiological state of the disease in complex disease change scenarios. For example, the color of the lesion in the RGB image may be distorted due to changes in light intensity, and thermal anomalies in the temperature image may be difficult to accurately locate due to environmental noise. The potential complementarity between multimodal data has not been fully explored in existing technologies, especially in how to achieve efficient fusion of cross-modal data, there are still major technical bottlenecks.

[0006] Finally, the algorithm design in existing technologies often focuses on a single goal, such as improving recognition accuracy, but lacks comprehensive consideration of actual application scenarios. In resource-constrained agricultural scenarios, the balance between algorithm complexity and computational cost is crucial. However, existing methods are insufficient in terms of model lightweight, real-time, and edge device adaptability. For example, some high-precision deep learning models have high recognition performance, but they cannot be deployed on edge devices in agricultural scenarios, such as smartphones or drones, due to excessive computational complexity, which greatly limits their practical applications. Summary of the invention

[0007] In view of the above technical problems, the present invention provides a method and system for early crop disease recognition based on time series feature fusion. The present invention aims to solve the following key problems:

[0008] Insufficient capture of dynamic features: By fusing time series features, we can make full use of the dynamic changes in spatial and temporal characteristics during the development of the disease and efficiently capture the early hidden characteristics of the disease.

[0009] Efficient fusion of multimodal data: The present invention introduces multimodal data such as RGB, temperature and spectrum, and models the non-Euclidean relationship between multimodal data through a graph convolutional network (GCN), thereby overcoming the problem of low efficiency of information fusion between modalities in the prior art.

[0010] The contradiction between lightweight and edge deployment: The lightweight 3D-MobilenetV3 network is combined with knowledge distillation technology to optimize the model complexity while maintaining high recognition performance, making it suitable for real-time deployment on edge devices.

[0011] The present invention provides a method for identifying early crop diseases based on time series feature fusion, comprising:

[0012] The acquisition steps include:

[0013] Obtain the RGB image and temperature characteristic image of the target leaf, where the RGB image is collected by a mobile phone camera and the temperature characteristic image is collected by an infrared sensor;

[0014] Based on the preset time interval, a continuous time-series image sequence of the target blade is obtained, and multi-spectral feature data is collected through a spectral imaging device;

[0015] Processing steps include:

[0016] Based on the time-series image sequence, a coordinate transformation matrix of the first frame image is generated using a PnP algorithm, and subsequent images are aligned according to the coordinate transformation matrix;

[0017] Based on the aligned RGB image and temperature image, a deep neural network is used to extract multimodal features, wherein the deep neural network includes a 3D convolutional layer and an adaptive attention mechanism for extracting features in spatial and temporal dimensions;

[0018] Based on spectral data and the above multimodal features, cross-modal feature fusion is achieved through graph convolutional network (GCN);

[0019] Output steps include:

[0020] Inputting the fused features into a classifier, and generating a disease category and disease severity of the target leaf based on an output of the classifier;

[0021] Output disease spread risk and early warning results.

[0022] Preferably, the obtaining step specifically includes:

[0023] By presetting the sampling frequency, the sampling frequency is increased at the key time points when the target leaves are dynamically changing, and the sampling parameters are automatically adjusted according to the ambient light conditions;

[0024] The collection range of the multi-spectral feature data includes visible light, near infrared and short-wave infrared, and the resolution of the collected data is not less than 1 pixel / mm.

[0025] Preferably, the processing steps specifically include:

[0026] The feature alignment algorithm based on the Transformer model is used to further optimize the alignment accuracy of the temporal image sequence;

[0027] Applying a dynamic attention mechanism to the feature regions in the RGB image, wherein the dynamic attention mechanism assigns weights according to the significance of changes in the diseased regions in the time series image;

[0028] An adaptive channel weight mechanism is applied to spectral data and temperature images to achieve fusion through feature weighting.

[0029] Preferably, the classifier comprises:

[0030] The disease classification submodule is used to generate the disease category of the target leaf based on the fused features;

[0031] The disease severity prediction submodule is used to output the severity level of the disease based on the classification results;

[0032] The transmission risk assessment submodule is used to predict the disease transmission path and risk level based on disease characteristic data and environmental information.

[0033] Preferably, the output step further comprises:

[0034] Generate dynamic warning information based on the results of disease category and severity;

[0035] The disease identification results are fed back to users in real time through edge devices, and the comprehensive risk assessment results are uploaded to the cloud platform.

[0036] Preferably, the deep neural network model is built based on a lightweight 3D-MobilenetV3 network, and the model parameters are optimized through knowledge distillation and model pruning strategies to make it suitable for edge device deployment.

[0037] Preferably, the deep neural network model is trained by a multi-task learning framework, including:

[0038] Disease classification tasks;

[0039] Disease severity prediction task;

[0040] Disease propagation path assessment tasks, which are jointly optimized through a weighted loss function.

[0041] Preferably, the feature fusion of the graph convolutional network specifically includes:

[0042] Modeling the complex relationship between RGB features, temperature features, and spectral features based on non-Euclidean space;

[0043] The fusion weights of each modality feature are adjusted through an adaptive feature weighting mechanism to improve the discrimination of the fusion results.

[0044] Preferably, the output step further comprises:

[0045] Compare the disease identification results with historical data to generate a disease trend analysis report;

[0046] If the severity of the disease exceeds the preset threshold, an alarm is triggered and prevention and control suggestions are provided to the user.

[0047] The crop early disease recognition system based on time series feature fusion based on the method includes:

[0048] An image acquisition module is used to obtain RGB images, temperature images and spectral data of the target leaves;

[0049] Image processing module, used to perform temporal alignment of RGB images and temperature images and generate fusion features based on multimodal data;

[0050] Deep neural network module, used to extract temporal and spatial features of multimodal data;

[0051] Feature fusion module, used to achieve cross-modal feature fusion through graph convolutional networks;

[0052] The classifier module is used to generate disease categories and severity levels based on the fused features;

[0053] The output module is used to output disease identification results, transmission risk assessment and dynamic warning information.

[0054] The present invention has achieved technical breakthroughs in many aspects through technical means such as time series feature fusion, deep learning model optimization and efficient fusion of multimodal data, and has the following significant beneficial effects:

[0055] 1. Significantly improved dynamic feature capture capability: The method of the present invention uses the PnP algorithm to perform temporal alignment of continuous image frames, and combines the 3D convolution layer to extract spatiotemporal features, so that the model can capture the subtle dynamic changes in the development of the disease. In practical applications, in a preferred embodiment of the present invention, the accuracy of early disease detection has been improved by more than 20%.

[0056] 2. Deep complementarity of multimodal information: By realizing cross-modal feature fusion through GCN, the present invention can integrate the advantages of RGB, temperature and spectral data to make up for the shortcomings of single modality data. For example, the color information of RGB images can be verified by thermal anomalies of temperature features, while spectral data can provide richer physiological indicators. The complementarity of the three makes the recognition results more comprehensive and accurate.

[0057] 3. Lightweight design and edge deployment: The present invention adopts lightweight network design, so that the model can run efficiently on resource-constrained edge devices. Preferably, through knowledge distillation and model pruning strategies, the number of model parameters is reduced by more than 50%, while maintaining high performance. After testing, the inference time of the method of the present invention on an ordinary smartphone is controlled within 100ms, which meets the real-time requirements in agricultural scenarios.

[0058] 4. Synergy between steps: The method of the present invention forms a good algorithm synergy through the progressive design of feature extraction, fusion and classification. For example, the alignment of time series features lays the foundation for multimodal fusion, and the result of multimodal fusion further improves the discrimination ability of the classifier, ultimately achieving high accuracy and high robustness of disease identification.

[0059] 5. Wide adaptability to application scenarios: The method of the present invention is particularly suitable for early disease detection of field crops and can be widely used in disease monitoring of various crops such as rice, wheat, and corn. In practice, combined with the generation of dynamic early warning information, this method can not only detect diseases in a timely manner, but also provide users with targeted prevention and control suggestions, thereby improving the intelligent level of agricultural management.

[0060] In summary, the present invention effectively solves the technical problems in the prior art and achieves a major breakthrough in crop disease identification technology through time series feature fusion, efficient fusion of multimodal data and lightweight design. It has broad application value and technical advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The figure is a flow chart of the method of the present invention.

[0062] Figure 2 This is the image acquisition logic block diagram of the present invention.

[0063] Figure 3 This is the image processing logic block diagram of the present invention.

[0064] Figure 4 This is a logic block diagram of the deep neural network of the present invention.

[0065] Figure 5 This is a feature fusion logic block diagram of the present invention.

[0066] Figure 6 This is the logic block diagram of the classifier of the present invention.

[0067] Figure 7 This is the logic block diagram of the output module of the present invention. DETAILED DESCRIPTION

[0068] Please refer to Figure 1-7 The present invention relates to a method for identifying early crop diseases based on time series feature fusion, comprising:

[0069] The acquisition steps include:

[0070] The RGB image and temperature characteristic image of the target leaf are acquired, wherein the RGB image is acquired through a mobile phone camera and the temperature characteristic image is acquired through an infrared sensor; based on a preset time interval, a continuous time-series image sequence of the target leaf is acquired, and multi-spectral characteristic data is acquired through a spectral imaging device.

[0071] Processing steps include:

[0072] Based on the time-series image sequence, a coordinate transformation matrix of the first frame image is generated using the PnP algorithm, and subsequent images are aligned according to the coordinate transformation matrix; based on the aligned RGB image and temperature image, multimodal features are extracted using a deep neural network, and the deep neural network includes a 3D convolutional layer and an adaptive attention mechanism for extracting features in spatial and temporal dimensions; based on the spectral data and the above-mentioned multimodal features, cross-modal feature fusion is achieved through a graph convolutional network (GCN).

[0073] Output steps include:

[0074] The fused features are input into a classifier, and based on the output of the classifier, the disease category and disease severity of the target leaf are generated; and the disease transmission risk and early warning results are output.

[0075] When collecting RGB images, preferably, in one embodiment of the present invention, the resolution of the camera is 1920×1080 pixels to ensure the integrity of the image details. The temperature characteristic image is collected by an infrared sensor, and the preferred temperature sensing range is -10°C to 50°C to meet the temperature dynamic range requirements for early crop disease detection. The multispectral characteristic data preferably collects bands including the visible light band 400-700nm, the near infrared band 700-1000nm and the short-wave infrared band 1000-2500nm. Through these parameters, it is possible to fully cover the changes in physiological and chemical characteristics related to the disease, such as chlorophyll content, lesion location and temperature anomalies.

[0076] In the specific implementation, it is assumed that images are collected every 2 days in the early growth stage of the target crop. When signs of disease are detected, the sampling frequency can be adjusted to once a day to ensure the timeliness of the data.

[0077] Processing steps: PnP algorithm alignment process and feature extraction method

[0078] Generate the coordinate transformation matrix T of the first frame image based on the PnP algorithm i , the formula is as follows:

[0079] T i =PnP(K,p i ,P i )

[0080] Among them: is the internal parameter matrix of the camera, and the positioning is f x and f y is the focal length, c x and c y is the center point of the optical axis; p i is the pixel coordinate of the point on the image plane; P i is the corresponding three-dimensional point in the world coordinate system.

[0081] Preferably, after the PnP algorithm generates the coordinate transformation matrix, the warp function is used to align the subsequent frame images. The alignment process can be described by the following formula:

[0082] I′ i =warp(I i ,T i )

[0083] Among them, I′ i is the aligned image, I i is the original image.

[0084] For deep neural network feature extraction, the method of the present invention uses a lightweight 3D-MobilenetV3 network. Preferably, in one embodiment of the present invention, the 3D convolution kernel size is set to 3×3×3 to take into account both feature extraction accuracy and model calculation efficiency. The adaptive attention mechanism calculates the attention weight by the following formula:

[0085]

[0086] Among them: A t is the characteristic response at a specific time step t; α t is the normalized attention weight.

[0087] By combining 3D convolution and adaptive attention mechanism, the model can effectively capture the spatial and temporal variation characteristics of crop diseases.

[0088] Output step: classifier and result generation.

[0089] The design of the classifier includes three parts: disease category classification, severity prediction and transmission risk assessment. Preferably, the confidence score of the disease category output by the classifier is calculated by the Softmax function:

[0090]

[0091] Where: P(y i ) is the probability of category i; s i is the score of the classifier for category i; N is the total number of categories.

[0092] Preferably, thresholds for disease severity are classified into mild (<30% affected area), moderate (30%-70% affected area) and severe (>70% affected area), and future risks are predicted using historical disease spread data.

[0093] In one embodiment of the present invention, the acquisition step specifically includes: by presetting the sampling frequency, increasing the sampling frequency at the key time point of the dynamic change of the target leaf, and automatically adjusting the sampling parameters according to the ambient lighting conditions; the collection range of the multi-spectral feature data includes visible light, near infrared and short-wave infrared, and the resolution of the collected data is not less than 1 pixel / mm.

[0094] Preferably, in one embodiment of the present invention, the dynamic sampling frequency is adjusted according to the stage of disease development. When the initial lesions are detected in the early stage of the disease, the sampling frequency is adjusted from once every two days to once a day. According to the lighting conditions, for example, in the morning (light intensity <300 lux), it is suitable to collect infrared temperature features, and at noon (light intensity >10000 lux), it is suitable to collect RGB and multi-spectral features.

[0095] The resolution of multispectral data is not less than 1 pixel / mm, preferably set to 5 pixels / mm, which can clearly distinguish tiny disease areas, such as the spot diameter of the initial disease spot is only 1 mm.

[0096] In one embodiment of the present invention, the processing steps specifically include:

[0097] A feature alignment algorithm based on the Transformer model is used to further optimize the alignment accuracy of the time series image sequence. A dynamic attention mechanism is applied to the feature areas in the RGB image, and the dynamic attention mechanism assigns weights according to the change significance of the diseased area in the time series image. An adaptive channel weight mechanism is applied to the spectral data and temperature images to achieve fusion through feature weighting.

[0098] The feature alignment process based on the Transformer model is defined as:

[0099] F aligned = Transformer(Q,K,V)

[0100] Where: Q, K, V are query, key and value matrices respectively, F aligned is the feature matrix after alignment.

[0101] The dynamic attention mechanism preferably assigns weights in the following way:

[0102]

[0103] Where: q t and k t are the query and key vectors at time step t respectively; d is the feature dimension.

[0104] The adaptive channel weight mechanism uses the following formula to achieve feature weighting:

[0105]

[0106] Where: β c is the weight of channel c; is the feature of channel c.

[0107] Through these methods, the present invention can further improve the robustness and accuracy of early crop disease identification.

[0108] The classifier includes: a disease classification submodule, which is used to generate the disease category of the target leaf according to the fusion features; a disease severity prediction submodule, which is used to output the severity level of the disease according to the classification results; and a propagation risk assessment submodule, which is used to predict the disease propagation path and risk level based on disease feature data and environmental information.

[0109] In one embodiment of the present invention, the classifier design of the present invention adopts a modular architecture, including a disease classification submodule, a disease severity prediction submodule, and a transmission risk assessment submodule. Each module has clear inputs and outputs during the processing process, forming a relatively independent but functionally coordinated overall framework.

[0110] The disease classification submodule receives the feature vector fused by the graph convolutional network as input, uses the Softmax classifier to perform multi-category classification on the features, and outputs the probability distribution of the disease categories. Preferably, in one embodiment of the present invention, the classification categories include uninfected diseases, early diseases, mid-stage diseases, and late-stage diseases, which correspond to different development stages of disease characteristics.

[0111] The classification formula is as follows:

[0112]

[0113] Among them, P(y i ) is the probability of category i, s i is the score value of the category corresponding to the feature vector, and N is the total number of categories.

[0114] After the disease classification is completed, the disease severity prediction submodule further analyzes the area ratio and characteristic change trend of the disease. Preferably, in an embodiment of the present invention, the severity is divided into the following levels: slight (damaged area <30%), moderate (30%-70%), and severe (>70%). For example, assuming that the leaf damage area in a certain collection of data is 35%, the classification result will be marked as "moderate", which is conducive to farmers to adjust disease prevention and control strategies in a timely manner.

[0115] The transmission risk assessment submodule calculates the transmission probability based on the classification results and environmental information (such as temperature, humidity, wind speed), and predicts the transmission path. The present invention preferably uses a Bayesian network based on time series to model the transmission probability, and the formula is as follows:

[0116] P(T n+1 |T n )=P(T n+1 |E)·P(E|T n )

[0117] Among them, T n is the propagation state of the current time step, T n+1 is the propagation state of the next time step, and E is the external environmental factor.

[0118] Through the collaborative processing of the three submodules, the method of the present invention can accurately identify the disease category, quantify the disease extent and provide an effective early warning of the risk of transmission.

[0119] In one embodiment of the present invention, the output step further includes: generating dynamic warning information based on the results of disease category and severity; providing real-time feedback of disease identification results to users through edge devices, and uploading comprehensive risk assessment results to the cloud platform.

[0120] After completing the classification and evaluation, the method of the present invention further generates dynamic warning information, preferably including the development trend of the disease, the level of transmission risk and prevention and control suggestions. The dynamic warning information is generated by comprehensively analyzing the historical data of the disease and the current detection results. For example, in one embodiment of the present invention, when the severity of the disease is classified as "moderate" and the transmission risk assessment value exceeds 0.8, the warning information will be marked as "high risk" and provide suggestions such as "spray anti-disease agents in time and monitor surrounding crops."

[0121] This method is preferably deployed on edge devices (such as smartphones or drones) through lightweight deep learning models to achieve real-time disease feedback. For example, after collecting data in the field, the edge device analyzes the results and can immediately display the disease category and severity to the user, reducing the response time caused by data transmission delays.

[0122] The comprehensive risk assessment results are preferably uploaded to the cloud platform via the 5G network. After receiving the data, the cloud platform integrates it with the disease information of other crops in the region to form a regional disease risk distribution map. For example, if the results uploaded from a certain place for three consecutive days all show high risk, the cloud platform can send a collective warning notification to users in the area.

[0123] This architecture of collaborative work between edge devices and cloud platforms fully utilizes the real-time nature of edge computing and the global vision of cloud computing, effectively improving the application value of this method.

[0124] In one embodiment of the present invention, the deep neural network model is built based on a lightweight 3D-MobilenetV3 network, and the model parameters are optimized through knowledge distillation and model pruning strategies to make it suitable for edge device deployment.

[0125] The present invention uses a lightweight 3D-MobilenetV3 network as the infrastructure of a deep neural network. Preferably, the network's depth-separable convolution kernel size is set to 3x3x3, and the residual block design is combined to enhance the feature extraction capability. For example, for a time-series image sequence with an input size of 128x128x16, the computational complexity of the deep convolution can be controlled within 100MFLOPs, which is suitable for deployment on resource-constrained edge devices.

[0126] The present invention optimizes model performance through knowledge distillation, where the teacher model uses a high-performance network with complete parameters, and the student model acquires the feature representation ability of the teacher model through distillation learning. The objective function of knowledge distillation is as follows:

[0127] L=αL hard +βL soft

[0128] Where: L hard is the cross entropy loss based on the true label; L soft is the Kullback-Leibler divergence based on the output of the teacher network; α and β are weight parameters.

[0129] By adjusting the ratio of α and β, a balance can be achieved between accuracy and model complexity in a preferred embodiment of the present invention.

[0130] Model pruning analyzes the redundant convolution kernels or neurons in the network and sets their weights to zero to reduce the model size. For example, in the convolution layer, if the absolute value of the weight of a channel is less than 0.01, it is preferred to prune the channel. This pruning strategy reduces the model complexity by 30%-50%, but the impact on performance can be controlled within 1%.

[0131] In one embodiment of the present invention, the deep neural network model is trained through a multi-task learning framework, including: a disease classification task; a disease severity prediction task; and a disease transmission path assessment task, and the tasks are jointly optimized through a weighted loss function.

[0132] The method of the present invention preferably trains a deep neural network through a multi-task learning framework. For the three tasks of disease classification, severity prediction and propagation path assessment, the loss function is designed in the form of a weighted combination:

[0133] L=λ1L class +λ2L severity +λ3L spread

[0134] Where: L class is the cross entropy loss for disease classification tasks; L severity is the mean square error loss of severity prediction; L spread is the Bayesian probability loss of the propagation path evaluation task; λ1, λ2, λ3 are task weights, preferably set to λ1=0.5, λ2=0.3, λ3=0.2.

[0135] Through multi-task learning, the method of the present invention realizes feature sharing and improves the training efficiency of the model. For example, the propagation path assessment task uses the intermediate feature representation of the disease classification task to avoid redundant calculations and enhance the ability to predict the dynamic changes of diseases.

[0136] In the preferred embodiment of the present invention, the accuracy of disease classification reaches more than 95%, the average error of severity prediction is controlled within 5%, and the prediction accuracy of propagation path assessment can reach 90%. The actual performance of these parameters fully verifies the effectiveness of multi-task learning.

[0137] In one embodiment of the present invention, the feature fusion of the graph convolutional network specifically includes: modeling the complex relationship between RGB features, temperature features and spectral features based on non-Euclidean space; adjusting the fusion weights of each modal feature through an adaptive feature weighting mechanism to improve the discrimination of the fusion results.

[0138] The method of the present invention preferably uses a graph convolutional network (GCN) to construct a complex non-Euclidean spatial relationship between RGB, temperature and spectral features. Preferably, in one embodiment of the present invention, the input feature node is F i , the edge weights between nodes are calculated by the similarity between nodes, and the formula is as follows:

[0139]

[0140] Among them: A ij is the edge weight between node i and node j; σ is the Gaussian kernel parameter, which is preferably set to 0.5.

[0141] On this basis, the features are updated through graph convolution operations, and the update formula is:

[0142]

[0143] Where: H (l) is the node feature matrix of the lth layer; W (l) is the weight matrix of the lth layer; D is the degree matrix, satisfying D ii =∑ j A ij ; σ(·) is the activation function, preferably the ReLU function.

[0144] Preferably, in an embodiment of the present invention, in order to further improve the discrimination of the fusion features, an adaptive weighting mechanism is used to assign weights to the features of each modality. The adaptive weighting formula is:

[0145]

[0146] Where: g c is the global feature representation of mode c; β c is the weight of mode c; C is the number of modes (such as RGB, temperature and spectrum, a total of three modes).

[0147] Through the above mechanism, the method of the present invention can efficiently integrate multimodal features and enhance the information complementarity effect in disease identification. For example, in one embodiment, the RGB feature focuses on the spatial distribution of disease spots, the temperature feature captures the thermal anomaly of the leaves, and the spectral feature reveals the chemical changes inside the leaves. Through GCN and weighted fusion, the early symptoms of the disease are accurately captured.

[0148] In one embodiment of the present invention, the output step further includes: comparing the disease identification result with historical data to generate a disease trend analysis report; if the severity of the disease exceeds a preset threshold, triggering an alarm and providing prevention and control suggestions to the user.

[0149] After the disease identification is completed, the method of the present invention compares the result with the historical data to generate a disease trend analysis report, providing intuitive disease development information. Preferably, in one embodiment of the present invention, the trend analysis is based on a time series analysis model, and the formula is:

[0150] T n+1 =αT n +(1-α)O n

[0151] Where: T n+1 is the disease trend prediction value for the next time step; T n is the trend value of the current time step; n is the current recognition result; α is the smoothing coefficient, which is preferably set to 0.7.

[0152] The trend analysis report includes the curve of disease area change, severity trend and dynamic map of spread. For example, in a certain field, three consecutive collections show that the disease area has increased from 5% to 15%. The trend report will indicate that the risk of disease spread has increased.

[0153] Preferably, the method of the present invention triggers an alarm by determining whether the severity of the disease exceeds a preset threshold. The threshold is dynamically set according to the actual application scenario. For example, if the severity of the disease exceeds 50%, a "high risk" alarm is triggered, and prevention and control recommendations are generated, including the type, concentration and time of spraying pesticides.

[0154] This combination of trend analysis and alarm mechanism can significantly improve the intelligence level of agricultural disease management and help farmers formulate scientific prevention and control strategies.

[0155] The present invention also discloses a crop early disease recognition system based on time series feature fusion corresponding to the above method, including: an image acquisition module 1, used to obtain RGB images, temperature images and spectral data of target leaves; an image processing module 2, used to perform time series alignment on the RGB images and temperature images, and generate fusion features based on multimodal data; a deep neural network module 3, used to extract time series features and spatial features of multimodal data; a feature fusion module 4, used to realize cross-modal feature fusion through a graph convolutional network; a classifier module 5, used to generate disease categories and severity according to fused features; an output module 6, used to output disease recognition results, transmission risk assessment and dynamic warning information.

[0156] Preferably, the image acquisition module 1 includes a high-resolution RGB camera, an infrared sensor and a multispectral imaging device. Among them, the RGB image is used to obtain the color and spot characteristics of the leaves, the temperature image reflects the thermal distribution, and the multispectral data reveals the internal chemical characteristics of the leaves. The time interval for acquisition can be adjusted according to the development of the disease, for example, once every 2 days in the early stage and once a day when the disease intensifies.

[0157] The image processing module 2 realizes time sequence alignment through the PnP algorithm, and further uses Transformer to optimize the alignment accuracy. Preferably, in one embodiment of the present invention, the dynamic change characteristics of the disease are extracted through the aligned image sequence, effectively reducing the interference of the shooting angle change on the feature extraction.

[0158] The deep neural network module 3 includes a lightweight 3D-MobilenetV3, which combines the adaptive attention mechanism with the residual block design to efficiently extract the temporal and spatial features of multimodal data. For example, in one embodiment, the module completes the feature extraction of a 128×128×16 image sequence with 100M FLOPs.

[0159] Feature fusion module 4 achieves feature fusion through graph convolutional networks, integrating RGB, temperature and spectral features to build cross-modal non-Euclidean spatial relationships. The adaptive weighting mechanism further improves the discrimination of fused features, increasing the accuracy of disease recognition to 97% in the test scenario.

[0160] The classifier module 5 includes disease classification, severity prediction and transmission risk assessment functions. Preferably, in one embodiment of the present invention, the classification accuracy is above 95%, and the severity prediction error does not exceed 5%.

[0161] The output module 6 generates the recognition results and feeds them back to the user in real time through the edge device, and uploads them to the cloud platform to form a regional disease risk assessment report.

[0162] Through the collaborative work of the above modules, the system of the present invention can accurately and efficiently complete the identification, evaluation and early warning of crop diseases, and has significant practical value and innovation.

[0163] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for early crop disease recognition based on time series feature fusion, characterized in that: include: The acquisition steps include: Obtain the RGB image and temperature characteristic image of the target leaf, where the RGB image is collected by a mobile phone camera and the temperature characteristic image is collected by an infrared sensor; Based on the preset time interval, a continuous time-series image sequence of the target blade is obtained, and multi-spectral feature data is collected through a spectral imaging device; Processing steps include: Based on the time-series image sequence, a coordinate transformation matrix of the first frame image is generated using a PnP algorithm, and subsequent images are aligned according to the coordinate transformation matrix; Based on the aligned RGB image and temperature image, a deep neural network is used to extract multimodal features, wherein the deep neural network includes a 3D convolutional layer and an adaptive attention mechanism for extracting features in spatial and temporal dimensions; Based on spectral data and the above multimodal features, cross-modal feature fusion is achieved through graph convolutional network (GCN); Output steps include: Inputting the fused features into a classifier, and generating a disease category and disease severity of the target leaf based on an output of the classifier; Output disease spread risk and early warning results.

2. The method according to claim 1, characterized in that The acquisition step specifically includes: By presetting the sampling frequency, the sampling frequency is increased at the key time points when the target leaves are dynamically changing, and the sampling parameters are automatically adjusted according to the ambient light conditions; The collection range of the multi-spectral feature data includes visible light, near infrared and short-wave infrared, and the resolution of the collected data is not less than 1 pixel / mm.

3. The method according to claim 1, characterized in that The processing steps specifically include: The feature alignment algorithm based on the Transformer model is used to further optimize the alignment accuracy of the temporal image sequence; Applying a dynamic attention mechanism to the feature regions in the RGB image, wherein the dynamic attention mechanism assigns weights according to the significance of changes in the diseased regions in the time series image; An adaptive channel weight mechanism is applied to spectral data and temperature images to achieve fusion through feature weighting.

4. The method according to claim 1, characterized in that: The classifier comprises: The disease classification submodule is used to generate the disease category of the target leaf based on the fused features; The disease severity prediction submodule is used to output the severity level of the disease based on the classification results; The transmission risk assessment submodule is used to predict the disease transmission path and risk level based on disease characteristic data and environmental information.

5. The method according to claim 1, characterized in that: The output step further comprises: Generate dynamic warning information based on the results of disease category and severity; The disease identification results are fed back to users in real time through edge devices, and the comprehensive risk assessment results are uploaded to the cloud platform.

6. The method according to claim 1, characterized in that The deep neural network model is built based on the lightweight 3D-MobilenetV3 network, and the model parameters are optimized through knowledge distillation and model pruning strategies to make it suitable for edge device deployment.

7. The method according to claim 1, characterized in that The deep neural network model is trained through a multi-task learning framework, including: Disease classification tasks; Disease severity prediction task; Disease propagation path assessment tasks, which are jointly optimized through a weighted loss function.

8. The method according to claim 1, characterized in that: The feature fusion of the graph convolutional network specifically includes: Modeling the complex relationship between RGB features, temperature features, and spectral features based on non-Euclidean space; The fusion weights of each modality feature are adjusted through an adaptive feature weighting mechanism to improve the discrimination of the fusion results.

9. The method according to claim 1, characterized in that: The output step further comprises: Compare the disease identification results with historical data to generate a disease trend analysis report; If the severity of the disease exceeds the preset threshold, an alarm is triggered and prevention and control suggestions are provided to the user.

10. A crop early disease recognition system based on time series feature fusion based on the method according to any one of claims 1 to 9, characterized in that: include: An image acquisition module, used to obtain RGB images, temperature images and spectral data of target leaves; Image processing module, used to perform temporal alignment of RGB images and temperature images and generate fusion features based on multimodal data; Deep neural network module, used to extract temporal and spatial features of multimodal data; Feature fusion module, used to achieve cross-modal feature fusion through graph convolutional networks; The classifier module is used to generate disease categories and severity levels based on the fused features; The output module is used to output disease identification results, transmission risk assessment and dynamic warning information.

Images