A broccoli early blight early diagnosis system and method based on multi-modal machine learning fusion

Abstract

The application relates to the technical field of intelligent agriculture and plant protection, and particularly discloses a broccoli black spot early diagnosis system and method based on multi-modal machine learning fusion, which comprises a portable collection platform, a mobile experiment terminal and a server. The portable collection platform is integrated with a hyperspectral imaging camera, a chlorophyll fluorescence imager, a dark-adapted leaf clamp and a camera for sample barcode recognition, and is provided with an adjustable height support and a controllable light source to ensure the consistency of data collection conditions. The mobile experiment terminal is internally provided with a physicochemical index rapid detection module, including a portable PCR instrument for on-site nucleic acid rapid extraction and amplification and a miniature ultraviolet spectrophotometer for rapid determination of pigment content. The broccoli black spot early diagnosis system and method based on multi-modal machine learning fusion are highly integrated, portable and automated, are suitable for a field environment and provide an efficient disease diagnosis tool for non-expert users.

Description

Technical Field

[0001] This invention relates to the fields of smart agriculture and plant protection technology, specifically to an early diagnostic system and method for broccoli black spot disease based on multimodal machine learning fusion. Background Technology

[0002] Broccoli black spot disease, primarily caused by *Alternaria brassicicola*, is one of the major diseases affecting broccoli production. Traditional diagnostic methods rely on manual observation of lesion morphology in the field. This method is highly subjective, inefficient, and can only identify the disease after symptoms appear (mid-to-late stage), at which point control efforts are ineffective, resulting in economic losses. In recent years, non-destructive testing techniques such as hyperspectral imaging and chlorophyll fluorescence imaging have been applied in plant disease detection. Hyperspectral imaging can capture subtle changes in spectral reflectance characteristics caused by physiological changes in plant tissues; chlorophyll fluorescence imaging can sensitively reflect the functional state of the photosynthetic system. However, existing technologies are mostly based on single data sources, which have the following limitations: 1. Insufficient specificity: Single optical signals are easily interfered with by environmental factors (light, water) and other stresses (nutrient stress, other diseases), leading to false alarms. 2. Limited early diagnostic capability: During the disease incubation period, the information changes of a single modality are weak and difficult to detect reliably. 3. Weak mechanistic correlation: It is difficult to directly link optical signals with intrinsic physiological and biochemical changes (such as toxin accumulation and defense substance synthesis), which limits the interpretability and reliability of the model.

[0003] Therefore, developing a system and method that can integrate multi-source information, reveal disease mechanisms, and achieve high-precision diagnosis in the early stages is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides the following technical solution: an early diagnostic system and method for broccoli black spot disease based on multimodal machine learning fusion, comprising:

[0005] A portable data acquisition platform, which integrates a hyperspectral imaging camera, a chlorophyll fluorescence imager, a dark-adaptive leaf clip, and a camera for sample barcode recognition. The platform has an adjustable-height stand and a controllable light source to ensure consistent data acquisition conditions.

[0006] A mobile experimental terminal, which has a built-in rapid detection module for physicochemical indicators, including a portable PCR instrument for rapid on-site nucleic acid extraction and amplification and a miniature ultraviolet spectrophotometer for rapid determination of pigment content;

[0007] A computing and control unit, wherein the computing and control unit is an industrial-grade ruggedized tablet computer with a built-in software system;

[0008] The power module is a high-capacity lithium battery pack that provides uninterrupted power to the entire system, supporting field work for ≥4 hours.

[0009] Preferably, the calculation and control unit is used for:

[0010] Control all imaging and sensing devices to work together;

[0011] Run data preprocessing and feature extraction algorithms;

[0012] Load and run the trained MK-SVM and DCNN fusion diagnostic model;

[0013] Generate visual diagnostic results.

[0014] An early diagnostic method for broccoli black spot disease based on multimodal machine learning fusion includes the following steps:

[0015] S1. Multimodal data acquisition and preprocessing, including acquiring hyperspectral images, acquiring chlorophyll fluorescence images, detecting physicochemical indicators, and detecting pigment content and metabolite content;

[0016] S2. Feature extraction and dataset construction, including extracting hyperspectral features, fluorescence features and physicochemical features. After standardizing all extracted features, they are aligned by sample ID to construct a multimodal feature dataset.

[0017] S3, Multimodal data fusion and diagnostic model construction, adopts a two-level model strategy for construction;

[0018] Model 1: Multi-kernel support vector machine fusion model, which adopts a multi-kernel learning framework based on gradient descent weight optimization, using Gaussian radial basis function (RBF) and polynomial kernel function (Poly) as the basis kernels;

[0019] Model 2; Feature-level fusion model based on deep convolutional neural networks;

[0020] S4. Model Validation and Diagnostic Application: The coefficient of determination (R²) and root mean square error (RMSE) are used to evaluate the accuracy of the regression model, and the accuracy of the classification model is evaluated using accuracy, precision, recall, and F1 score. The trained model is deployed in the diagnostic system to quickly diagnose new unknown samples and output the disease status, severity, and reliability.

[0021] Preferably, the general expression for the kernel function in Model 1 of step S3 is:

[0022] ;

[0023] Where Kn(xi,xj) is the nth basic kernel function, n represents the number of kernel functions, dn is the weight coefficient of each kernel function, and stdn≥0 indicates that the weight of each basic kernel cannot be negative. This indicates that the sum of all weights is 1. This model is used for preliminary qualitative diagnosis (health / disease) and provides feature weight contribution analysis.

[0024] Preferably, the model input and construction process in step S3 of Model 1 is as follows:

[0025] Input feature vector construction:

[0026] For a given sample i, the feature vectors extracted from different modalities are concatenated to form a unified, high-dimensional multimodal feature vector. Assuming h hyperspectral features, f fluorescence features, and p physicochemical features are extracted from the i-th sample, then all features of that sample are concatenated into a unified multimodal feature vector. This process can be expressed by the following formula:

[0027] ;

[0028] in, Let i be the unified multimodal feature vector of sample i. Let i be the hyperspectral feature vector of sample i. Let i be the chlorophyll fluorescence feature vector. Let i be the physicochemical feature vector. For vector concatenation operations, the feature vectors of all samples are combined to form the input matrix X and the corresponding label vector y. The result is: healthy = 0, sick = 1.

[0029] It has the following beneficial effects:

[0030] 1. Very early diagnosis: By fusing fluorescence signals that are extremely sensitive to stress with spectral signals that reflect chemical composition, diseases can be detected in the incubation period (precursor stage), and the diagnosis time is 5-7 days earlier than traditional methods.

[0031] 2. High precision and high robustness: Multimodal data complementarity effectively overcomes environmental interference. The dual model of MK-SVM and DCNN ensures the reliability of diagnostic results, with an overall accuracy of over 95%.

[0032] 3. Mechanism-driven: By linking optical signals with actual pathogen load and changes in key metabolites, the diagnostic model has a physiological and biochemical basis, which not only answers "whether the disease occurs" but also provides a preliminary explanation of "how the disease occurs".

[0033] 4. High practicality: The designed system is highly integrated, portable, and automated, suitable for field environments, and provides non-expert users with an efficient disease diagnosis tool. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall structure of the diagnostic system of the present invention;

[0035] Figure 2 This is a flowchart illustrating the overall process of the diagnostic method of the present invention.

[0036] Figure 3 This is a schematic diagram illustrating the principle of the multi-core support vector machine fusion model of the present invention;

[0037] Figure 4 This is a structural diagram of the feature-level fusion model based on deep convolutional neural networks in this invention. Detailed Implementation

[0038] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Please refer to Figures 1-4 This invention provides a technical solution: an early diagnostic system for broccoli black spot disease based on multimodal machine learning fusion, comprising:

[0040] Portable data acquisition platform: It integrates a hyperspectral imaging camera, a chlorophyll fluorescence imager, a dark-adaptive leaf clip, and a camera for sample barcode recognition. The platform has an adjustable height stand and a controllable light source to ensure consistent data acquisition conditions.

[0041] Mobile experimental terminal: Built-in rapid detection module for physicochemical indicators, including:

[0042] Portable PCR instrument for rapid on-site nucleic acid extraction and amplification;

[0043] Miniature ultraviolet spectrophotometer for rapid determination of pigment content;

[0044] Computing and Control Unit: An industrial-grade ruggedized tablet PC with built-in software system for:

[0045] Control all imaging and sensing devices to work together;

[0046] Run data preprocessing and feature extraction algorithms;

[0047] Load and run the trained MK-SVM and DCNN fusion diagnostic model;

[0048] Visualize diagnostic results (e.g., generate disease distribution maps and generate diagnostic reports);

[0049] Power module: High-capacity lithium battery pack provides uninterrupted power supply to the entire system, supporting field work for ≥4 hours;

[0050] An early diagnostic method for broccoli black spot disease based on multimodal data fusion includes the following steps:

[0051] S1. Multimodal data acquisition and preprocessing;

[0052] Hyperspectral image acquisition: In a darkroom environment, a hyperspectral imaging system was used to acquire the spatial-spectral information of the broccoli canopy. The main parameters were: spectral range: 400-1000nm; spectral resolution: ≤3nm; spatial resolution: ≤0.5mm / pixel. The raw hyperspectral data was preprocessed by black and white correction, spectral cropping, and noise filtering to obtain the corrected spectral cube.

[0053] Chlorophyll fluorescence image acquisition: A pulse-modulated chlorophyll fluorescence imaging system was used. After acquiring hyperspectral images, fluorescence images of the canopy of the same plant after dark adaptation (at least 30 minutes) were acquired. The main parameters were: fluorescence parameters including maximum quantum yield (Fv / Fm), actual quantum yield (Y(II)), non-photochemical quenching (NPQ), etc. The average value of fluorescence parameters and spatial distribution heterogeneity of the entire leaf were extracted as features.

[0054] Physicochemical index testing: Leaf samples from the same canopy were collected and subjected to destructive testing;

[0055] Pathogen detection: PCR technology was used to amplify the pathogen using Alternaria brassicicola specific primers to confirm whether the pathogen was infected and the pathogen load (Ct value).

[0056] Pigment content: The contents of chlorophyll a, chlorophyll b and carotenoids were determined by ultraviolet spectrophotometer using the acetone-ethanol extraction method.

[0057] Metabolite content: The contents of AB toxin, soluble sugar, starch, lignin and glucosinolate were determined by gas chromatography-mass spectrometry (GC-MS).

[0058] S2. Feature extraction and dataset construction;

[0059] Hyperspectral feature extraction: The average spectrum of the region of interest (ROI) is extracted from the preprocessed hyperspectral image. Machine learning algorithms (such as RandomFrog and Continuous Projection Algorithm-SPA) are used to screen the feature wavelengths most relevant to the early stage of black spot disease infection (exemplary feature wavelength range: 520-535nm, 680-690nm, 750-770nm, etc.).

[0060] Fluorescence feature extraction: extract the mean, standard deviation and spatial heterogeneity index of fluorescence kinetic parameters such as Fv / Fm, Y(II), and NPQ;

[0061] Physicochemical characteristics: PPCRCt value, pigment content, and metabolite content are used as numerical characteristics;

[0062] After standardizing (Z-score) all extracted features (spectral features, fluorescence features, physicochemical features), they are aligned by sample ID to construct a multimodal feature dataset.

[0063] S3, Multimodal data fusion and diagnostic model construction;

[0064] This step employs a two-level model strategy, balancing interpretability and high accuracy;

[0065] Model 1 (Interpretable Model): Multi-kernel Support Vector Machine (MK-SVM) fusion model;

[0066] A multi-kernel learning framework based on gradient descent weight optimization is adopted, using the Gaussian radial basis function (RBF) and the polynomial kernel function (Poly) as the basis kernels;

[0067] The general expression for a kernel function is:

[0068] ;

[0069] Where Kn(xi,xj) is the nth basic kernel function, n represents the number of kernel functions, dn is the weight coefficient of each kernel function, and stdn≥0 indicates that the weight of each basic kernel cannot be negative. This indicates that the sum of all weights is 1. This model is used for preliminary qualitative diagnosis (health / disease) and provides feature weight contribution analysis.

[0070] Model input and construction process:

[0071] Input feature vector construction:

[0072] For a given sample i, the feature vectors extracted from different modalities are concatenated to form a unified, high-dimensional multimodal feature vector. Assuming h hyperspectral features, f fluorescence features, and p physicochemical features are extracted from the i-th sample, then all features of that sample are concatenated into a unified multimodal feature vector. This process can be expressed by the following formula:

[0073] ;

[0074] in, Let i be the unified multimodal feature vector of sample i. Let i be the hyperspectral feature vector of sample i. Let i be the chlorophyll fluorescence feature vector. Let i be the physicochemical feature vector. For vector concatenation operations, the feature vectors of all samples are combined to form the input matrix X and the corresponding label vector y (the result is: healthy=0, sick=1).

[0075] Detailed build process:

[0076] Data standardization: Z-score standardization is performed on each dimension of the input matrix X to eliminate the influence of dimensions;

[0077] Kernel function selection and design: Select two kernel functions with complementary properties:

[0078] To enhance local capture and nonlinear modes, thereby improving the fitting ability of complex boundaries in the feature space, a Gaussian radial basis function (RBF) kernel is selected, as shown in the following formula:

[0079] ;

[0080] in, It is a sample and In the similarity metric using the Gaussian kernel function, γ is the kernel coefficient, a hyperparameter greater than 0 that determines the range of influence of a single sample and the curvature of the decision boundary. A larger γ value results in a smaller range of influence, a more curvature decision boundary, and a higher likelihood of overfitting; conversely, a smaller γ value results in a larger range of influence, a smoother decision boundary, and a higher likelihood of underfitting. It is the square of the Euclidean distance between two sample vectors, used to measure the spatial distance between two sample points;

[0081] To capture the interactions between global features and improve the sensitivity to the intrinsic structure of the data, a polynomial kernel is chosen, the mathematical formula of which is:

[0082] ;

[0083] in, It is a sample and In the similarity metric under the polynomial kernel function, γ is the scaling factor, which affects the inner product result; r is a constant term, a hyperparameter used to adjust the constant offset; d is the degree of the polynomial, which must be a positive integer and determines the dimension of the feature space after mapping and the complexity of the model. The higher the degree, the more complex the model.

[0084] Multi-core learning and weight optimization:

[0085] Combined kernel functions, the formula is as follows:

[0086] ;

[0087] Where d1, d2: kernel weights, representing the relative importance of each basis kernel function in the final decision, d1, d2 ≥ 0, and d1 + d2 = 1;

[0088] Innovative optimization: A strategy combining grid search and cross-validation is used to optimize the weights d1 and d2 as well as the internal parameters of each kernel function (such as γ for RBF and d for Poly). The objective function is to maximize the average accuracy of cross-validation. This not only automatically determines the optimal kernel weights but also optimizes the kernel parameters simultaneously, significantly improving model performance.

[0089] Model training: Train the SVM classifier using the optimized combined kernel function Kcombined to find the optimal classification hyperplane;

[0090] Basis for preliminary qualitative diagnosis (healthy / sick):

[0091] Decision function: The trained MK-SVM model for a new sample Its decision function is:

[0092] ;

[0093] in, It is for new samples The output value of the decision function. It is the number of support vectors. It is the Lagrange multiplier corresponding to the i-th support vector. , This represents the true class label corresponding to the i-th support vector. Let b be the kernel function, and b be the bias.

[0094] Diagnostic basis:

[0095] like If so, it is determined to be an illness;

[0096] like If so, it is considered healthy;

[0097] Innovative Interpretation: The interpretability of MK-SVM is reflected in feature weights and kernel weights. By analyzing the contribution of each feature in the decision function of the final model (e.g., through importance scoring based on permutation), the most critical spectral bands, fluorescence parameters, or physicochemical indicators for early diagnosis can be identified (such as "absorption valley at 680nm", "decline in Fv / Fm", "slight increase in AB toxin content"). This provides direct data support for early disease mechanism research and realizes the integration of "diagnosis" and "mechanism analysis", which cannot be achieved by a single model.

[0098] Model 2 (High-Precision Diagnostic Model): A feature-level fusion model based on deep convolutional neural networks (DCNN);

[0099] Build a customized DCNN network structure, such as Figure 4 As shown, its innovation lies in the mid-level feature fusion layer;

[0100] Input layer: Receives hyperspectral feature vectors and fluorescence feature vectors respectively. The input layer contains two branches: Branch 1 (Hyperspectral): Inputs a vector of shape (p,), which is processed through two layers of 1D convolution (Conv1D, kernel_size=3), batch normalization (BatchNorm), and ReLU activation to extract high-order spectral features. Branch 2 (Fluorescence): Inputs a vector of shape (q,). It undergoes similar processing (which can be a single Conv1D layer) to extract high-order fluorescence features.

[0101] Feature Fusion Layer (Core Innovation): This layer flattens the high-order feature tensors (no longer the original features) output from the two branches. These flattened tensors are then concatenated with the standardized original physicochemical feature vectors. The innovation lies in the fact that this fusion is not a simple early-stage fusion (concatenating the original features), but rather a fusion performed in the middle layer of the network. The two CNN branches first learn to extract the most effective abstract representation within their respective modalities (e.g., "spectral shape sensitive to disease" or "specific fluorescence response pattern"), and then combine this high-level abstraction with precise physicochemical indicators. This greatly enhances the model's ability to capture complex nonlinear relationships.

[0102] Classification / Regression Layer: The fused features are passed through a fully connected layer and a Dropout layer, and finally output the severity level of the disease (e.g., healthy, latent period, early symptoms, late symptoms) through the Softmax function, or output the predicted value of physicochemical indicators (e.g., AB toxin content) through a linear function.

[0103] Model training parameters: Optimizer: Adam; Learning rate: 0.001; Batch size: 32; Loss function: Cross-entropy loss function for classification tasks, mean squared error loss function for regression tasks;

[0104] Define the severity level of the disease and the Softmax output;

[0105] Disease severity level (classification task):

[0106] Level 0: Healthy - No symptoms, normal physical and chemical indicators;

[0107] Level 1: Latent - Asymptomatic, but PCR test is positive (Ct value > preset threshold, such as Ct < 28), and / or key metabolites (AB toxin) begin to accumulate. The Ct value refers to the number of amplification cycles required for the fluorescence signal to reach the set threshold.

[0108] Level 2: Early-Symptomatic - tiny, invisible (<1mm) or visible chlorotic spots appear;

[0109] Level 3: Late-Symptomatic - Characteristic dark brown lesions appear, possibly accompanied by a yellow halo;

[0110] Softmax output and threshold:

[0111] a. The Softmax function converts the network's final output into a probability distribution vector with a sum of 1, for example, [0.02, 0.88, 0.08, 0.02].

[0112] b. Diagnosis result: Take the category with the highest probability as the prediction level (e.g., Level 1: incubation period in the example above).

[0113] c. Confidence level: This maximum probability value (such as 0.88 in the example above) is the confidence level of this prediction, which measures the degree of certainty that the model is in predicting this prediction;

[0114] Confidence level calculation and threshold standards;

[0115] For classification models (such as DCNN), the confidence score is the probability that a sample belongs to the class to be classified, which is directly output by the Softmax function;

[0116] Define threshold criteria (for decision-making):

[0117] a. High confidence interval (Confidence ≥ 0.85): The prediction results are highly reliable and can be directly adopted and implemented with corresponding prevention and control recommendations;

[0118] b. Medium confidence interval (0.70≤Confidence<0.85): The prediction result is relatively reliable, but the system will mark it as "uncertain". It is recommended that users verify the results in conjunction with the actual field conditions, or conduct a second test on the plant after 24 hours.

[0119] c. Low confidence interval (Confidence < 0.70): The model cannot make a reliable judgment. The system will prompt "Diagnosis failed" and suggest checking whether the sensor is blocked, whether the plant is subjected to other abnormal stresses (such as pesticide damage, fertilizer damage), or uploading the data to the cloud to apply for expert manual diagnosis;

[0120] Model validation and diagnostic applications;

[0121] The accuracy of the regression model is evaluated using the coefficient of determination (R²) and root mean square error (RMSE), while the accuracy of the classification model is evaluated using precision, accuracy, recall, and F1 score. The trained model is then deployed in the diagnostic system to quickly diagnose new unknown samples and output the disease status, severity, and reliability.

[0122] Controlled experiment:

[0123] 1. Control group setup:

[0124] A control group was set up, and the experiment included:

[0125] Control Group: Healthy plants that were sprayed with sterile water or did not undergo any treatment.

[0126] Inoculated Group: Plants inoculated by spraying with Alternaria brassicicola spore suspension.

[0127] 2. Multiple sets of experimental data tables:

[0128] Preliminary experiments verify the model's performance.

[0129] Table 1: Qualitative diagnostic performance of the MK-SVM model at different infection stages

[0130] ;

[0131] dpi: days post-inoculation (number of days after vaccination)

[0132] Table 2: Diagnostic performance of DCNN model for disease severity level (confusion matrix)

[0133] ;

[0134] Note 1: Level 0: Healthy; Level 1: Incubation period; Level 2: Early stage of symptoms; Level 3: Late stage of symptoms;

[0135] Note 2: *Overall accuracy: 115 / 120 = 95.8%*;

[0136] Table 3: Performance of DCNN model regression prediction of key physicochemical indicators (Table shows: AB toxin content)

[0137] ;

[0138] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art and related fields based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described and explained in the present invention, unless otherwise specified or limited, shall be implemented according to conventional means in the art.

Claims

1. A method for early diagnosis of black spot disease of broccoli based on multi-modal machine learning fusion, characterized in that, Comprising the following steps: S1, multi-modal data acquisition and preprocessing, including acquisition of hyperspectral images, acquisition of chlorophyll fluorescence images, detection of physicochemical indicators, detection of pigment content and metabolite content; S2, feature extraction and dataset construction, including extraction of hyperspectral features, extraction of fluorescence features and physicochemical features, after standardization processing of all extracted features, alignment according to sample ID, construction of multi-modal feature dataset; S3, multi-modal data fusion and diagnosis model construction, constructed by adopting two-level model strategy; Model one: multi-kernel support vector machine fusion model, adopting multi-kernel learning framework based on gradient descent weight optimization, taking Gaussian radial basis kernel function and polynomial kernel function as base kernel, model input and construction process as follows: Input feature vector construction: For a given sample i, the feature vectors extracted from different modalities are spliced to form a unified, high-dimensional multi-modal feature vector, assuming that h hyperspectral features, f fluorescence features and p physicochemical features are extracted from the i th sample, then all features of the sample are spliced into a unified multi-modal feature vector, which is represented by formula: ； wherein, is a unified multi-modal feature vector of sample i, is a hyperspectral feature sub-vector of sample i, is a chlorophyll fluorescence feature sub-vector of sample i, is a physicochemical feature sub-vector of sample i, is a vector concatenation operation, which groups the feature vectors of all samples into an input matrix X, and the corresponding label vector y; Model two: feature-level fusion model based on deep convolutional neural network, construction process as follows: Adopting medium-order feature fusion layer; Input layer: receiving hyperspectral feature vector and fluorescence feature vector respectively, input layer containing 2 branches, branch one: input shape (p,) vector, after 2 layers of 1D convolution, batch normalization and ReLU activation, high-order spectral features are extracted, branch two: input shape (q,) vector, after similar structure processing, high-order fluorescence features are extracted; Feature fusion layer: the high-order feature tensors output by the two branches are flattened, and the two flattened tensors are spliced with the original physicochemical feature vector after standardization processing; S4, model verification and diagnosis application, using coefficient of determination and root mean square error to evaluate the precision of regression model, using accuracy, precision, recall and F1 score to evaluate the precision of classification model, deploying the trained model in the diagnosis system to quickly diagnose new unknown samples, and outputting disease state, severity and reliability.

2. The method according to claim 1, wherein the method is characterized by, The general expression of kernel function in model one of step S3 is: ； where Kn(xi, xj) is the nth basis kernel function, n represents the number of kernel functions, dn is the weight coefficient of each kernel function, and s.t. dn≥0 indicates that the weight of each base kernel cannot be negative, which indicates that the sum of all weights is 1. This model is used for preliminary qualitative diagnosis and provides feature weight contribution degree analysis.

Images