Pea powdery mildew detection method based on multi-modal image fusion

Abstract

The present application relates to image analysis technology, specifically to a method for detecting pea powdery mildew based on multi-modal image fusion, comprising a pretreatment module, a modal fusion module and a registration verification module; the present application fuses three kinds of modal information of the texture form of the RGB image, the spectral reflection of the multispectral image and the temperature distribution of the thermal infrared image, and constructs a more comprehensive and more discriminative disease characteristic expression system; the introduced attention fusion mechanism can adaptively assign appropriate weights to the features of different modalities, effectively utilize the complementary advantages of multi-source information, thereby realize more accurate recognition when the disease is in the incubation period or the symptoms are slight, reduce the false positive and false negative rates, realize the deep fusion and complement of multi-modal information, and significantly improve the accuracy and early identification ability of detection; based on the quantitative evaluation index of feature point error and gray level mutual information, a three-layer progressive troubleshooting process of "hardware-data-algorithm" is constructed to quickly locate the source of registration error.

Description

Technical Field

[0001] This invention relates to image analysis technology, specifically a method for detecting powdery mildew in peas based on multimodal image fusion. Background Technology

[0002] Powdery mildew in peas severely impacts pea yield and quality, and its early and accurate detection is crucial for precision plant protection. Current image-based disease detection methods primarily rely on single-modal data, which has significant limitations. Methods based on RGB images rely on visible disease features, but early symptoms are not obvious and are easily affected by background and lighting, leading to inaccurate segmentation and a high false detection rate.

[0003] Multispectral imaging-based methods can capture spectral information, but the equipment is expensive, data processing is complex, and spectral features are easily affected by non-disease factors such as moisture and nutrients, resulting in insufficient specificity and difficulty in accurate localization.

[0004] Existing multimodal fusion methods mostly employ simple concatenation strategies, failing to achieve adaptive feature weighting and thus limiting performance improvement.

[0005] In addition, accurate registration between multimodal images is a prerequisite for effective fusion. However, due to differences in sensors and asynchronous acquisition, registration misalignment often occurs, and there is a lack of systematic accuracy assessment and troubleshooting mechanisms, which affects the reliability of the system. To address the aforementioned technical problems, this application proposes a solution. Summary of the Invention

[0006] The purpose of this invention is to solve the problems raised in the background art and to propose a method for detecting powdery mildew in peas based on multimodal image fusion.

[0007] The objective of this invention can be achieved through the following technical solutions: A method for detecting powdery mildew in peas based on multimodal image fusion includes the following steps: G1: Simultaneously acquire RGB, multispectral, and thermal infrared images to form a multimodal dataset; perform HSV spatial transformation on the RGB images and use the U-Net network to generate leaf masks, extract pure leaf regions, and perform normalization processing; G2: RGB texture features are extracted using ResNet-50, multispectral features are extracted using a custom CNN, and weighted fusion is performed using an attention mechanism to obtain bimodal fusion features; thermal infrared images are converted into pseudo-color heatmaps, features are extracted using MobileNetV2 and aligned with bimodal features, and weighted fusion is performed using a trimodal attention mechanism, followed by refinement through residual connections to obtain fusion features; the fusion features are input into a Softmax classifier to output disease probabilities; the model is trained end-to-end using cross-entropy loss and Adam optimizer, and data augmentation and cosine annealing learning rate strategies are employed to evaluate model performance on the test set; G3: Multimodal image registration is achieved through feature point matching and homography matrix. The registration accuracy is evaluated by calculating the average pixel error and gray-level mutual information. If the accuracy is insufficient, a systematic investigation and optimization are carried out in three aspects: hardware synchronization and calibration, data format and ROI extraction, and feature matching algorithm.

[0008] In a preferred embodiment of the present invention, the detection method is implemented through a preprocessing module, a modal fusion module, and a registration and verification module; The preprocessing module performs HSV space conversion on the RGB image and initially separates the background based on hue and saturation thresholds; it uses the U-Net semantic segmentation network to generate a binary mask for the leaves and applies it to the RGB and multispectral images to remove the background; and it normalizes the obtained pure leaf region image to unify the pixel value range. The modality fusion module extracts RGB texture features and uses a custom CNN to extract multispectral features, which are then weighted and fused using an attention mechanism. Based on a multimodal dataset, the model is trained end-to-end using cross-entropy loss and the Adam optimizer. Furthermore, thermal infrared features are integrated, aligned with bimodal features, and adaptively fused using trimodal attention. Finally, the disease probability is output by a Softmax classifier. The registration verification module calculates the average pixel error and grayscale mutual information of feature points to quantitatively evaluate the registration accuracy. If the accuracy is insufficient, it performs a three-level investigation: hardware checks sensor synchronization and calibration; data verification verifies image format and ROI extraction; and algorithm optimization optimizes feature point matching and transformation matrix parameters.

[0009] In a preferred embodiment of the present invention, the preprocessing module specifically performs the following steps: M1: Perform color space conversion on the input RGB image from RGB to HSV space, and perform preliminary background separation based on preset hue (H) and saturation (S) thresholds; use the pre-trained U-Net semantic segmentation network to accurately extract the leaf region in the RGB image and generate a binary mask, where the leaf region is 1 and the background is 0. M2: Apply the binary mask to the RGB image and the multispectral image respectively, remove background pixels and retain only the pure leaf area; normalize the extracted pure leaf area image to unify the pixel value range to the [0,1] interval, so that it meets the input requirements of the subsequent convolutional neural network.

[0010] In a preferred embodiment of the present invention, the modal fusion module specifically performs the following steps: N1: Resize the RGB image to 224×224×3, input it into the pre-trained ResNet-50 network, remove its original classification layer, retain the convolution and pooling structures, extract texture features, and output a 2048-dimensional feature vector. The multispectral image (5 channels) was resized to 224×224×5 and input into a custom 5-layer convolutional neural network. Each layer included convolution, ReLU activation, and batch normalization operations, with max pooling layers inserted in between. Finally, a 2048-dimensional feature vector was output through a fully connected layer. ; N2: will and By stitching along the channel dimension, the stitching feature is obtained. The input is processed through a fully connected layer for dimensionality reduction, and then the weight vector is generated by the Sigmoid function. ,in and These represent the importance weights of RGB and multispectral features, respectively. N3: Weighted fusion of features from the two modalities based on their weights: ;Will Input a Softmax classifier and output the probability of leaf health. With disease probability .

[0011] In a preferred embodiment of the present invention, the modal fusion module further performs the following three-modal fusion steps: E1: Temperature conversion of thermal infrared images: , , , These are the camera's thermal constant, sensitivity coefficient, and emissivity correction factor, respectively. The sensor's radiation value; the 5% and 95% quantiles of the temperature data were extracted based on the RGB blade mask. , Normalization is performed: And apply the "viridis" color mapping to generate a three-channel pseudo-color heatmap; E2: Input the pseudo-color heatmap into the pre-trained MobileNetV2 network to extract 2048-dimensional thermal infrared features. ; through a fully connected layer Project to Same feature space: ;Will and The weights are concatenated and fed into a fully connected layer, then processed by Softmax to generate a three-modal weight vector. ; E3: Trimodal Weighted Fusion Refining and fusing features through fully connected layers with residual connections: ;Will Input a Softmax classifier and output the final disease probability distribution.

[0012] In a preferred embodiment of the present invention, the registration verification module performs a registration accuracy evaluation step: W1: Select the leaf edge inflection point and the center of the disease spot as feature points in the RGB image, and record their coordinates. Locate the corresponding feature points in the registered multispectral and thermal infrared images and record their coordinates. ; Calculate the average pixel error ,like If the number of pixels is 1, then the registration accuracy is considered sufficient; W2: Calculate gray-level mutual information between RGB and multispectral images. ,like If the grayscale correlation between the modes is strong, the registration accuracy meets the standard. If the results obtained by both methods indicate that the registration accuracy is sufficient, then preprocessing is performed. Otherwise, the core reasons for insufficient positioning and registration accuracy are investigated.

[0013] In a preferred embodiment of the present invention, if the registration accuracy is insufficient, the registration verification module performs a three-layer troubleshooting step: U1: Check the trigger synchronization, calibration status, and installation stability of the RGB, multispectral, and thermal infrared cameras. Ensure a windless and stable lighting environment for data acquisition, with a shooting distance of 5-10cm and vertical alignment. U2: Verify that the images of each modality are uniformly scaled to 224×224 pixels (bilinear interpolation), confirm that RGB is 3-channel 8-bit, multispectral is 5-channel reflectance data, and thermal infrared is a pseudo-color 3-channel image; check whether the leaf mask generated by U-Net is accurate and does not contain backgrounds such as soil and weeds. U3: Verify whether the number of SIFT feature points extracted is sufficient (>100 per image), optimize the nearest neighbor matching ratio threshold (≤0.75), confirm the use of homography matrix as the transformation model, and adjust the number of RANSAC iterations (≥1000) and the inlier tolerance error (≤1 pixel).

[0014] In a preferred embodiment of the present invention, the method further includes model training and evaluation steps: J1: Construct a multimodal image dataset, with each sample including RGB, multispectral, and thermal infrared images, labeled by experts as either healthy or diseased, and divided into training, validation, and test sets in a 7:2:1 ratio; use the cross-entropy loss function. Predicted probability ; J2: Employs the Adam optimizer, initial learning rate... The learning rate was dynamically adjusted using a cosine annealing strategy, with a batch size of 32, 100 training epochs, and data augmentation using random pruning and rotation. Accuracy was calculated on the test set. Accuracy Recall rate and The fractions are compared and analyzed with the single-modal method.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. By fusing three modal information—texture morphology of RGB images, spectral reflectance of multispectral images, and temperature distribution of thermal infrared images—a more comprehensive and discriminative disease feature expression system was constructed. The introduced attention fusion mechanism can adaptively assign appropriate weights to features of different modalities, effectively utilizing the complementary advantages of multi-source information. This enables more accurate identification during the incubation period of diseases or when symptoms are mild, reducing false alarms and false negatives. It achieves deep fusion and complementarity of multimodal information, significantly improving detection accuracy and early identification capabilities. 2. Based on the quantitative evaluation index of feature point error and gray-scale mutual information, a three-layer progressive investigation process of "hardware-data-algorithm" is innovatively constructed. This mechanism can quickly locate the source of registration error and provide clear optimization directions, which greatly improves the stability and maintainability of the detection system in complex real-world environments. 3. High-precision leaf segmentation is achieved using the U-Net network, effectively eliminating non-target regions; by standardizing thermal infrared data and mapping it to pseudo-color images, it can be efficiently processed by general convolutional networks; a structure combining a pre-trained lightweight network with a custom CNN is adopted to control model complexity while ensuring feature extraction capabilities. Attached Figure Description

[0016] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0017] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example: Please see Figure 1 As shown, the pea powdery mildew detection method based on multimodal image fusion includes a preprocessing module, a modality fusion module, and a registration and verification module. The preprocessing module performs HSV space conversion on the RGB image and initially separates the background based on hue and saturation thresholds; it uses the U-Net semantic segmentation network to generate a binary mask for the leaves and applies it to the RGB and multispectral images to remove the background; and it normalizes the obtained pure leaf region image to unify the pixel value range. The modality fusion module extracts RGB texture features and uses a custom CNN to extract multispectral features, which are then weighted and fused using an attention mechanism. Based on a multimodal dataset, the model is trained end-to-end using cross-entropy loss and the Adam optimizer. Furthermore, thermal infrared features are integrated, aligned with bimodal features, and adaptively fused using trimodal attention. Finally, the disease probability is output by a Softmax classifier. The registration verification module calculates the average pixel error and grayscale mutual information of feature points to quantitatively evaluate the registration accuracy. If the accuracy is insufficient, the process is carried out in three layers: hardware checks sensor synchronization and calibration; data verification verifies image format and ROI extraction; and algorithm optimization optimizes feature point matching and transformation matrix parameters. The input RGB image is converted to HSV color space, and the background is initially separated based on hue and saturation thresholds. The U-Net semantic segmentation network is used to accurately extract the leaf region and generate a binary mask. The mask is applied to the RGB image and the multispectral image to remove background interference and retain the leaf region. The extracted leaf image is normalized so that its pixel value distribution meets the model input requirements. The acquired RGB image is processed by a pre-trained ResNet-50 network. Texture features are extracted to obtain the features output by the RGB branch. ,in Given an RGB image as input, a multispectral image is obtained. Spectral features are extracted using a custom 5-layer convolutional network, and the features output by the multispectral branch are... ,in For the input multispectral image, This indicates that the input multispectral image is processed by a convolutional neural network. Input RGB image The image was resized to a fixed size of 224×224×3 and input into a ResNet-50 network. After removing the original classification layer, the network retained its convolutional and pooling layer structures. It extracted texture and morphological features from the image through multiple convolutional layers and residual connections, and finally output a 2048-dimensional feature vector from the last convolutional layer. The network consists of four convolutional layers, each followed by a ReLU activation function and batch normalization. Two max-pooling layers are inserted in between to progressively reduce the spatial dimension. The input image size is uniformly 224×224×N, where N is the number of spectral channels (the multispectral image used in this method is 5-channel data). After layer-by-layer convolution and pooling, a fully connected layer adjusts the output feature dimension to 2048, resulting in... ; Texture cues and spectral cues were extracted from RGB and multispectral images, respectively, and then stitched together along the channel dimension to obtain the total cues. After feature concatenation, it is processed through a fully connected layer. Perform dimensionality reduction, and through a function Fully connected layer The two scalars after dimensionality reduction are mapped to the [0,1] interval, ultimately yielding the weight vectors of texture cues and spectral cues. , This indicates the importance of RGB features in the current sample. The importance of multispectral features is represented; the fused features are obtained by weighting the corresponding terms. ; the fused features Input function The probability of whether the corresponding leaf in the fused image has powdery mildew is obtained. , To fuse the probability of healthy leaves corresponding to the images. This is used to determine the probability of leaf diseases corresponding to the fused images.

[0020] A multimodal image dataset of pea leaves containing both healthy and powdery mildew-infected leaves was constructed. Each sample set includes one RGB image and one multispectral image. Each image was manually labeled by agricultural experts, with the labels being binary classifications. The dataset was randomly divided into training, validation, and test sets in a 7:2:1 ratio. The cross-entropy loss function was used as the training objective function, and the Adam optimizer was used for parameter updates. The initial learning rate was set to [value missing]. The learning rate was dynamically adjusted using a cosine annealing strategy. Data augmentation methods such as random pruning and rotation were employed during training. The batch size was 32, and the model was trained for 100 epochs. The model achieved a high accuracy on the test set. Accuracy Recall rate and The scores were used as an evaluation metric, and comparative experiments were conducted with single-modal methods that used only RGB or multispectral methods. Cross-entropy loss function , The total number of samples in the batch. For the first The true label of each sample Indicates healthy leaves. Indicates diseased leaves. For the first The model prediction for a sample, i.e., the probability. Accuracy accuracy Recall rate F1 score ,in , , and These represent the number of samples that were determined to be true positive, true negative, false positive, and false negative, respectively.

[0021] A checkerboard calibration plate was used to simultaneously capture images in a multimodal acquisition system. Through feature point detection and homography matrix calculation, the thermal infrared images were spatially registered with RGB and multispectral images at the pixel level to ensure consistent positioning of the same blade region across all modes. Based on the thermal infrared camera calibration parameters, the original grayscale values ​​were converted into actual temperature values, and a temperature matrix was generated. , , , These are the camera's thermal constant, sensitivity coefficient, and emissivity correction factor, respectively. The sensor's radiation value; based on the blade region mask obtained from RGB image segmentation, the 5% and 95th quantiles of the temperature data within this region are extracted as normalization boundaries, and the normalized temperature value is... By mapping temperature values ​​to the [0,1] range, the influence of ambient temperature fluctuations is eliminated. A very small constant is used to prevent the denominator from being zero; the normalized temperature matrix is ​​applied with "viridis" color mapping to generate a three-channel pseudo-color heatmap; After obtaining the pseudo-color heatmap, a lightweight convolutional neural network, MobileNetV2, was used as the thermal infrared feature extractor. The input image size was uniformly adjusted to 224×224×3. The network was initialized based on weights pre-trained on the ImageNet dataset, and its top classification layer was removed, retaining the convolutional and inverse residual structures to extract temperature distribution and gradient features. After forward propagation, a 2048-dimensional feature vector was output from the last convolutional layer. This ensures that its dimensions are consistent with the RGB and multispectral branch outputs; Through the fully connected layer Projecting thermal infrared features onto... Same feature space: Achieve feature space alignment; and By stitching along the channel dimension, the stitching feature is obtained. ;Will Input fully connected layer Output three scalars, then... The function generates a normalized weight vector: Weighted fusion of the original features based on weights; Then through a fully connected layer with residual connections The fusion features were refined to obtain ; Refined characteristics Input includes The classification layer of the function outputs the probability distribution of leaf health and disease: .

[0022] After completing the registration operation, mark several feature points in the RGB image at the leaf edge inflection point and the center of the disease spot, and record the corresponding coordinates. The corresponding feature points were located sequentially in the registered multispectral and thermal infrared images, and their coordinates were recorded. Error calculated based on coordinates , The number of feature points, and The first The x-coordinate of each feature point on the corresponding image and The first The ordinates of each feature point on the corresponding image; if If the number of pixels is 1, then the registration accuracy is considered sufficient; if... If the number of pixels is less than a certain threshold, the registration accuracy is deemed insufficient. Obtain the grayscale value range of an RGB image, denoted as a variable. Obtain the grayscale value range of the multispectral image, denoted as variable. Then the correlation of gray-level distributions of the two modal images Joint histogram This indicates that "the grayscale value in RGB is..." Meanwhile, the gray level in the multispectral range is The percentage of pixels with the number of "" in the total number of pixels, edge histogram In an RGB image, the grayscale value is... Pixel ratio, edge histogram The gray level in the multispectral image is Pixel ratio; if The grayscale correlation between modes is strong, and the registration accuracy meets the standard. If both methods yield results indicating sufficient registration accuracy, then preprocessing is performed; otherwise, the core cause of insufficient positioning and registration accuracy is investigated. First, verify the sensor synchronization, i.e., check whether the trigger timestamps of the RGB, multispectral, and thermal infrared cameras are strictly synchronized. If there is a significant delay (such as thermal infrared lagging behind RGB shooting), it will cause blade displacement in a light wind environment, resulting in registration misalignment. This can be further verified by repeatedly acquiring images of the checkerboard calibration board and analyzing the registration error fluctuations of the same mode at different time points. Second, verify the sensor installation and calibration status, checking whether each sensor is firmly fixed to the same rigid bracket and whether there is any module tilting or loosening, which will cause deviations in the external parameters (spatial position relationship) between sensors. The internal and external parameter calibration process needs to be repeated. A checkerboard calibration board should be used to calibrate the lens distortion parameters of each sensor and determine the relative poses between multiple sensors. If the registration error is significantly reduced after calibration, it indicates that the original calibration has failed. Finally, verify the acquisition environment and operating procedures, confirming that the shooting was conducted in a windless and stable lighting environment, and checking that the shooting distance and angle meet the document requirements (e.g., vertical alignment, distance 5-10cm). If there is an angle tilt or excessive distance during acquisition, perspective distortion will occur in the image, increasing the difficulty of registration. These three steps complete the troubleshooting of the hardware acquisition process. First, verify the consistency of data format and resolution. Check whether RGB, multispectral, and thermal infrared images have been uniformly scaled to 224×224 pixels as required by the document, and whether the scaling method is bilinear interpolation. Incorrect stretching or compression of a modality image will distort the pixel spatial relationship. Simultaneously, confirm the format specifications of each modality data. For example, RGB should be 3-channel 8-bit grayscale, multispectral should be converted to 5-channel surface reflectance data, and thermal infrared should generate a pseudo-color three-channel image. Any abnormal modality format will lead to feature extraction distortion. Second, verify the effectiveness of the region of interest (ROI) extraction. The binary mask of the leaf generated according to the U-Net segmentation method described in the document must ensure that the mask only covers the pea leaf area and does not include soil, weeds, or other background. If the mask includes non-leaf areas due to improper NDVI threshold settings, the registration algorithm may mismatch background feature points, thus introducing systematic errors. This can be addressed by re-cropping the image using the corrected mask and performing registration. If the error is significantly reduced, it confirms a problem in the ROI extraction stage. These two steps complete the troubleshooting of the data preprocessing stage. To verify the effectiveness of feature point detection and matching, the SIFT algorithm is used for feature point extraction. The number of feature points extracted from each modality image is checked to ensure it is sufficient (>100 per image). If a modality (e.g., thermal infrared pseudo-color image) has too few feature points, it may be due to improper algorithm selection or insufficient image quality. Simultaneously, the feature point matching quality needs to be evaluated. Nearest neighbor matching is used with an appropriate nearest neighbor distance ratio (NNDR) threshold (≤0.75). If the mismatch rate exceeds 30% or the proportion of outliers removed by RANSAC is too high, the matching strategy needs optimization. Secondly... Verify the rationality of the transformation matrix estimation and confirm whether the transformation model used is a homography matrix applicable to planar images. Misuse of affine transformation may lead to local registration deviations. It is also necessary to check the key parameters of the RANSAC algorithm, such as the number of iterations (≥1000 times) and the interior point tolerance error (≤1 pixel). Improper parameter settings will directly affect the stability and accuracy of matrix estimation. By adjusting the above algorithm parameters and re-performing the registration, if the error decreases significantly, it can be determined that the registration algorithm is the core reason for insufficient accuracy. The above two steps complete the investigation of the registration algorithm.

[0023] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for detecting powdery mildew in peas based on multimodal image fusion, characterized in that, This is implemented through a preprocessing module, a modality fusion module, and a registration verification module, including the following steps: G1: The preprocessing module simultaneously acquires RGB, multispectral, and thermal infrared images to form a multimodal dataset; it performs HSV spatial transformation on the RGB images, initially separates the background based on hue and saturation thresholds, then uses the U-Net network to generate leaf masks, extracts pure leaf regions from the RGB and multispectral images based on the leaf masks, and completes normalization processing. G2: The modality fusion module extracts RGB texture features through ResNet-50, extracts multispectral features through a custom CNN, and uses an attention mechanism for weighted fusion to obtain dual-modality fusion features; it converts thermal infrared images into pseudo-color heatmaps, extracts thermal infrared features through MobileNetV2 and aligns them with the dual-modality features, then performs weighted fusion through a trimodal attention mechanism, and obtains the final fusion features after refinement by residual connections; the fusion features are input into a Softmax classifier to output disease probabilities; end-to-end model training is performed using cross-entropy loss and Adam optimizer, combined with data augmentation and cosine annealing learning rate strategies, and model performance is evaluated on the test set; G3: The registration verification module achieves multimodal image registration through feature point matching and homography matrix, and calculates the average pixel error and gray-level mutual information to evaluate the registration accuracy. If the registration accuracy is not up to standard, the registration process is systematically checked and optimized in three stages: hardware synchronization and calibration, data format and ROI extraction, and feature matching algorithm. The modality fusion module specifically performs the following steps: N1: Resize the RGB image to 224×224×3, input it into the pre-trained ResNet-50 network, remove its original classification layer, retain the convolution and pooling structures, extract texture features, and output a 2048-dimensional feature vector. The multispectral image was resized to 224×224×5 and input into a custom 5-layer convolutional neural network. Each layer included convolution, ReLU activation, and batch normalization operations, with max pooling layers inserted in between. Finally, a 2048-dimensional feature vector was output through a fully connected layer. ; N2: will and By stitching along the channel dimension, the stitching feature is obtained. The input is processed through a fully connected layer for dimensionality reduction, and then the weight vector is generated by the Sigmoid function. ,in and These represent the importance weights of RGB and multispectral features, respectively. N3: Weighted fusion of features from the two modalities based on their weights: ;Will Input a Softmax classifier and output the probability of leaf health. With disease probability ; The modality fusion module further performs the following three-modality fusion steps: E1: Temperature conversion of thermal infrared images: , , , These are the camera's thermal constant, sensitivity coefficient, and emissivity correction factor, respectively. The sensor's radiation value; the 5% and 95% quantiles of the temperature data were extracted based on the RGB blade mask. , Normalization is performed: And apply "viridis" color mapping to generate a three-channel pseudo-color heatmap; E2: Input the pseudo-color heatmap into the pre-trained MobileNetV2 network to extract 2048-dimensional thermal infrared features. ; through a fully connected layer Project to Same feature space: ;Will and The weights are concatenated and fed into a fully connected layer, then processed by Softmax to generate a three-modal weight vector. ; E3: Trimodal Weighted Fusion Refining and fusing features through fully connected layers with residual connections: ;Will Input a Softmax classifier and output the final disease probability distribution.

2. The method for detecting powdery mildew in peas based on multimodal image fusion according to claim 1, characterized in that, The preprocessing module specifically performs the following steps: M1: Performs color space conversion on the input RGB image, converting it from RGB to HSV space, and performs preliminary background separation based on preset hue (H) and saturation (S) thresholds; The leaf region in the RGB image is accurately extracted using a pre-trained U-Net semantic segmentation network, and a binary mask is generated, where the leaf region is 1 and the background is 0. M2: Apply the binary mask to the RGB image and the multispectral image respectively, remove background pixels and retain only the pure leaf area; normalize the extracted pure leaf area image to unify the pixel value range to the [0,1] interval, so that it meets the input requirements of the subsequent convolutional neural network.

3. The method for detecting powdery mildew in peas based on multimodal image fusion according to claim 1, characterized in that, The registration verification module performs the registration accuracy evaluation step: W1: Select the leaf edge inflection point and the center of the disease spot as feature points in the RGB image, and record their coordinates. Locate the corresponding feature points in the registered multispectral and thermal infrared images and record their coordinates. ; Calculate the average pixel error ,like If the number of pixels is 1, then the registration accuracy is considered sufficient. W2: Calculate gray-level mutual information between RGB and multispectral images. ,like If the grayscale correlation between the modes is strong, the registration accuracy meets the standard. If the results obtained by both methods indicate that the registration accuracy is sufficient, then preprocessing is performed. Otherwise, the core reasons for insufficient positioning and registration accuracy are investigated.

4. The method for detecting powdery mildew in peas based on multimodal image fusion according to claim 3, characterized in that, If the registration accuracy is insufficient, the registration verification module performs a three-layer troubleshooting process: U1: Check the trigger synchronization, calibration status, and installation stability of the RGB, multispectral, and thermal infrared cameras. Ensure a windless and stable lighting environment for data acquisition, with a shooting distance of 5-10cm and vertical alignment. U2: Verify that the images of each modality are uniformly scaled to 224×224 pixels, confirm that RGB is 3-channel 8-bit, multispectral is 5-channel reflectance data, and thermal infrared is a pseudo-color 3-channel image; check whether the blade mask generated by U-Net is accurate. U3: Verify whether the number of SIFT feature points extracted is sufficient, optimize the nearest neighbor matching ratio threshold, confirm the use of homography matrix as the transformation model, and adjust the number of RANSAC iterations and the interior point tolerance error.

5. The method for detecting powdery mildew in peas based on multimodal image fusion according to claim 1, characterized in that, The method also includes model training and evaluation steps: J1: Construct a multimodal image dataset, with each sample including RGB, multispectral, and thermal infrared images, labeled by experts as either healthy or diseased, and divided into training, validation, and test sets in a 7:2:1 ratio; use the cross-entropy loss function. Predicted probability ; J2: Employs the Adam optimizer, initial learning rate... The learning rate was dynamically adjusted using a cosine annealing strategy, with a batch size of 32, 100 training epochs, and data augmentation using random pruning and rotation. Accuracy was calculated on the test set. Accuracy Recall rate and The fractions are compared and analyzed with the single-modal method.

Images