An apple anthracnose leaf blight identification five-dimensional data fusion method based on a patrol robot

Abstract

The application discloses a five-dimensional data fusion method for identifying apple anthracnose leaf blight based on a patrol robot, relates to the technical field of anthracnose leaf blight identification, and solves the problems of limited sensing dimension, great light interference and inaccurate spatial positioning in the prior art. The method comprises the following steps: S1, acquiring the curvature feature of the surface of an apple leaf, collecting an RGB-D image of the leaf, constructing a three-dimensional topological structure of the leaf, and positioning the spatial distribution of a disease spot; S2, acquiring the spectral reflection information of the apple leaf, quantifying the biochemical components of chlorophyll a and carotenoids, and constructing a vegetation index feature set sensitive to diseases; S3, collecting the temperature information of the apple tree crown layer, constructing a transpiration anomaly detection model, and realizing multi-modal fusion of thermal infrared and point cloud data; and S4, integrating the spatial distribution, spectral reflection and thermal radiation information, constructing a five-dimensional phenotype feature matrix model, and comprehensively analyzing the apple phenotype features. The application can significantly improve the recognition accuracy under the conditions of leaf shielding, uneven light, leaf posture change and environmental interference.

Description

Technical Field

[0001] This invention relates to the field of apple anthracnose leaf blight identification technology, specifically a five-dimensional data fusion method for apple anthracnose leaf blight identification based on an inspection robot. Background Technology

[0002] Anthracnose leaf blight is a fungal disease that has become widespread in major apple-producing areas of my country in recent years. Its typical symptoms include the initial appearance of indistinct brown spots on leaves, which rapidly expand to cause necrosis of the entire leaf within 1-2 days, followed by scorching and drop. When fruit is affected, brown sunken spots form, with reddish edges on the peel and spongy brown discoloration of the flesh inside. The anthracnose fungus typically spreads outwards from a central infected plant, overwintering mainly on diseased branches and mummified fruit. It begins to multiply and infect in spring when temperatures rise above 15°C, often causing severe leaf drop and fruit rot, significantly impacting yield and quality.

[0003] Traditional methods for monitoring and diagnosing apple orchard diseases mainly rely on manual inspections and experience-based judgment. This increases the workload of orchard managers, reduces overall work efficiency, and is prone to missed diagnoses and misdiagnoses, hindering refined orchard management. With the continuous maturation of technologies such as autonomous navigation, visual recognition, and multimodal data acquisition in industrial inspection robots, orchard inspections, growth data collection, and pest and disease identification can be achieved. These robots adapt to inspection and monitoring work in complex environments, enabling fruit growers to promptly identify and address issues with fruit trees, adjust orchard management plans, and improve orchard production efficiency and fruit quality.

[0004] While significant progress has been made in the research of intelligent inspection technology for leaf diseases in orchards both domestically and internationally, many challenges remain in dynamic inspections in real orchard environments. Traditional image recognition methods suffer from low accuracy and poor deployment adaptability under conditions of natural light interference, missing geometric information, and insufficient physiological indicators, making it difficult to support accurate identification and dynamic early warning of typical environmentally sensitive diseases such as anthracnose leaf blight.

[0005] Therefore, it is urgent to construct a highly adaptable five-dimensional data fusion model that integrates image and environmental sensing data, and integrate it into a field inspection robot platform to address the dynamic identification tasks of typical diseases such as apple anthracnose leaf blight, thereby improving the system's generalization ability and practical value in multi-temporal and spatial scales and unstructured environments. Summary of the Invention

[0006] The purpose of this invention is to overcome the technical bottlenecks in existing orchard anthracnose leaf blight identification methods, such as limited perception dimensions, large light interference, inaccurate spatial positioning, and weak generalization ability of identification models. This invention proposes a five-dimensional data fusion method for apple anthracnose leaf blight identification based on an inspection robot. By constructing a five-dimensional phenotypic feature matrix model 5D-CPM that integrates spatial distribution, visible light images, spectral reflectance, thermal infrared temperature, and time series information, and combining it with an autonomous navigation field inspection robot and a multimodal sensing system, the accuracy, practicality, and real-time performance of disease detection in orchard environments are improved.

[0007] The technical solution adopted by the present invention to solve its technical problem is: a five-dimensional data fusion method for identifying apple anthracnose leaf blight based on inspection robots, comprising the following steps.

[0008] S1. Obtain the surface curvature characteristics of apple leaves, collect RGB-D images of the leaves, construct the three-dimensional topological structure of the leaves, and locate the spatial distribution of lesions.

[0009] S2. Obtain spectral reflectance information of apple leaves, quantify chlorophyll a and carotenoid biochemical components, and construct a set of vegetation index features sensitive to diseases.

[0010] S3. Collect apple tree canopy temperature information, construct an anomaly detection model for transpiration, and achieve multimodal fusion of thermal infrared and point cloud data.

[0011] S4. Integrating spatial distribution, spectral reflectance, and thermal radiation information, a five-dimensional phenotypic feature matrix model 5D-CPM is constructed to comprehensively analyze the phenotypic features of apples.

[0012] Furthermore, in step S1, the three-dimensional spatial structure and curvature information of the apple leaf are acquired by a rotating lidar, and the Gaussian curvature of each point is estimated using a spherical fitting curvature algorithm. It is used to identify the uneven features of leaf edges, wrinkles, and suspected lesions; (1), where, This represents the height function of the blade surface in three-dimensional space. To describe along The trend of curvature change in the direction. To describe along The trend of curvature change in the direction. For the reaction surface in and Bending coupling relationship under directional interaction.

[0013] Furthermore, images of the front area of ​​the fruit tree were acquired using a binocular RGB-D camera to obtain RGB-D images of the leaves; illumination normalization was performed using the Retinex illumination correction model to enhance the contrast of the lesion images; a visual point cloud was constructed by combining the depth map with the corrected RGB image; a multi-layer feature extraction, cross-attention enhancement, and transform estimation module was used to achieve precise geometric anchoring of the lesion region through high-dimensional feature alignment of the source / target point clouds; the iterative nearest point ICP algorithm was used to rigidly register the lidar point cloud and the RGB-D visual point cloud to align the spatial coordinates; time synchronization adopted the PPS pulse triggering mechanism combined with the ROS clock correction method to align the inter-frame timestamps of each sensor to ensure that the error is less than 1ms; by analyzing the overlap between the high curvature point region and the suspected lesion region in the RGB image, the spatial region of the lesion was located for multimodal mapping through connected component segmentation and boundary tracking.

[0014] Furthermore, multi-channel reflectance spectra of apple tree canopy were collected using a multispectral sensor to focus on characteristic absorption bands related to plant physiology, including the strong absorption characteristics of chlorophyll a in the red and red edge regions, and the reflectance changes of carotenoids in the blue region.

[0015] Furthermore, to eliminate the interference of ambient light variations and atmospheric scattering on the original spectrum, the original reflectance data were preprocessed with radiometric and atmospheric corrections. Radiometric correction employed standard whiteboard reflectance calibration technology to convert the digital values ​​output by the sensor into physical relative reflectance. Atmospheric correction used the inverse radiative transfer method based on the MODTRAN model to eliminate the influence of atmospheric path radiation and aerosol interference, thereby obtaining the true spectral response of the near-surface layer.

[0016] Furthermore, based on the pretreatment, three types of disease-sensitive vegetation indices were constructed: the Normalized Difference Vegetation Index (NDVI), the Photochemical Reflectance Index (PRI), and the Modified Simple Ratio Index (mSR705). (2); where, Indicates near-infrared reflectivity. Indicates the reflectivity in the red light band; (3); where R 531 R is the reflectance at a wavelength of 531nm. 570 The reflectance at a wavelength of 570nm; (4); where R 750 R is the reflectance at a wavelength of 750nm. 445 The reflectance is at a wavelength of 445nm.

[0017] Furthermore, the temperature distribution of the apple tree canopy was monitored using a thermal infrared imaging module, and the acquired results were a two-dimensional temperature image. This image was then mapped to a point cloud space after internal parameter calibration and external parameter registration. A theoretical temperature prediction equation for healthy leaves under natural conditions was constructed, and a leaf transpiration anomaly detection model was defined, as shown in the following equation: (5); In the formula This indicates the predicted leaf temperature. Indicates photosynthetically active radiation; Ambient temperature; Relative humidity; For wind speed; by fitting historical health data, the regression coefficients of each parameter are determined. , , , and Using actual observations of blade temperature With predicted temperature The deviation ΔT is used for evaporation disorder diagnosis. ΔT > a set threshold is used to determine an abnormal heating zone, indicating the presence of lesions. Vector superposition is achieved through multimodal data association and resolution compensation, mapping multispectral reflectance and thermal infrared temperature to three-dimensional point cloud vertices to construct five-dimensional attribute points, as shown in the following formula: (6), where, These are the five-dimensional attribute points corresponding to spatial points; These are the spatial coordinates in a 3D point cloud. This represents the corresponding multispectral reflectance; This corresponds to the thermal infrared temperature. It is a five-dimensional real space; bicubic interpolation is used to perform super-resolution reconstruction of temperature images, and pixel-to-point cloud alignment is performed with the three-dimensional point cloud through nearest neighbor or KNN mapping strategy to achieve sub-millimeter level high-precision spatial fusion.

[0018] Furthermore, a two-stage optimization process is performed on the five-dimensional phenotypic feature matrix: outlier removal and point cloud data compression. Outlier removal utilizes a Mahalanobis distance-based outlier identification method, calculated as follows: (7); where, Let represent the feature vector of the i-th data point; ε is the mean vector of the dataset. Represents the square of the Mahalanobis distance; T represents the thermal radiation temperature; set threshold Remove Outliers are identified to suppress the interference of abnormal data on subsequent analysis results. Point cloud data compression uses an octree structure to perform hierarchical partitioning and voxelization of the point cloud, and compresses the data volume by 70% through a spatial redundancy elimination strategy while preserving the topological connectivity.

[0019] Furthermore, the training process of the five-dimensional phenotypic feature matrix model includes the following steps: S4.1, Data preparation and augmentation; collecting leaf samples from apple trees under different health conditions, covering multiple typical disease stages including healthy, early-stage lesions, mid-stage expansion, and late-stage necrosis; acquiring the three-dimensional spatial point cloud, spectral reflectance, and thermal radiation temperature information of each leaf sample through a structured light, multispectral, and thermal infrared fusion sensing system, thereby constructing a five-dimensional phenotypic feature matrix that integrates spatial distribution structure, spectral physiological properties, and temperature stress characteristics; S4.2, Constructing a two-stream three-dimensional convolutional neural network architecture, including an image preprocessing module, a feature extraction module, and a feature fusion module; S4.3, Constructing and optimizing the loss function; the loss function is: (8); where, Losses are categorized by leaf disease. Loss of lesion location repositioning (9), among which, For the prediction box, For the true frame, It is the minimum envelope rectangle; For the target existence confidence loss, (10), among which, For the predicted probability of the target class, and For hyperparameters; S4.4, introduce transfer learning and domain adaptation; add a gradient reversal layer at the end of the physiological flow, and combine it with a domain discriminator to form an adversarial network module, whose loss function is: (11), where, The true domain label represents the domain probability predicted by the model. This represents the probability that the model predicts the i-th data point as the target domain; S4.5, Regularization is introduced: Zeroing is randomly applied to feature maps in block-like regions to block local dependencies and enhance the model's robustness to locally redundant features; S4.6, Phased training: The five-dimensional phenotypic feature matrix model is trained using a coarse-tuning-fine-tuning-online update approach.

[0020] Furthermore, to enhance the diversity and robustness of the training data, various perturbation factors in natural scenes are simulated. In the geometric dimension, random rotation, translation, scaling, and occlusion are used to simulate the differences in acquisition under different shooting angles and occlusion conditions. In the spectral dimension, band perturbation and reflectance drift strategies are introduced to simulate the effects of leaf humidity changes, sensor errors, and illuminance changes on spectral characteristics. In the thermal dimension, temperature fluctuations caused by abnormal transpiration or differences in sunlight are simulated through temperature shift and local thermal anomaly enhancement. Noise superposition and point density variation enhancement are used to simulate the fluctuations in acquisition accuracy and occlusion interference in actual orchard operations.

[0021] Furthermore, the image preprocessing module takes low-resolution multispectral images and high-resolution panchromatic images as input. The multispectral images are upsampled to align spatially with the panchromatic images, while the panchromatic images are downsampled to construct multi-scale representations. The feature extraction module employs parallel geometric and physiological channels to perform deep encoding on spatial topological features and spectral and thermal radiation features, respectively. The feature fusion module introduces a disease-specific attention mechanism, fusing channel attention and spatial attention to achieve semantic alignment and region enhancement between the geometric and physiological flows, enabling the focusing and discrimination of lesion regions. The fused high-resolution feature map is then input into the subsequent decision module for disease detection and classification, providing support for the anchoring and distribution estimation of lesion regions.

[0022] Furthermore, the phased training specifically includes the following steps.

[0023] S4.6.1. In the coarse-tuning stage, the network backbone is initially trained using a large number of labeled samples, so that the network can quickly converge to a stable state and master the basic geometric and spectral-thermal radiation feature characterization capabilities.

[0024] S4.6.2 Fine-tuning stage: Optimize the interaction and fusion module between geometric flow and physiological flow, introduce a lesion-specific attention mechanism, and improve the recognition accuracy of lesion boundaries and morphological changes.

[0025] S4.6.3, In the online update phase, based on low-confidence samples from the historical inference process, a pseudo-label generation and dynamic fine-tuning strategy is introduced to continuously update network parameters. The optimization objective of the model training process is defined using a multi-task joint loss function, covering the bounding box regression loss, target confidence loss, and class discrimination loss in the target detection subtask. The bounding box regression loss corresponds to the network's spatial localization, the target confidence loss corresponds to lesion recognition, and the class discrimination loss corresponds to the performance measurement of disease severity discrimination. After each round of training, the changing trends of bounding box loss, target loss, and classification loss on the validation set are recorded synchronously to monitor the model's generalization ability and overfitting risk.

[0026] The beneficial effects of this invention are as follows: This invention differs from existing methods for identifying apple anthracnose and leaf blight using single image or two-dimensional spectral analysis. By integrating three-dimensional leaf morphology, biochemical spectral and thermal response characteristics, it constructs a data expression model with five-dimensional attributes and achieves highly robust identification through a multimodal deep learning network. This significantly improves the identification accuracy under conditions of leaf shading, uneven lighting, changes in leaf posture, and environmental interference, and has good adaptability to actual orchard deployment and promotion prospects. Attached Figure Description

[0027] Figure 1 A schematic diagram of a depth feature alignment network for spatial topology construction and lesion anchoring.

[0028] Figure 2 This is a flowchart of the training process for a five-dimensional phenotypic feature matrix model.

[0029] Figure 3 It is a two-stream three-dimensional convolutional neural network architecture.

[0030] Figure 4 This is a diagram illustrating the effectiveness of the present invention in identifying apple anthracnose leaf blight.

[0031] Figure 5 This is a graph showing the convergence of the loss function and the fluctuation of the evaluation metrics. Detailed Implementation

[0032] This invention proposes a five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot. Relying on a field intelligent inspection platform integrating multimodal sensing equipment, it fuses spatial distribution structure, spectral reflectance characteristics, and thermal response information to construct a unified five-dimensional attribute phenotypic matrix, enabling early and accurate detection and disease warning of diseased leaves in orchard environments. The following description, in conjunction with the accompanying drawings, illustrates this five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot.

[0033] A five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot includes the following steps: S1, acquiring the surface curvature features of apple leaves, collecting RGB-D images of the leaves, constructing the three-dimensional topological structure of the leaves, and locating the spatial distribution of lesions.

[0034] A rotating lidar was used to acquire the three-dimensional spatial structure and curvature information of apple leaves, providing a unified coordinate reference for multi-source data fusion. The rotating lidar, located on top of the inspection robot, was set with an angular resolution of 0.1° and a linear velocity frequency of 20Hz, performing a 360° scan to acquire a high-density point cloud of the fruit tree target. A spherical curvature fitting algorithm was used to estimate the Gaussian curvature of each point. It is used to identify the uneven features of leaf edges, wrinkles, and suspected lesions. (1), where, It represents the height function of the blade surface in three-dimensional space. To describe along The trend of curvature change in the direction. To describe along The trend of curvature change in the direction. For the reaction surface in and Bending coupling relationship under directional interaction.

[0035] Images of the front area of ​​fruit trees were acquired using a binocular RGB-D camera to obtain RGB-D images of the leaves. The RGB resolution was no less than 3840×2160, and the depth map accuracy was better than 1mm. Illumination normalization was performed using the Retinex illumination correction model to enhance the contrast of lesion images. The depth map and the corrected RGB image were combined to construct a visual point cloud, compensating for the data sparsity problem of the lidar in the leaf edge area.

[0036] like Figure 1 The diagram shows a registration neural network structure for spatial alignment of point cloud data. This network employs multi-layer feature extraction, cross-attention enhancement, and transform estimation modules. Through high-dimensional feature alignment of the source / target point clouds, it achieves precise geometric anchoring of the lesion region. The Iterative Closest Point (ICP) algorithm is used for rigid registration of the LiDAR point cloud and the RGB-D visual point cloud, ensuring spatial coordinate alignment with an error controlled within 2mm. Time synchronization utilizes a PPS pulse triggering mechanism combined with the ROS clock correction method to align the timestamps between sensor frames, ensuring an error of less than 1ms.

[0037] By analyzing the overlap between high curvature point regions and suspected lesion regions in the RGB image, and through connected component segmentation and boundary tracing, the spatial regions of lesions are located for multimodal mapping.

[0038] S2. Obtain spectral reflectance information of apple leaves and quantify key biochemical components such as chlorophyll a and carotenoids to further construct a vegetation index feature set sensitive to diseases.

[0039] Multi-channel reflectance spectra of the apple tree canopy were acquired using a multispectral sensor covering the wavelength range of 400-1000 nm, with a channel interval of 10 nm, collecting data from a total of 16 spectral channels. The focus was on characteristic absorption bands relevant to plant physiology, including the strong absorption characteristics of chlorophyll a in the red and red-edge regions, and the reflectance changes of carotenoids in the blue region, to quantify the concentration changes of key biochemical components. In the formula, the red region refers to the 660 nm band, the red-edge region to the 705 nm band, and the blue region to the 470 nm band.

[0040] To eliminate the interference of ambient light variations and atmospheric scattering on the original spectrum, the original reflectance data were preprocessed with radiometric and atmospheric corrections.

[0041] Radiation correction employs standard whiteboard reflectance calibration technology to convert the digital values ​​output by the sensor into physical relative reflectance.

[0042] Atmospheric correction employs the inverse radiative transfer method based on the MODTRAN model to eliminate the influence of atmospheric path radiation and aerosol interference, thereby obtaining the true spectral response of the near-surface layer.

[0043] Based on the above pretreatment, three types of disease-sensitive vegetation indices were further constructed: the Normalized Difference Vegetation Index (NDVI), the Photochemical Reflectance Index (PRI), and the Modified Simple Ratio Index (mSR705).

[0044] (2). In the formula, Indicates near-infrared reflectivity. NDVI represents the reflectance of the red light band and reflects vegetation vitality; diseased areas are usually accompanied by a decrease in NDVI.

[0045] (3). In the formula, PRI is the photochemical reflectance index, used to quantify changes in the efficiency of plant photosynthesis. R 531 R is the reflectance at a wavelength of 531nm. 570 The reflectance is at a wavelength of 570nm.

[0046] (4). In the formula, A modified simple ratio index centered at 705 nm was used to enhance sensitivity to differences in chlorophyll concentration. R 750 R is the reflectance at a wavelength of 750nm. 445 The reflectance is at a wavelength of 445nm.

[0047] S3. Collect apple tree canopy temperature information, construct an anomaly detection model for transpiration, and achieve multimodal fusion of thermal infrared and point cloud data.

[0048] Temperature distribution in the apple tree canopy is monitored using a thermal infrared imaging module. The device's temperature measurement accuracy is controlled within ±0.5℃, and the acquisition frequency is 10 frames per second, enabling real-time capture of micro-regional thermal anomalies induced by disease. The acquired results are two-dimensional temperature images, which are then mapped to point cloud space after internal parameter calibration and external parameter registration.

[0049] To determine whether temperature anomalies in diseased areas are attributable to non-physiological transpiration processes, a theoretical temperature prediction equation for healthy leaves under natural conditions is constructed, and a leaf transpiration anomaly detection model is defined as follows: (5). In the formula This indicates the predicted leaf temperature. This represents photosynthetically active radiation, with a value ranging from 0 to 20000. . The ambient temperature. This refers to relative humidity. Wind speed, unit: By fitting historical health data, the regression coefficients of each parameter were determined. , , , and .

[0050] Based on the above model, actual observed blade temperature With predicted temperature The deviation ΔT is used to diagnose transpiration imbalance, with a threshold set at 1.5℃. ΔT > 1.5℃ is considered an abnormal temperature rise zone, indicating the presence of lesions.

[0051] Vector overlay is achieved through multimodal data association and resolution compensation, mapping multispectral reflectance and thermal infrared temperature to 3D point cloud vertices to construct five-dimensional attribute points, as shown in the following equation: (6), where, These are the five-dimensional attribute points corresponding to spatial points. These are the spatial coordinates in a 3D point cloud. This represents the corresponding multispectral reflectance. The corresponding thermal infrared temperature, It is a five-dimensional real number space, representing that each fused data point consists of five real values.

[0052] Since the resolution of thermal infrared images is much lower than that of point clouds, bicubic interpolation is used to perform super-resolution reconstruction of temperature images, and pixel-to-point cloud alignment is performed with the three-dimensional point cloud through nearest neighbor or KNN mapping strategies to achieve sub-millimeter level high-precision spatial fusion.

[0053] S4. Integrating spatial distribution, spectral reflectance, and thermal radiation information, a five-dimensional phenotypic feature matrix model 5D-CPM is constructed to comprehensively analyze the phenotypic features of apples.

[0054] The 5D-CPM five-dimensional phenotypic feature matrix model systematically characterizes the spatial morphology, biochemical features, and physiological state of apple leaves, providing multimodal support for accurate identification of lesion areas. To improve the model's robustness and computational efficiency, the initially constructed five-dimensional phenotypic feature matrix was optimized in two stages: outlier removal and point cloud data compression.

[0055] Outlier Removal: During synchronous acquisition by multiple sensors, outliers may exist due to reflectivity drift, thermal image mismatch, or edge blurring. Therefore, an outlier identification method based on Mahalanobis distance is introduced. The calculation formula is as follows: (7). In the formula, Let represent the feature vector of the i-th data point. ε is the mean vector of the dataset, representing the average value of each dimension. This represents the square of the Mahalanobis distance. T represents the thermal radiation temperature. (Settings...) threshold Remove This helps identify outliers and thus suppress the interference of abnormal data on subsequent analysis results.

[0056] Point cloud data compression: To reduce the storage and computational burden of five-dimensional point cloud data in subsequent recognition models, an octree structure is used to perform hierarchical division and voxelization of the point cloud. The data volume is compressed by 70% through a spatial redundancy elimination strategy, while preserving the topological connection relationship.

[0057] like Figure 2 As shown, the training process of the five-dimensional phenotypic feature matrix model includes the following steps:

[0058] S4.1 Data Preparation and Enhancement.

[0059] To construct a multimodal disease identification dataset with high representativeness and generalization ability, no fewer than 10,000 leaf samples of apple trees under different health conditions were collected, covering multiple typical disease stages, including healthy, early-stage lesions, mid-stage expansion, and late-stage necrosis. For each leaf sample, three-dimensional spatial point cloud, spectral reflectance, and thermal radiation temperature information were acquired through a structured light, multispectral, and thermal infrared fusion sensing system, thereby constructing a five-dimensional phenotypic feature matrix that integrates spatial distribution structure, spectral physiological properties, and temperature stress characteristics.

[0060] To enhance the diversity and robustness of training data, a series of data augmentation strategies were designed to simulate various perturbation factors in natural scenes. (1) In the geometric dimension, random rotation, translation, scaling and occlusion were used to simulate the differences in acquisition under different shooting angles and occlusion conditions. (2) In the spectral dimension, band perturbation and reflectance drift strategies were introduced to simulate the effects of leaf surface humidity changes, sensor errors and illuminance changes on spectral characteristics. (3) In the thermal dimension, temperature fluctuations caused by abnormal transpiration or differences in sunlight were simulated by temperature shift and local thermal anomaly enhancement; noise superposition and point density change enhancement were used to simulate the fluctuations in acquisition accuracy and occlusion interference in actual orchard operations.

[0061] S4.2 Build the model architecture.

[0062] like Figure 3 The invention constructs a dual-stream three-dimensional convolutional neural network architecture, which includes an image preprocessing module, a feature extraction module, and a feature fusion module. By fusing structural and functional information from multimodal data, the accuracy and robustness of disease identification are improved.

[0063] Image preprocessing module: Input data includes low-resolution multispectral images and high-resolution panchromatic images. The low-resolution multispectral images provide physiological information in the spectral dimension, while the high-resolution panchromatic images assist in enhancing spatial resolution. First, the multispectral images are upsampled to align spatially with the panchromatic images. Simultaneously, the panchromatic images are downsampled to construct multi-scale representations, mitigating scale inconsistency issues. This stage establishes a unified multidimensional representation foundation by fusing input images from different scales.

[0064] Feature extraction module: Parallel geometric and physiological channels are employed to perform deep encoding on spatial topological features and spectral and thermal radiation features, respectively. The physiological channel extracts anomalous information in reflectivity and canopy temperature across different wavelengths through cascaded convolutional blocks and residual structures, forming mid- to high-dimensional physiological feature maps. The geometric channel, based on voxelized representations of spatial point clouds, extracts local structural perturbations and macroscopic deformation patterns on the leaf surface, such as curling and bending. Both channels undergo multi-level convolution-pooling operations to progressively enhance the semantic hierarchy of feature representation and retain multi-scale responses from intermediate layers for subsequent fusion.

[0065] Feature fusion module: Introducing a disease-specific attention mechanism, this module fuses channel attention and spatial attention to perform semantic alignment and region enhancement between geometric and physiological flows, enabling focusing and discrimination of lesion regions. The fused high-resolution feature map is then input into the subsequent decision module for disease detection and classification, providing support for lesion region anchoring and distribution estimation.

[0066] This architecture fully integrates two complementary features: spatial structure and physiological response. Through an end-to-end training mechanism, it significantly improves the model's ability to perceive minute lesions and early disease signals.

[0067] S4.3 Constructing and optimizing the loss function.

[0068] To address the issues of class imbalance and ambiguous boundaries in apple leaf anthracnose detection, this invention constructs a multi-task loss function to jointly optimize three sub-tasks: classification, localization, and target confidence. The overall loss function is as follows: (8). In the formula, To mitigate class imbalance, a weighted cross-entropy method is used to classify losses due to leaf diseases. For the lesion location regression loss, GIoU is used to optimize the overlap between the predicted bounding box and the ground truth bounding box, and its expression is as follows: (9). In the formula, This is the predicted bounding box. This is a true bounding box. It is the minimum envelope rectangle. To address the target existence confidence loss, a Focal Loss mechanism is introduced to suppress the influence of easily classifiable samples on gradient dominance. Its form is as follows: (10). In the formula, For the predicted probability of the target class, and These are hyperparameters used to balance class weights and adjust the attention given to difficult samples. In terms of optimization, this invention employs a Nesterov Accelerated Gradient form of the Adam optimizer, introducing forward gradient acceleration while maintaining first-order adaptive updates to improve training convergence speed and stability. Cosine annealing scheduling is introduced to dynamically adjust the learning rate, avoiding getting trapped in local optima. The overall training framework supports early stopping and gradient pruning, ensuring the model has strong convergence and generalization capabilities in high-dimensional feature spaces.

[0069] S4.4 Introducing transfer learning and domain adaptation.

[0070] Considering the significant differences between laboratory environments and real-world orchard settings in terms of lighting conditions, background interference, and leaf morphology, directly training deep learning models in an orchard setting faces challenges such as sample scarcity and inconsistent data distribution. This invention introduces a transfer learning strategy, utilizing the publicly available PlantVillage dataset as the source domain data to pre-train the physiological flow branch. This fully leverages the fundamental correlation between spectral and temperature features, providing stable initialization parameters for downstream tasks. Furthermore, a domain adaptation mechanism is integrated to minimize the feature distribution differences between the source and target domains through adversarial training. Specifically, a gradient inversion layer is added at the end of the physiological flow, combined with a domain discriminator to form an adversarial network module. The loss function is as follows: (11). In the formula, The labels are the true domain labels, with 0 for the source domain and 1 for the target domain, representing the domain probabilities predicted by the model. This represents the probability that the model predicts the i-th data point as belonging to the target domain. During backpropagation via the gradient reversal layer, the backbone network is guided to learn domain-indistinguishable features, thus achieving cross-domain feature alignment. To avoid overfitting due to the small sample size in causal orchards, feature normalization and batch normalization mechanisms are introduced to mitigate statistical bias caused by small batches of data during training, improving the model's robustness and generalization ability in unstructured orchard environments.

[0071] S4.5, Introduction of Regularization.

[0072] To enhance the model's generalization ability in high-dimensional multimodal feature learning and suppress overfitting caused by the limited distribution of training samples, multiple regularization mechanisms are introduced during network training. The DropBlock method effectively blocks local dependencies by randomly zeroing out feature maps in block-like regions, enhancing the model's robustness to locally redundant features. Random channel discarding randomly masks some channels during training, preventing excessive reliance on specific channels during feature extraction and promoting more dispersed and stable feature representations. Spectral normalization limits the Lipschitz constant of the convolutional kernel during feature transformation, stabilizing gradient propagation and preventing training instability or weight explosion. These multiple regularization strategies work synergistically to significantly improve the model's robustness and generalization ability in complex orchard scenarios while ensuring convergence efficiency.

[0073] S4.6, Phased training.

[0074] To adapt to the practical needs of multimodal high-dimensional feature learning tasks in dynamic agricultural scenarios, a phased model training mechanism is designed, covering three strategies: coarse tuning, fine tuning, and online update.

[0075] S4.6.1 In the coarse-tuning stage, the backbone of the network is initially trained using a large number of labeled samples, so that the network can quickly converge to a stable state and master the basic geometric and spectral-thermal radiation feature characterization capabilities.

[0076] S4.6.2. Then, we enter the fine-tuning stage, focusing on optimizing the interaction and fusion module between geometric flow and physiological flow, introducing a lesion-specific attention mechanism, and improving the recognition accuracy of lesion boundaries and morphological changes.

[0077] S4.6.3, In the online update phase, based on low-confidence samples from the historical inference process, a pseudo-label generation and dynamic fine-tuning strategy is introduced to continuously update network parameters. This mitigates catastrophic forgetting caused by sample drift during long-term deployment, improving the system's long-term adaptability and stability. The optimization objective of the model training process is defined using a multi-task joint loss function, encompassing the bounding box regression loss, target confidence loss, and class discrimination loss in the target detection subtask. The bounding box regression loss corresponds to the network's spatial localization, the target confidence loss corresponds to lesion recognition, and the class discrimination loss corresponds to the performance measurement of disease severity discrimination. After each training round, the changing trends of the bounding box loss, target loss, and classification loss on the validation set are recorded synchronously to monitor the model's generalization ability and overfitting risk. The dynamic change curves of the relevant loss functions during the training and validation phases are shown below. Figure 5 As shown, this characterizes the performance evolution of the model under a multi-stage training framework.

[0078] The five-dimensional data fusion method of this invention was experimentally verified, and the model showed good results in identifying apple anthracnose. Figure 4 As shown, the present invention can achieve accurate monitoring of anthrax.

[0079] This invention demonstrates significant advantages in identifying apple anthracnose leaf blight, achieving a detection rate of 91.7% for early-stage disease, a 23.5% improvement over the manual method, and providing an average early warning time 3.8 days earlier. In the severity grading task, the highest accuracy rate (96.2%) was achieved for healthy leaves, with accuracy rates of 88.5% and 82.1% for levels 1 and 2, respectively, indicating that there is still room for improvement in the sensitivity of this invention to small-area lesions. The false alarm rate was reduced by 16.6% compared to the manual method.

Claims

1. A five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot, characterized in that, Includes the following steps: S1. Obtain the surface curvature characteristics of apple leaves, collect RGB-D images of the leaves, construct the three-dimensional topological structure of the leaves, and locate the spatial distribution of lesions; S2. Obtain spectral reflectance information of apple leaves, quantify chlorophyll a and carotenoid biochemical components, and construct a set of vegetation index features sensitive to diseases. S3. Collect temperature information of apple tree canopy, construct an anomaly detection model for transpiration and realize multimodal fusion of thermal infrared and point cloud data; S4. Integrating spatial distribution, spectral reflectance, and thermal radiation information, a five-dimensional phenotypic feature matrix model 5D-CPM is constructed to comprehensively analyze the phenotypic features of apples. In step S1, the three-dimensional spatial structure and curvature information of apple leaves are obtained by rotating lidar, and the Gaussian curvature K of each point is estimated by spherical fitting curvature algorithm, which is used to identify the uneven features of leaf edges, wrinkles and suspected lesions. In the formula, z=f(x,y) represents the height function of the blade surface in three-dimensional space; To describe the curvature variation trend along the x-direction, To describe the curvature variation trend along the y-direction, This represents the bending coupling relationship of the reaction surface under the interaction of the x and y directions; Temperature distribution in the apple tree canopy was monitored using a thermal infrared imaging module. The acquired results were two-dimensional temperature images, which were then mapped to point cloud space after intrinsic parameter calibration and extrinsic parameter registration. A theoretical temperature prediction equation for healthy leaves under natural conditions was constructed, and a leaf transpiration anomaly detection model was defined, as shown in the following equation: T pred =β0 + β1PAR + β2T air +β3RH+β4μ(2); where, T pred This indicates the predicted leaf temperature; PAR represents photosynthetically active radiation; T air RH is ambient temperature; μ is relative humidity; μ is wind speed. By fitting historical health data, the regression coefficients β0, β1, β2, β3, and β4 of each parameter were determined; the actual observed leaf temperature T was used. obs With predicted temperature T pred The deviation ΔT is used for evaporation disorder diagnosis. ΔT > a set threshold is used to determine an abnormal heating zone, indicating the presence of lesions. Vector superposition is achieved through multimodal data association and resolution compensation, mapping multispectral reflectance and thermal infrared temperature to three-dimensional point cloud vertices to construct five-dimensional attribute points, as shown in the following formula: P i =(x i y i , z i , λ i T i )∈R 5 (3), where P i The five-dimensional attribute point corresponding to the spatial point; x i y i z i λ represents the spatial coordinates in a 3D point cloud. i T represents the corresponding multispectral reflectance. i R represents the corresponding thermal infrared temperature. 5 It is a five-dimensional real space; bicubic interpolation is used to perform super-resolution reconstruction of temperature images, and pixel-to-point cloud alignment is performed with the three-dimensional point cloud through nearest neighbor or KNN mapping strategy to achieve sub-millimeter level high-precision spatial fusion.

2. The five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot according to claim 1, characterized in that, Images of the front area of ​​fruit trees were acquired using a binocular RGB-D camera to obtain RGB-D images of the leaves. Illumination normalization was performed using the Retinex illumination correction model to enhance the contrast of the lesion images. A visual point cloud was constructed by combining the depth map with the corrected RGB image. A multi-layer feature extraction, cross-attention enhancement, and transform estimation module was used to achieve precise geometric anchoring of the lesion region through high-dimensional feature alignment of the source / target point clouds. The iterative nearest point (ICP) algorithm was used to rigidly register the lidar point cloud and the RGB-D visual point cloud to align spatial coordinates. Time synchronization employed a PPS pulse triggering mechanism combined with the ROS clock correction method to align the timestamps between sensor frames. By analyzing the overlap between high-curvature point regions and suspected lesion regions in the RGB image, and through connected component segmentation and boundary tracking, the spatial region of the lesion was located for multimodal mapping.

3. The five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot according to claim 2, characterized in that, Multi-channel reflectance spectra of apple tree canopy were collected using a multispectral sensor to focus on characteristic absorption bands related to plant physiology, including the strong absorption characteristics of chlorophyll a in the red and red edge regions, and the reflectance changes of carotenoids in the blue region.

4. The five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot according to claim 3, characterized in that, The raw reflectance data were preprocessed with radiometric and atmospheric corrections. Radiometric correction used standard whiteboard reflectance calibration technology to convert the digital values ​​output by the sensor into physical relative reflectance. Atmospheric correction used the inverse radiative transfer method based on the MODTRAN model to eliminate the influence of atmospheric path radiation and aerosol interference, and obtain the true spectral response of the near-surface layer.

5. The five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot according to claim 4, characterized in that, Based on the pretreatment, three types of disease-sensitive vegetation indices were further constructed: including the Normalized Difference Vegetation Index (NDVI), the Photochemical Reflectance Index (PRI), and the Modified Simple Ratio Index (mSR705). In the formula, R NIR R represents the reflectivity in the near-infrared band. Red Indicates the reflectivity in the red light band; In the formula, R 531 R is the reflectance at a wavelength of 531nm. 570 The reflectance at a wavelength of 570nm; In the formula, R 750 R is the reflectance at a wavelength of 750nm. 445 The reflectance is at a wavelength of 445nm.

6. The five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot according to claim 1, characterized in that, A two-stage optimization process is performed on the five-dimensional phenotypic feature matrix: outlier removal and point cloud data compression. Outlier removal introduces an outlier identification method based on Mahalanobis distance, and the calculation formula is as follows: In the formula, Q i Let represent the feature vector of the i-th data point; ε is the mean vector of the dataset. Represents the square of the Mahalanobis distance; T represents the thermal radiation temperature; set Threshold δ, remove Outliers are identified to suppress the interference of abnormal data on subsequent analysis results; point cloud data compression uses an octree structure to perform hierarchical division and voxelization of the point cloud, and compresses the data volume by 70% through a spatial redundancy elimination strategy, while preserving the topological connection relationship.

7. The five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot according to claim 6, characterized in that, The training process of the five-dimensional phenotypic feature matrix model includes the following steps: S4.1, Data preparation and augmentation; collecting leaf samples from apple trees under different health conditions, covering multiple typical disease stages including healthy, early-stage lesions, mid-stage expansion, and late-stage necrosis; acquiring the three-dimensional spatial point cloud, spectral reflectance, and thermal radiation temperature information of each leaf sample through a structured light, multispectral, and thermal infrared fusion sensing system, thereby constructing a five-dimensional phenotypic feature matrix that integrates spatial distribution structure, spectral physiological properties, and temperature stress characteristics; S4.2, Constructing a two-stream three-dimensional convolutional neural network architecture, including an image preprocessing module, a feature extraction module, and a feature fusion module; S4.3, Constructing and optimizing the loss function; the loss function is: In the formula, Losses are categorized by leaf disease. Loss of lesion location regression, Where A is the predicted bounding box, B is the ground truth bounding box, and C is the minimum envelope rectangle; For the target existence confidence loss, Where, p t Let α be the predicted probability of the target class. t γ are hyperparameters; S4.4, introduce transfer learning and domain adaptation; add a gradient reversal layer at the end of the physiological flow, and combine it with a domain discriminator to form an adversarial network module, whose loss function is: In the formula, To counteract the domain discrimination loss during training, d i The true domain label represents the domain probability predicted by the model. S4.5, Regularization is introduced; in the feature map, zeros are randomly set in block regions to block local dependencies and enhance the robustness of the model to local redundant features; S4.6, Phased training; the five-dimensional phenotypic feature matrix model is trained by coarse-fine-tuning-online update.

8. A five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot according to claim 7, characterized in that, To enhance the diversity and robustness of training data, various perturbation factors in natural scenes are simulated. In the geometric dimension, random rotation, translation, scaling, and occlusion are used to simulate the differences in acquisition under different shooting angles and occlusion conditions. In the spectral dimension, band perturbation and reflectance drift strategies are introduced to simulate the impact of leaf humidity changes, sensor errors, and illuminance changes on spectral characteristics. In the thermal dimension, temperature fluctuations caused by abnormal transpiration or differences in sunlight are simulated through temperature shift and local thermal anomaly enhancement. Noise superposition and point density variation enhancement are used to simulate the fluctuations in acquisition accuracy and occlusion interference in actual orchard operations.

9. A five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot according to claim 8, characterized in that, The input data for the image preprocessing module includes low-resolution multispectral images and high-resolution panchromatic images. The multispectral images are upsampled to align with the panchromatic images in spatial dimensions, while the panchromatic images are downsampled to construct multi-scale representations. The feature extraction module uses parallel geometric and physiological channels to perform deep encoding on spatial topological features and spectral and thermal radiation features, respectively. feature The fusion module introduces a disease-specific attention mechanism, which integrates channel attention and spatial attention to perform semantic alignment and region enhancement between geometric and physiological flows, thereby enabling the focusing and discrimination of lesion areas. The fused high-resolution feature map is input into the subsequent decision-making module for disease detection and classification, providing support for the anchoring and distribution estimation of lesion areas.

10. A five-dimensional data fusion method for identifying apple anthracnose leaf blight based on an inspection robot according to claim 9, characterized in that, Phased training specifically includes the following steps: S4.6.

1. In the coarse-tuning stage, the network backbone is initially trained using a large number of labeled samples, so that the network can quickly converge to a stable state and master the basic geometric and spectral-thermal radiation feature characterization capabilities. S4.6.2 Fine-tuning stage: Optimize the interaction and fusion module between geometric flow and physiological flow, introduce a lesion area-specific attention mechanism, and improve the recognition accuracy of lesion boundaries and morphological changes; S4.6.

3. In the online update phase, based on low-confidence samples from the historical inference process, a pseudo-label generation and dynamic fine-tuning strategy is introduced to continuously update the network parameters. The optimization objective of the model training process is defined using a multi-task joint loss function, which covers the box regression loss, target confidence loss, and class discrimination loss in the target detection subtask. The box regression loss corresponds to the network's spatial localization, the target confidence loss corresponds to lesion recognition, and the class discrimination loss corresponds to the performance measurement of disease level discrimination. After each round of training, the changing trends of box loss, target loss, and classification loss on the validation set are recorded synchronously to monitor the model's generalization ability and overfitting risk.

Images