A plant leaf spot segmentation method and system based on RGB-D cross-modal fusion

Abstract

The present invention belongs to the field of agricultural information perception and relates to a plant leaf spot segmentation method and system based on RGB-D cross-modal fusion, comprising: using a synchronous acquisition device to collect color images and depth images of crops; registering the color images and depth images to obtain a registered color image and its corresponding registered depth image; constructing an initial segmentation network and performing model training on the initial segmentation network; the initial segmentation network includes a feature aggregation module based on cross-attention and an adaptive feature fusion module guided by attention; inputting the registered color image and the registered depth image into the trained segmentation network for segmentation to obtain a segmentation result; through the heterogeneous data fusion of a depth sensor and a visible light camera, a multi-dimensional feature system covering two-dimensional texture and three-dimensional deformation is constructed; combining the cross-modal feature complementary mechanism to enhance the recognition robustness of weakly dominant pests and diseases, and significantly improve the segmentation accuracy in complex lighting and branch and leaf occlusion scenes.

Description

Technical Field

[0001] The present invention relates to the field of agricultural information perception, and specifically discloses a plant leaf spot segmentation method and system based on RGB-D cross-modal fusion. Background Art

[0002] Currently, there are certain technical bottlenecks in the field of intelligent detection of agricultural pests and diseases. For example, traditional analysis methods based on visible light images face multiple challenges in complex farmland environments: subject to the inherent limitations of two-dimensional imaging, multi-layer occlusion of leaves can easily lead to the loss of characteristic information of lesions, and fluctuations in lighting conditions cause distortion of color features. The similarity in texture and color between healthy tissue and early disease and pest areas further increases the difficulty of accurate identification. In addition, existing technologies lack quantitative analysis of the three-dimensional deformation characteristics of crop surfaces, making it difficult to assess the spatial distribution of disease and pest infestations.

[0003] In view of this, the present invention proposes a plant leaf lesion segmentation method and system based on RGB-D cross-modal fusion, introduces a collaborative analysis mechanism for RGB-D multimodal data, and constructs a multi-dimensional feature system covering two-dimensional texture and three-dimensional deformation through the heterogeneous data fusion of depth sensors and visible light cameras. By using depth information to accurately characterize microscopic deformation features such as leaf wrinkles and lesion ridges, and combining the cross-modal feature complementary mechanism to enhance the robustness of recognition of weakly dominant pests and diseases, compared with traditional methods, the present invention significantly improves the segmentation accuracy in complex lighting and branch and leaf occlusion scenes, provides a real-time decision-making basis for precise variable spraying equipment, and promotes the paradigm shift of agricultural pest and disease control from empirical judgment to data intelligence drive. It has important practical value in achieving the reduction of pesticide use, ensuring food production safety, and promoting the construction of a smart agricultural system. Summary of the Invention

[0004] The purpose of the present invention is to provide a plant leaf spot segmentation method based on RGB-D cross-modal fusion, which solves the inherent limitations of existing technologies that are subject to two-dimensional imaging. Fluctuations in lighting conditions cause color feature distortion, and the similarity in texture and chromaticity between healthy tissue and early disease and insect pest areas further increases the difficulty of accurate identification. Existing technologies lack quantitative analysis of three-dimensional deformation characteristics of crop surfaces, making it difficult to assess the spatial distribution of disease and insect pest infestation. This improves segmentation accuracy in complex lighting and foliage occlusion scenarios, provides real-time decision-making basis for precise variable spraying equipment, and promotes the paradigm shift in agricultural disease and insect pest control from empirical judgment to data intelligence-driven. It has important practical value in achieving reduced pesticide use, ensuring food safety production, and promoting the construction of a smart agricultural system. The specific plan is as follows:

[0005] A plant leaf spot segmentation method based on RGB-D cross-modal fusion comprises: using a synchronous acquisition device to collect color images and depth images of crops; the synchronous acquisition device comprises a visible light camera and a depth sensor; registering the color image and the depth image to obtain a registered color image and its corresponding registered depth image; constructing an initial segmentation network and performing model training on the initial segmentation network; the initial segmentation network comprises a feature aggregation module based on cross-attention and an adaptive feature fusion module guided by attention; inputting the registered color image and the registered depth image into the trained segmentation network for segmentation to obtain a segmentation result; the segmentation result comprises a complete leaf and a diseased spot area.

[0006] Furthermore, it also includes using a data set to evaluate the severity of leaf spots on the segmentation results to obtain an evaluation result; the data set is an image data set related to leaf spots.

[0007] Furthermore, the severity of the lesions in the segmentation results is evaluated by a comprehensive scoring model to obtain an evaluation result, including: constructing a weighted scoring model based on lesion indicators; the lesion indicators include lesion type, lesion area and depth offset characteristics; constructing a judgment matrix based on the importance between lesion indicators; calculating the eigenvector of the judgment matrix to determine the weight of the weighted scoring model; using the weighted scoring model with determined weights as a comprehensive scoring model, and calculating the scores through the comprehensive scoring model, and grading the scores to obtain the evaluation result.

[0008] Furthermore, the weighted scoring model is:

[0009] ;

[0010] in, For rating; 、 and are the first scoring weight, the second scoring weight, and the third scoring weight, respectively; i is the lesion type variable; n is the total number of lesion types; is the type hazard severity parameter (set based on expert experience or literature); is the number of pixels of the i-th type of lesion; is the total number of pixels; is the combined proportion of all lesions to the leaf area; is the combined depth shift percentage of all lesions;

[0011] The judgment matrix is:

[0012] ;

[0013] in, is the importance of indicator 1 relative to indicator 2; The importance of indicator 1 relative to indicator 3; is the importance of indicator 2 relative to indicator 1; The importance of indicator 2 relative to indicator 3; The importance of indicator 3 relative to indicator 1; The importance of indicator 3 relative to indicator 2;

[0014] The evaluation results are:

[0015] ;

[0016] Among them, I is mild injury; II is moderate injury; III is severe injury.

[0017] Furthermore, collecting training samples for the model training includes: using a crop canopy acquisition device to synchronously acquire color images and depth images of crops; the crop canopy acquisition device integrates a high-resolution visible light camera and a depth sensor; spatially aligning the visible light camera and the depth sensor through a checkerboard calibration method to ensure pixel-level coordinate consistency; annotating the color image to create label data and obtain a label image; the label image is used for model training; the label data is created by a crop annotation tool, and includes crop part areas, crop diseased spot areas, and background areas.

[0018] Furthermore, the initial segmentation network adopts a semantic segmentation framework, including an encoder and a decoder; the encoder includes a dual-branch feature extraction module, a cross-modal aggregation module and a feature fusion module; the dual-branch feature extraction module includes a first extraction branch and a second extraction branch; the color image and the depth image are respectively input into the first extraction branch and the second extraction branch to obtain multi-scale color features and depth features; the multi-scale color features and their corresponding depth features are respectively input into the cross-modal aggregation module to obtain multi-scale aggregation features; the multi-scale aggregation features are input into the feature fusion module to obtain fusion features; the fusion features are input into the decoder to obtain the segmentation result.

[0019] Furthermore, the first extraction branch and the second extraction branch each include a plurality of extraction blocks; each extraction block includes an overlapping block embedding, a transformer, and a downsampling;

[0020] The overlapping block embedding is used to map the input image into the initial transformed features through an overlapping convolution operation:

[0021] ;

[0022] in, is the initial transformation feature; To perform overlapping block operations on the input image; input is the input image; and are the height and width of the input image respectively; is the number of characteristic channels; R is the real number space;

[0023] The transformer is used to perform transformation operations on the transformation features of each scale, including clustering linear attention and hybrid feedforward networks:

[0024] ;

[0025] Each transformer block in the transformer consists of a stack of multiple submodules:

[0026] ;

[0027] ;

[0028] in, is the transformation feature of the i-th scale output; i is the scale variable; is the change operation of the i-th scale; is the transformation feature output at the i-1th scale; R is the real number space; is the output feature of the nth submodule in the i-th scale; is the output feature of the n-1th submodule in the i-th scale; is the focused linear attention function; is the hybrid feedforward network function; N is the total number of stacked submodules in each transformer block;

[0029] The downsampling is used to perform overlapping downsampling on the mixed features:

[0030] ;

[0031] in, is the attention feature of the downsampled output; For downsampling processing.

[0032] Furthermore, the focused linear attention function is:

[0033] ;

[0034] ;

[0035] in, is the output linear attention feature; , , are the query vector, key vector, and value vector used for attention calculation, is the mapping function; represents the element-by-element power operation, and P is a hyperparameter that controls the degree of feature focusing; Represents the similarity calculation after focusing function mapping; represents depthwise convolution; Indicates the transposition operation of K; x represents the function variable, here refers to Q and ;ReLU(x) represents the ReLU activation function; represents the power normalization function; Represents the L2 norm operation.

[0036] Furthermore, the cross-modal aggregation module is a cross-focus linear attention feature aggregation module, which is used to aggregate the features of the two modalities in each stage. The cross-focus linear attention formula is:

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] in, is the global context vector of visible light; is the key vector of the visible light image used for attention calculation; Calculate for transposition; is the value vector of the visible light image, also used for attention calculation; is the global context vector of the depth image; is the key vector of the depth image; is the value vector of the depth image; Calculate the results of attention for visible light; is the query vector of the visible light image; It is the input visible light feature of the cross-modal aggregation module; The attention calculation results of the depth image; is the query vector of the depth image; is the SoftMax activation function; is the input deep image feature of the cross-modal aggregation module; It is the feature aggregation result after the cross-focus linear attention module; To merge two features in a specified dimension;

[0043] The feature fusion module is used to fuse the features of each stage:

[0044] ;

[0045] ;

[0046] in, Represents the feature fusion results of each stage; It is a convolution operation with a convolution kernel size of 1×1; and They are features from different levels of the top and bottom layers respectively; The weight is calculated after content attention; for pixel attention; Merge two features on the specified dimension; for spatial attention; For channel attention.

[0047] The purpose of the present invention is also to provide a plant leaf spot segmentation system based on RGB-D cross-modal fusion, comprising a data processing module, a data acquisition module, a network training module, a network segmentation module and an evaluation module; the data acquisition module is configured to acquire color images and depth images of plant leaves; the data processing module is configured to align the acquired color images and depth images to obtain aligned color images and aligned depth images in preparation for model training; the network training module is configured to learn the aligned plant leaf spot images to obtain a segmentation network for identifying spot types; the network segmentation module is configured to perform spot segmentation on plant leaves to obtain segmentation results; the evaluation module is configured to evaluate the severity of plant leaf spots to obtain evaluation results.

[0048] The technical solution of the embodiment of the present invention has at least the following advantages and beneficial effects:

[0049] The present invention uses the optimized Segformer encoder part as the encoder to construct a semantic segmentation model. Compared with the original Segformer encoder, it has similar or even lower computational complexity, but has stronger expressive ability, which makes the attention distribution of the model more focused when processing features, and significantly improves the ability to focus on important features.

[0050] The present invention designs a cross-aggregation linear attention to fuse the features of the two modalities in each stage of the encoder, so that the features of the two modalities fully utilize the complementary features of the two modalities during feature fusion, achieving full interaction of features. The shallow cross-attention aggregation module enhances the fusion of texture detail features, and the deep cross-aggregation linear attention aggregation module optimizes the consistency of the target structure while avoiding feature conflicts caused by direct splicing. The introduced feature fusion module enables the model to focus more on important areas and channels, thereby improving the effect of feature fusion and the overall performance of the model. This feature fusion method introduces an attention mechanism, so that the model does not simply superimpose features, but selectively fuses them according to the information content and importance of the features themselves. This not only reduces unnecessary information redundancy, but also enhances the model's ability to capture key information, thereby improving the model's ability to process complex data while maintaining computational efficiency.

[0051] This lesion severity assessment system, constructed by combining lesion type, area, and depth offset, effectively addresses the technical challenge of monocular vision systems in accurately assessing lesion three-dimensional deformation under conditions of similar color and texture. This multimodal fusion assessment method provides a new technical paradigm for disease diagnosis in precision agriculture. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of a process for plant leaf spot segmentation based on RGB-D cross-modal fusion provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of the structure of a segmented network in an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of the structure of a cross-focus linear attention feature aggregation module provided in an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of the structure of an attention-guided feature fusion module provided in an embodiment of the present invention;

[0056] Figure 5 Schematic diagram of the modules of the plant leaf spot segmentation system based on RGB-D cross-modal fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0058] like Figure 1 As shown in Figure 2, the plant leaf spot segmentation method based on RGB-D cross-modal fusion includes the following:

[0059] Color and depth images of crops are captured using a synchronous acquisition device, which includes a visible light camera and a depth sensor. The synchronous acquisition device is a customized crop canopy acquisition device used to synchronously acquire RGB color and depth images at a fixed height and angle in a farmland scene. A visible light camera is a camera that captures color images. For example, a visible light camera can include a dedicated RGB-D camera or a high-resolution RGB camera. A depth sensor is an image acquisition device capable of acquiring image depth information. For example, a ToF or binocular camera can capture depth information.

[0060] The color image and the depth image are registered to obtain a registered color image and a corresponding registered depth image. The registered color image and the registered depth image are images obtained by adjusting and registering the color image and the depth image. For example, after a dedicated RGB-D camera captures an image, the spatial coordinates of the RGB image and the depth image are aligned to ensure a one-to-one correspondence between the color information and the depth value of each pixel.

[0061] An initial segmentation network is constructed and trained. The initial segmentation network includes a cross-attention-based feature aggregation module and an attention-guided adaptive feature fusion module for segmenting intact leaves and diseased areas. Model training can be performed by training the segmentation network using the collected RGB and depth image datasets to obtain a trained segmentation network. For example, the registered RGB and depth images of plant leaves are used as input to the trained segmentation network to obtain segmentation results.

[0062] In some embodiments, collecting training samples for model training includes: using a crop canopy acquisition device to simultaneously acquire color and depth images of the crop; the crop canopy acquisition device integrates a high-resolution visible light camera and a depth sensor. Spatial alignment of the visible light camera and the depth sensor is achieved using a checkerboard calibration method to ensure pixel-level coordinate consistency. The RGB color image is annotated to generate labeled data, resulting in a labeled image; the labeled image is used for model training. This labeled data, created using a crop annotation tool, includes crop part regions, crop disease spot regions, and background regions. The labeled image is a color image annotated with the crop part regions, crop disease spot regions, and background regions within the image. The crop annotation tool is a specialized tool for identifying and annotating crop pests and diseases. For example, a professional annotation tool can be used to create labeled categories (e.g., leaf, disease_type1, disease_type2, background) for an RGB image of a plant leaf, and assign a color to each category (e.g., green for leaf, red for disease spot A, yellow for disease spot B).

[0063] like Figure 2 As shown, the initial segmentation network uses the encoder part of the optimized semantic segmentation framework (SegFormer) as the encoder part of the segmentation network. The overall framework of semantic segmentation includes an encoder and a decoder. Specifically, Focus Linear Attention is used to replace the attention mechanism in the semantic segmentation framework, so that the attention distribution when the model processes features is more focused, focusing on important features and improving segmentation performance. The encoder includes a dual-branch feature extraction module, a cross-modal aggregation module, and a feature fusion module; the dual-branch feature extraction module includes a first extraction branch and a second extraction branch; the first extraction branch and the second extraction branch block each include module 1, module 2, module 3, and module 4 for feature extraction. The cross-modal aggregation module includes aggregation 1, aggregation 2, aggregation 3, and aggregation 4 for cross-modal aggregation. The dual-branch feature extraction module is used to perform cross-modal feature extraction on RGB-D images through two-way feature extraction operations.

[0064] The color image and depth image are input into the first extraction branch and the second extraction branch, respectively, to obtain multi-scale color features and depth features. The first extraction branch and the second extraction branch each include feature extraction modules at four scales, respectively used to extract attention features at different scales. In some embodiments, the first extraction branch and the second extraction branch each include multiple extraction blocks; each extraction block includes an overlapping block embedding, a transformer, and downsampling.

[0065] The overlapping block embedding is used to map the input image into the initial transformed features through an overlapping convolution operation:

[0066] ;

[0067] in, is the initial transformation feature; To perform overlapping block operations on the input image; input is the input image; and are the height and width of the input image respectively; is the number of characteristic channels; R is the real number space;

[0068] The transformer is used to perform a transformation (Transformer) operation on the transformation features of each scale, including Focus Linear Attention and Mixed Feedforward Network (Mix-FFN):

[0069] ;

[0070] Each transformer block in the transformer consists of a stack of multiple (e.g., N) sub-modules:

[0071] ;

[0072] ;

[0073] in, is the transformation feature of the i-th scale output; i is the scale variable; is the change operation of the i-th scale; is the transformation feature output at the i-1th scale; R is the real number space; is the output feature of the nth submodule in the i-th scale; is the output feature of the n-1th submodule in the i-th scale; is the focused linear attention function; is the hybrid feedforward network function; N is the total number of stacked submodules in each transformer block.

[0074] In some embodiments, the focused linear attention function is:

[0075] ;

[0076] ;

[0077] in, is the output linear attention feature; , , are the query vector, key vector, and value vector used for attention calculation, is the mapping function; represents the element-by-element power operation, and P is a hyperparameter that controls the degree of feature focusing; Represents the similarity calculation after focusing function mapping; represents depthwise convolution; Indicates the transposition operation of K; x represents the function variable, here refers to Q and ;ReLU(x) represents the ReLU activation function; represents the power normalization function; Represents the L2 norm operation.

[0078] The mixed features output by each Transformer Block will be passed to the next stage through overlapping downsampling (Patch Merging), reducing the spatial size and increasing the feature dimension:

[0079] ;

[0080] in, is the attention feature of the downsampled output; For downsampling processing.

[0081] The final encoder outputs multi-scale features:

[0082] ;

[0083] in, These are the first, third, and fourth attention features, respectively.

[0084] The multi-scale color features and their corresponding depth features are input into the cross-modal aggregation module to obtain multi-scale aggregated features. The cross-modal aggregation module is used to aggregate the features output by the two branches. Figure 3 As shown, the cross-modal aggregation module is a cross-focus linear attention feature aggregation module, which is used to aggregate the features of the two modalities in each stage, respectively taking the input visible light features as and the input deep image features of the cross-modal aggregation module As input, each of them splits the query vector of the visible light image and the query vector of the depth image , the key vector of the visible light image and the key vector of the depth image Vector of values ​​with values ​​visible light image and the value vector of the depth image ;then of and 、 of and After multiplication, the global context vector of visible light is generated Global context vector with depth image ; Then enter the cross-modal interaction, and 、 and Multiply them separately to allow the two modal features to integrate each other's attention information; after interaction, the features are stabilized and normalized to retain the original features. Finally, the two processing results are combined to output the feature aggregation result C. The cross-focus linear attention formula is:

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] ;

[0090] in, is the global context vector of visible light; is the key vector of the visible light image used for attention calculation; Calculate for transposition; is the value vector of the visible light image, also used for attention calculation; is the global context vector of the depth image; is the key vector of the depth image; is the value vector of the depth image; Calculate the results of attention for visible light; is the query vector of the visible light image; It is the input visible light feature of the cross-modal aggregation module; The attention calculation results of the depth image; is the query vector of the depth image; is the SoftMax activation function; is the input deep image feature of the cross-modal aggregation module; is the feature aggregation result after the cross-focus linear attention module, including the feature aggregation result , feature aggregation results , feature aggregation results and feature aggregation results ; To merge two features along a specified dimension.

[0091] like Figure 4As shown in the figure, the multi-scale aggregated features are input into the feature fusion module to obtain the fused features; in the upper part, the input features enter the CGA (Collaborative Guidance Attention) module, where the spatial attention focuses on the key spatial areas of the features, and the channel attention mines the important information of the channel dimension. The outputs of the two are fused and merged by addition, and then the pixel attention is used to further refine the details to generate the attention weight W; in the lower part, the new two-way input is initially fused by addition, and then passed into the CGA module. The W and complementary weights (1-W) produced in the upper part are used to weight the two-way branch features respectively. After the weighted results are added and fused, the channel dimension is compressed by a (1x1) size convolution, and the feature fusion result is finally output. The feature fusion module includes fusion module 1, fusion module 2 and fusion module 3 for fusing multi-scale features:

[0092] ;

[0093] ;

[0094] in, Represents the feature fusion results of each stage; It is a convolution operation with a convolution kernel size of 1×1; and They are features from different levels of the top and bottom layers respectively; The weight is calculated after content attention; for pixel attention; Merge two features on the specified dimension; for spatial attention; For channel attention.

[0095] The fused features are input into the decoder to obtain the segmentation result.

[0096] The registered color image and the registered depth image are input into a trained segmentation network for segmentation to obtain a segmentation result; the segmentation result includes a complete leaf and a diseased spot area.

[0097] In some embodiments, the method further includes using a dataset to evaluate the severity of leaf lesions on the segmentation results to obtain an evaluation result; the dataset is a dataset related to leaf lesion images. Specifically, the segmentation results are evaluated for lesion severity using a comprehensive scoring model to obtain an evaluation result. The comprehensive scoring model is a comprehensive evaluation model based on multidimensional features such as lesion type, area, and depth offset value, proposed based on the constructed leaf lesion image dataset and its segmentation results, and aims to achieve quantitative analysis of the degree of leaf pathological damage. The comprehensive scoring model includes:

[0098] A weighted scoring model is constructed based on lesion indicators; the lesion indicators include lesion type, lesion area and depth deviation characteristics.

[0099] The formula for calculating the area of ​​different lesion types is as follows:

[0100] First, count all non-background pixels in the mask (i.e., leaf area, including healthy and diseased spots):

[0101] ;

[0102] in, is a non-background pixel; i and j are the horizontal and vertical pixel variables of the mask respectively; is an indicator function, which is 1 if the condition is met and 0 if it is not met; for mask.

[0103] Secondly, statistics Number of lesion-like pixels:

[0104] ;

[0105] in, is the number of pixels of the kth type of lesion; k is the lesion type variable; The mask type is k.

[0106] Calculate the area ratio of a single type of lesion:

[0107] ;

[0108] in, is the area ratio of the kth type of lesions.

[0109] Calculate the combined proportion of all lesions to the leaf area:

[0110] ;

[0111] in, It is the comprehensive ratio of all lesions to the leaf area.

[0112] To calculate the depth offset, first calculate the average depth of healthy leaves:

[0113] ;

[0114] in, is the average depth of healthy leaves; for The depth value of The mask type is 1.

[0115] Next, calculate the average depth of each type of lesion area:

[0116] ;

[0117] in, is the average depth of the k-th type of lesion area.

[0118] Calculate the depth offset of each type of lesion:

[0119] ;

[0120] in, is the depth offset of the kth type of lesion.

[0121] Calculate the depth shift percentage of each lesion type and the combined depth shift percentage of all lesions:

[0122] ;

[0123] ;

[0124] in, is the depth deviation percentage of the kth type of lesion; is the combined depth shift percentage of all lesions.

[0125] The weighted scoring model is:

[0126] ;

[0127] in, For rating; 、 and They are the first scoring weight, the second scoring weight, and the third scoring weight, which can be determined by the analytic hierarchy process, comparing the importance of features pairwise based on expert experience or literature (scale 1-9); is the variable of lesion type; n is the total number of lesion types; is the type hazard severity parameter (set based on expert experience or literature); For the Number of lesion-like pixels; is the total number of pixels; is the combined proportion of all lesions to the leaf area; is the combined depth shift percentage of all lesions.

[0128] A judgment matrix is ​​constructed based on the importance of the lesion indicators. For example, the judgment matrix is ​​constructed by comparing the three indicators (lesion type, lesion area, and depth deviation) in pairs:

[0129] ;

[0130] in, is the importance of indicator 1 relative to indicator 2; The importance of indicator 1 relative to indicator 3; is the importance of indicator 2 relative to indicator 1; The importance of indicator 2 relative to indicator 3; The importance of indicator 3 relative to indicator 1; The importance of indicator 3 relative to indicator 2;

[0131] Calculating the eigenvectors of the judgment matrix to determine the weights of the weighted scoring model;

[0132] The weighted scoring model with the determined weights is used as the comprehensive scoring model, and the scores are calculated using the comprehensive scoring model, and the scores are graded to obtain the evaluation results. The evaluation results are:

[0133] ;

[0134] Among them, I is mild injury; II is moderate injury; III is severe injury.

[0135] In summary, through key technologies such as RGB-D cross-modal fusion, the Focus Linear Attention mechanism, and three-dimensional lesion assessment, we achieved accurate segmentation of lesion areas on crop leaves and objective assessment of disease severity. RGB-D cross-modal fusion fully combines the texture and color features of visible light RGB images with the spatial morphological characteristics of depth maps, enabling this method to maintain stable and reliable lesion recognition performance in complex field scenarios such as strong illumination changes and leaf occlusion. The Focus Linear Attention mechanism focuses the segmentation network's attention on key lesion feature areas, effectively improving the segmentation accuracy of weakly visible or obstructed lesions and significantly enhancing the model's robustness in real field scenarios. The depth-based three-dimensional lesion assessment method breaks through the limitations of traditional disease assessment that relies solely on two-dimensional images. By comprehensively considering the lesion area ratio and three-dimensional deformation characteristics, it more objectively and accurately quantifies the severity and spatial distribution of the disease. This provides a new technical approach for the monitoring and prevention of agricultural pests and diseases, and can provide real-time and reliable decision-making for smart agriculture applications such as precise variable-rate pesticide application in the field.

[0136] like Figure 5 As shown in the figure, the plant leaf spot segmentation system based on RGB-D cross-modal fusion includes a data processing module, a data acquisition module, a network training module, a network segmentation module and an evaluation module;

[0137] The data acquisition module is configured to acquire color images and depth images of plant leaves;

[0138] The data processing module is configured to register the acquired color image and depth image to obtain a registered color image and a registered depth image in preparation for model training;

[0139] The network training module is configured to learn the registered plant leaf lesion images, thereby obtaining a segmentation network that accurately identifies various lesion types;

[0140] The network segmentation module is configured to accurately segment the diseased spots on the plant leaves to obtain a segmentation result;

[0141] The evaluation module is configured to perform an objective and comprehensive evaluation of the severity of plant leaf spots to obtain an evaluation result.

[0142] It can be understood that the plant leaf lesion segmentation system based on RGB-D cross-modal fusion provided in this embodiment and the plant leaf lesion segmentation method based on RGB-D cross-modal fusion provided in the above embodiment are based on the same concept. For more specific working principles of each module in the embodiment of the present invention, please refer to the above embodiment and will not be repeated in the embodiment of the present invention.

[0143] The above are preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A plant leaf spot segmentation method based on RGB-D cross-modal fusion, characterized in that: include: Using a synchronous acquisition device to acquire color images and depth images of crops; the synchronous acquisition device includes a visible light camera and a depth sensor; Registering the color image and the depth image to obtain a registered color image and a corresponding registered depth image; Constructing an initial segmentation network and performing model training on the initial segmentation network; the initial segmentation network includes a feature aggregation module based on cross attention and an adaptive feature fusion module guided by attention; Inputting the registered color image and the registered depth image into a trained segmentation network for segmentation to obtain a segmentation result; the segmentation result includes the complete leaf and the diseased spot area; The segmentation result is evaluated for severity of leaf spots using a dataset to obtain an evaluation result; the dataset is an image dataset related to leaf spots; the evaluation result includes: Constructing a weighted scoring model based on lesion indicators; the lesion indicators include lesion type, lesion area and depth deviation characteristics; Construct a judgment matrix based on the importance of lesion indicators; Calculating the eigenvectors of the judgment matrix to determine the weights of the weighted scoring model; The weighted scoring model with determined weights is used as the comprehensive scoring model, and the scores are calculated through the comprehensive scoring model, and the scores are graded to obtain the evaluation results.

2. The plant leaf spot segmentation method based on RGB-D cross-modal fusion according to claim 1 is characterized in that: The weighted scoring model is: ； in, For rating; 、 and are the first scoring weight, the second scoring weight, and the third scoring weight, respectively; i is the lesion type variable; n is the total number of lesion types; is the type hazard severity parameter; is the number of pixels of the i-th type of lesion; is the total number of pixels; is the combined proportion of all lesions to the leaf area; is the combined depth shift percentage of all lesions; The judgment matrix is: ； in, is the importance of indicator 1 relative to indicator 2; The importance of indicator 1 relative to indicator 3; is the importance of indicator 2 relative to indicator 1; The importance of indicator 2 relative to indicator 3; The importance of indicator 3 relative to indicator 1; The importance of indicator 3 relative to indicator 2; The evaluation results are: ； Among them, I is mild injury; II is moderate injury; III is severe injury.

3. The plant leaf spot segmentation method based on RGB-D cross-modal fusion according to claim 1, characterized in that: Collecting training samples for the model training includes: A crop canopy acquisition device is used to synchronously acquire color images and depth images of crops; the crop canopy acquisition device integrates a high-resolution visible light camera and a depth sensor; The visible light camera and depth sensor are spatially aligned using a checkerboard calibration method to ensure pixel-level coordinate consistency. The color image is annotated to generate label data and obtain a label image; the label image is used for model training; the label data is created by a crop annotation tool and includes crop part areas, crop diseased spot areas and background areas.

4. The plant leaf spot segmentation method based on RGB-D cross-modal fusion according to claim 1, characterized in that: The initial segmentation network adopts a semantic segmentation framework, including an encoder and a decoder; the encoder includes a dual-branch feature extraction module, a cross-modal aggregation module and a feature fusion module; the dual-branch feature extraction module includes a first extraction branch and a second extraction branch; Input the color image and the depth image into the first extraction branch and the second extraction branch respectively to obtain multi-scale color features and depth features; The multi-scale color features and their corresponding depth features are input into the cross-modal aggregation module to obtain multi-scale aggregated features; Input the multi-scale aggregated features into the feature fusion module to obtain the fused features; The fused features are input into the decoder to obtain the segmentation result.

5. The plant leaf spot segmentation method based on RGB-D cross-modal fusion according to claim 4 is characterized in that: The first extraction branch and the second extraction branch each include a plurality of extraction blocks; each extraction block includes an overlapping block embedding, a transformer, and a downsampling; The overlapping block embedding is used to map the input image into the initial transformed features through an overlapping convolution operation: ； in, is the initial transformation feature; To perform overlapping block operations on the input image; input is the input image; and are the height and width of the input image respectively; is the number of characteristic channels; R is the real number space; The transformer is used to perform transformation operations on the transformation features of each scale, including aggregated linear attention and hybrid feedforward networks: ； Each transformer block in the transformer consists of a stack of multiple submodules: ； ； in, is the transformation feature of the i-th scale output; i is the scale variable; is the change operation of the i-th scale; is the transformation feature output at the i-1th scale; R is the real number space; is the output feature of the nth submodule in the i-th scale; is the output feature of the n-1th submodule in the i-th scale; is the focused linear attention function; is the hybrid feedforward network function; N is the total number of stacked submodules in each transformer block; The downsampling is used to perform overlapping downsampling on the mixed features: ； in, is the attention feature of the downsampled output; For downsampling processing.

6. The plant leaf spot segmentation method based on RGB-D cross-modal fusion according to claim 5, characterized in that: The focused linear attention function is: ； ； in, is the output linear attention feature; , , are the query vector, key vector, and value vector used for attention calculation, is the mapping function; represents the element-by-element power operation, and P is a hyperparameter that controls the degree of feature focusing; Represents the similarity calculation after focusing function mapping; represents depthwise convolution; Indicates the transposition operation of K; x represents the function variable, here refers to Q and ;ReLU(x) represents the ReLU activation function; represents the power normalization function; Represents the L2 norm operation.

7. The plant leaf spot segmentation method based on RGB-D cross-modal fusion according to claim 4, characterized in that: The cross-modal aggregation module is a cross-focused linear attention feature aggregation module, which is used to aggregate the features of the two modalities in each stage. The cross-focused linear attention formula is: ； ； ； ； ； in, is the global context vector of visible light; is the key vector of the visible light image used for attention calculation; Calculate for transposition; is the value vector of the visible light image, also used for attention calculation; is the global context vector of the depth image; is the key vector of the depth image; is the value vector of the depth image; Calculate the results of attention for visible light; is the query vector of the visible light image; It is the input visible light feature of the cross-modal aggregation module; The attention calculation results for the depth image; is the query vector of the depth image; is the SoftMax activation function; is the input deep image feature of the cross-modal aggregation module; It is the feature aggregation result after the cross-focus linear attention module; To merge two features in a specified dimension; The feature fusion module is used to fuse the features of each stage: ； ； in, Represents the feature fusion results of each stage; It is a convolution operation with a convolution kernel size of 1×1; and They are features from different levels of the top and bottom layers respectively; The weight is calculated after content attention; for pixel attention; Merge two features on the specified dimension; for spatial attention; For channel attention.

8. A plant leaf lesion segmentation system using the plant leaf lesion segmentation method based on RGB-D cross-modal fusion according to any one of claims 1 to 7, characterized in that: It includes data processing module, data acquisition module, network training module, network segmentation module and evaluation module; The data acquisition module is configured to acquire color images and depth images of plant leaves; The data processing module is configured to register the acquired color image and depth image to obtain a registered color image and a registered depth image in preparation for model training; The network training module is configured to learn the registered plant leaf spot images to obtain a segmentation network for identifying the type of the spot; The network segmentation module is configured to segment the diseased spots on the plant leaves to obtain a segmentation result; The evaluation module is configured to evaluate the severity of plant leaf spots and obtain an evaluation result; the evaluation result includes: Constructing a weighted scoring model based on lesion indicators; the lesion indicators include lesion type, lesion area and depth deviation characteristics; Construct a judgment matrix based on the importance of lesion indicators; Calculating the eigenvectors of the judgment matrix to determine the weights of the weighted scoring model; The weighted scoring model with determined weights is used as the comprehensive scoring model, and the scores are calculated through the comprehensive scoring model, and the scores are graded to obtain the evaluation results.

Images