Plant leaf scab segmentation method and system based on RGB-D cross-modal fusion

Abstract

The invention belongs to the field of agricultural information perception, and relates to a plant leaf scab segmentation method and system based on RGB-D cross-modal fusion, and the method comprises the steps: collecting a color image and a depth image of a crop through a synchronous collection device; registering the color image and the depth image to obtain a registered color image and a corresponding registered depth image; constructing an initial segmentation network, and performing model training on the initial segmentation network; the initial segmentation network comprises a cross attention-based feature aggregation module and an attention-guided adaptive feature fusion module; inputting the registered color image and the registered depth image into a trained segmentation network for segmentation to obtain a segmentation result; through heterogeneous data fusion of a depth sensor and a visible light camera, a multi-dimensional feature system covering two-dimensional textures and three-dimensional deformation is constructed; in combination with a cross-modal feature complementation mechanism, the recognition robustness of weakly dominant diseases and insect pests is enhanced, and the segmentation precision in a complex illumination and branch and leaf shielding scene is significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of agricultural information perception, and specifically discloses a method and system for plant leaf disease spot segmentation based on RGB-D cross-modal fusion. Background Art

[0002] There are certain technical bottlenecks in the current field of intelligent detection of agricultural pests and diseases. For example, traditional analysis methods based on visible light images face multiple challenges in complex farmland environments: restricted by the inherent limitations of two-dimensional imaging, the multi-layer occlusion of leaves easily leads to the loss of disease spot feature information, the fluctuation of light conditions causes color feature distortion, and the similarity in texture and chromaticity between healthy tissues and early pest and disease areas further increases the difficulty of accurate identification; in addition, existing technologies lack quantitative analysis of the three-dimensional deformation characteristics of crop surfaces and are difficult to evaluate the spatial distribution trend of pest and disease infestation.

[0003] In view of this, the present invention proposes a method and system for plant leaf disease spot segmentation based on RGB-D cross-modal fusion, introducing a collaborative analysis mechanism for RGB-D multi-modal data. Through the heterogeneous data fusion of a depth sensor and a visible light camera, a multi-dimensional feature system covering two-dimensional texture and three-dimensional deformation is constructed. By using depth information to accurately characterize microscopic deformation features such as leaf wrinkles and disease spot bulges, and combining the cross-modal feature complementarity mechanism to enhance the recognition robustness of weakly dominant pests and diseases, compared with traditional methods, the present invention significantly improves the segmentation accuracy in complex lighting and foliage occlusion scenarios, provides a real-time decision-making basis for precision variable spraying equipment, promotes the transformation of agricultural pest and disease control from empirical judgment to a data-intelligent-driven paradigm, and has important practical value for realizing the reduction of pesticide use, ensuring food safety production, and promoting the construction of a smart agriculture system. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for plant leaf disease spot segmentation based on RGB-D cross-modal fusion, which solves the problems that existing technologies are restricted by the inherent limitations of two-dimensional imaging, the fluctuation of light conditions causes color feature distortion, and the similarity in texture and chromaticity between healthy tissues and early pest and disease areas further increases the difficulty of accurate identification; existing technologies lack quantitative analysis of the three-dimensional deformation characteristics of crop surfaces and are difficult to evaluate the spatial distribution trend of pest and disease infestation, and realizes the improvement of the segmentation accuracy in complex lighting and foliage occlusion scenarios, provides a real-time decision-making basis for precision variable spraying equipment, promotes the transformation of agricultural pest and disease control from empirical judgment to a data-intelligent-driven paradigm, and has important practical value for realizing the reduction of pesticide use, ensuring food safety production, and promoting the construction of a smart agriculture system. The specific solutions are as follows: A method for segmenting plant leaf lesions based on RGB-D cross-modal fusion, comprising: collecting a color image and a depth image of a crop using a synchronous acquisition device; the synchronous acquisition device includes a visible light camera and a depth sensor; registering the color image and the depth image to obtain a registered color image and its corresponding registered depth image; constructing an initial segmentation network and training the model of the initial segmentation network; the initial segmentation network includes a feature aggregation module based on cross-attention and an adaptive feature fusion module guided by attention; inputting the registered color image and the registered depth image into the trained segmentation network for segmentation to obtain a segmentation result; the segmentation result includes a complete leaf and a lesion area.

[0005] Further, it also includes evaluating the severity of leaf lesions in the segmentation result using a data set to obtain an evaluation result; the data set is an image data set related to leaf lesions.

[0006] Further, evaluating the severity of lesions in the segmentation result through a comprehensive scoring model to obtain an evaluation result, including: constructing a weighted scoring model based on lesion indicators; the lesion indicators include lesion type, lesion area, and depth offset feature; constructing a judgment matrix based on the importance degree among the lesion indicators; calculating the eigenvector of the judgment matrix to determine the weight of the weighted scoring model; taking the weighted scoring model with the determined weight as the comprehensive scoring model, and calculating a score through the comprehensive scoring model, and grading the score to obtain the evaluation result.

[0007] Further, the weighted scoring model is: ; where is the score; 、 and are the first scoring weight, the second scoring weight, and the third scoring weight respectively; i is the lesion type variable; n is the total number of lesion types; is the type hazard severity parameter (set according to expert experience or literature); is the number of pixels of the i-th type of lesion; is the total number of pixels; is the comprehensive proportion of all lesions in the leaf area; is the comprehensive depth offset percentage of all lesions; The judgment matrix is: ; where is the importance of index 1 relative to index 2; is the importance of index 1 relative to index 3; is the importance of index 2 relative to index 1; is the importance of Index 2 relative to Index 3; is the importance of Index 3 relative to Index 1; is the importance of Index 3 relative to Index 2; The evaluation result is: ; where I is mild injury; II is moderate injury; III is severe injury.

[0008] Further, training samples for the model training are collected, including: using a crop canopy acquisition device to synchronously acquire color images and depth images of the crop; the crop canopy acquisition device integrates a high-resolution visible light camera and a depth sensor; spatially aligning the visible light camera and the depth sensor through a checkerboard calibration method to ensure pixel-level coordinate consistency; annotating the color images to create label data and obtain a label image; the label image is used for model training; the label data is created through a crop annotation tool and includes a crop part area, a crop disease spot area, and a background area.

[0009] Further, the initial segmentation network adopts a semantic segmentation framework, including an encoder and a decoder; the encoder includes a dual-branch feature extraction module, a cross-modal aggregation module, and a feature fusion module; the dual-branch feature extraction module includes a first extraction branch and a second extraction branch; the color image and the depth image are respectively input into the first extraction branch and the second extraction branch to obtain multi-scale color features and depth features; the multi-scale color features and their corresponding depth features are respectively input into the cross-modal aggregation module to obtain multi-scale aggregated features; the multi-scale aggregated features are input into the feature fusion module to obtain fused features; the fused features are input into the decoder to obtain a segmentation result.

[0010] Further, both the first extraction branch and the second extraction branch include multiple extraction blocks; each extraction block includes an overlapping block embedding, a transformer, and downsampling; The overlapping block embedding is used to map the input image into initial transformed features through a convolutional operation with overlap: ; where is the initial transformed feature; is the block operation with overlap on the input image; input is the input image; and are respectively the height and width of the input image; is the number of feature channels; R is the real number space; The transformer is used to perform transformation operations on the transformed features at each scale, including aggregated linear attention and a hybrid feed-forward network: ; Each transformer block in the transformer includes a stack of multiple sub-modules: ; ; wherein, is the transformed feature output at the i-th scale; i is the scale variable; is the transformation operation at the i-th scale; is the transformed feature output at the (i - 1)-th scale; R is the real number space; is the output feature of the n-th sub-module in the i-th scale; is the output feature of the (n - 1)-th sub-module in the i-th scale; is the focused linear attention function; is the hybrid feed-forward network function; N is the total number of sub-modules stacked in each transformer block; The downsampling is used to perform overlapping downsampling on the hybrid features: ; wherein, is the attention feature output by the downsampling; is the downsampling process.

[0011] Furthermore, the focused linear attention function is: ; ; wherein, is the output linear attention feature; , , are the query vector, key vector, and value vector for attention calculation, is the mapping function; represents element-wise power operation, P is the hyperparameter controlling the degree of feature focusing; represents the similarity calculation after being mapped by the focusing function; represents depth convolution; represents the transpose operation on K; x represents the function independent variable, here referring to Q and ; ReLU(x) represents the ReLU activation function; represents the power normalization function; represents the L2 norm operation.

[0012] Furthermore, the cross-modal aggregation module is a cross-focused linear attention feature aggregation module, which is used to aggregate the features of two modalities within each stage. The cross-focused linear attention formula is: ; ; ; ; ; Among them, is the global context vector of visible light; is the key vector of the visible light image for attention calculation; is the transpose calculation; is the value vector of the visible light image, also used for attention calculation; is the global context vector of the depth image; is the key vector of the depth image; is the value vector of the depth image; is the attention calculation result of visible light; is the query vector of the visible light image; is the input visible light feature of the cross-modal aggregation module; is the attention calculation result of the depth image; is the query vector of the depth image; is the SoftMax activation function; is the input depth image feature of the cross-modal aggregation module; is the feature aggregation result after passing through the cross-focus linear attention module; is to merge the two features in the specified dimension; The feature fusion module is used to fuse the features of each stage: ; ; Among them, represents the feature fusion result of each stage; is the convolution operation with a convolution kernel size of 1×1; and are the features of different levels from the high layer and the low layer respectively; is the weight obtained through content attention calculation; is the pixel attention; is to merge the two features in the specified dimension; is the spatial attention; is the channel attention.

[0013] The object of the present invention also lies in providing a plant leaf lesion segmentation system based on RGB-D cross-modal fusion, including a data processing module, a data acquisition module, a network training module, a network segmentation module, and an evaluation module; the data acquisition module is configured to obtain a color image and a depth image of a plant leaf; the data processing module is configured to register the obtained color image and depth image to obtain a registered color image and a registered depth image, preparing for model training; the network training module is configured to learn the registered plant leaf lesion image to obtain a segmentation network for identifying lesion types; the network segmentation module is configured to segment the lesions of the plant leaf to obtain a segmentation result; the evaluation module is configured to evaluate the severity of the plant leaf lesions to obtain an evaluation result.

[0014] The technical solution of the embodiment of the present invention has at least the following advantages and beneficial effects: The present invention uses the optimized Segformer encoder part as the encoder to construct a semantic segmentation model. Compared with the original Segformer encoder, it has a similar or even lower computational complexity, but has stronger expressive ability, making the attention distribution more focused when the model processes features, and significantly improving the ability to focus on important features.

[0015] The present invention designs a cross-aggregated linear attention to fuse the features of the two modalities within each stage of the encoder, enabling the features of the two modalities to make full use of the complementary features of the two modalities during feature fusion, achieving full interaction of features. The shallow cross-attention aggregation module enhances the fusion of texture detail features, and the deep cross-aggregated linear attention aggregation module optimizes the target structure consistency while avoiding feature conflicts caused by direct splicing. The introduced feature fusion module enables the model to focus more on important regions and channels, thereby improving the effect of feature fusion and the overall performance of the model. This feature fusion method, by introducing an attention mechanism, enables the model to not only simply stack features, but to selectively fuse them according to the information content and importance of the features themselves. This not only reduces unnecessary information redundancy, but also enhances the model's ability to capture key information, thereby improving the model's processing ability for complex data while maintaining computational efficiency.

[0016] The lesion severity evaluation system constructed by the present invention in combination with lesion type, lesion area, and depth offset effectively solves the technical problem that it is difficult for a monocular vision system to accurately evaluate the three-dimensional deformation of lesions under the condition of similar color textures. This multi-modal fusion evaluation method provides a new technical paradigm for disease diagnosis in precision agriculture. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic flowchart of a plant leaf lesion segmentation method based on RGB-D cross-modal fusion provided by an embodiment of the present invention; Figure 2 It is a schematic structural diagram of the segmentation network in the embodiment of the present invention; Figure 3 It is a schematic structural diagram of the cross-focus linear attention feature aggregation module provided by the embodiment of the present invention; Figure 4 It is a schematic structural diagram of the attention-guided feature fusion module provided by the embodiment of the present invention; Figure 5 It is a module schematic diagram of the plant leaf disease spot segmentation system based on RGB-D cross-modal fusion provided by the embodiment of the present invention. Detailed implementation manners

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and illustrated in the drawings here can be arranged and designed in various different configurations.

[0019] As Figure 1 shown, the plant leaf disease spot segmentation method based on RGB-D cross-modal fusion includes the following: Use a synchronous acquisition device to collect the color image and depth image of the crop; the synchronous acquisition device includes a visible light camera and a depth sensor. The synchronous acquisition device is a customized crop canopy acquisition device for synchronously obtaining RGB color images and depth images at a fixed height and angle in a farmland scene. The visible light camera is a camera for obtaining color images. For example, the visible light camera can include a dedicated RGB-D camera or a high-resolution RGB camera. The depth sensor is an image acquisition device that can obtain the depth information of the image. For example, a ToF or binocular camera.

[0020] Register the color image and the depth image to obtain a registered color image and its corresponding registered depth image. The registered color image and the registered depth image are images obtained after the color image and the depth image are adjusted and registered with each other. For example, after a dedicated RGB-D camera collects an image, the spatial coordinates of the RGB image and the depth image are aligned to ensure that the color information and the depth value of each pixel correspond one by one.

[0021] Construct an initial segmentation network and perform model training on the initial segmentation network; the initial segmentation network includes a feature aggregation module based on cross-attention and an attention-guided adaptive feature fusion module, which are used to segment the complete leaf and lesion regions. Model training can be to train the segmentation network using the collected RGB and depth image datasets to obtain a trained segmentation network. For example, the registered RGB and depth images of plant leaves are used as inputs, and the trained segmentation network is input for segmentation to obtain a segmentation result.

[0022] In some embodiments, training samples for the model training are collected, including: using a crop canopy acquisition device to synchronously acquire color images and depth images of crops; the crop canopy acquisition device integrates a high-resolution visible light camera and a depth sensor. The spatial alignment of the visible light camera and the depth sensor is calibrated by the checkerboard calibration method to ensure pixel-level coordinate consistency. The RGB color image is annotated to produce label data to obtain a label image; the label image is used for model training; the label data is created by a crop annotation tool, including crop part regions, crop lesion regions, and background regions. A label image refers to a color image after annotating the crop part regions, crop lesion regions, and background regions in the image. A crop annotation tool refers to a professional tool for identifying and annotating the pest and disease conditions of crops. For example, the RGB image of a plant leaf is used to create label categories (such as leaf, disease_type1, disease_type2, background) using a professional annotation tool, and a color is assigned to each category (such as green - leaf, red - lesion A, yellow - lesion B).

[0023] As Figure 2 shown, the encoder part of the optimized semantic segmentation framework (SegFormer) is used as the encoder part of the segmentation network for the initial segmentation network. The overall framework of semantic segmentation includes an encoder and a decoder; specifically, the attention mechanism in the semantic segmentation framework is replaced with Focus Linear Attention, so that the attention distribution is more focused when the model processes features, focusing on important features and improving the segmentation performance. The encoder includes a dual-branch feature extraction module, a cross-modal aggregation module, and a feature fusion module; the dual-branch feature extraction module includes a first extraction branch and a second extraction branch; both the first extraction branch and the second extraction branch blocks include modules 1, 2, 3, and 4 for feature extraction. The cross-modal aggregation module includes aggregations 1, 2, 3, and 4 for cross-modal aggregation. The dual-branch feature extraction module is used to perform cross-modal feature extraction on RGB-D images through two-way feature extraction operations.

[0024] The color image and the depth image are respectively input into the first extraction branch and the second extraction branch to obtain multi-scale color features and depth features. The first extraction branch and the second extraction branch each include 4-scale feature extraction modules, which are respectively used to extract attention features of different scales. In some embodiments, both the first extraction branch and the second extraction branch include multiple extraction blocks; each extraction block includes overlapping patch embedding, a transformer, and downsampling.

[0025] The overlapping patch embedding is used to map the input image into initial transformed features through a convolutional operation with overlap: ; where is the initial transformed feature; is the overlapping block operation on the input image; input is the input image; and are respectively the height and width of the input image; is the number of feature channels; R is the real number space; The transformer is used to perform a transformation (Transformer) operation on the transformed features of each scale, including Focus Linear Attention and Mix-FFN: ; Each transformer block in the transformer includes a stack of multiple (such as N) sub-modules: ; ; where is the transformed feature output at the i-th scale; i is the scale variable; is the transformation operation at the i-th scale; is the transformed feature output at the (i - 1)-th scale; R is the real number space; is the output feature of the n-th sub-module in the i-th scale; is the output feature of the (n - 1)-th sub-module in the i-th scale; is the Focus Linear Attention function; is the Mix-FFN function; N is the total number of sub-modules stacked in each transformer block.

[0026] In some embodiments, the Focus Linear Attention function is: ; ; where is the output linear attention feature; , , are the query vector, key vector, and value vector for attention calculation, is the mapping function; represents element-wise power operation, and P is the hyperparameter controlling the degree of feature focusing; represents the similarity calculation after being mapped by the focusing function; represents depth convolution; represents the transpose operation on K; x represents the independent variable of the function, here referring to Q and ; ReLU(x) represents the ReLU activation function; represents the power normalization function; represents the L2 norm operation.

[0027] The mixed features output by each Transformer Block will undergo overlapping downsampling (Patch Merging) to the next stage, reducing the spatial size and increasing the feature dimension: ; Among them, is the attention feature output by downsampling; is the downsampling process.

[0028] The final encoder outputs multi-scale features: ; Among them, are the first, third, and fourth attention features respectively.

[0029] The multi-scale color features and their corresponding depth features are respectively input into the cross-modal aggregation module to obtain multi-scale aggregated features. The cross-modal aggregation module is used to aggregate the features output by the two branches. As Figure 3 shown, the cross-modal aggregation module is a cross-focus linear attention feature aggregation module, which is used to aggregate the features of the two modalities within each stage, respectively taking the input visible light feature and the input depth image feature of the cross-modal aggregation module as inputs. Both of them split out the query vector of the visible light image and the query vector of the depth image, the key vector of the visible light image and the key vector of the depth image and the value vector of the visible light image and the value vector of the depth image; then of and Generate the global context vector of visible light through multiplication operation With the global context vector of the depth image ; Subsequently, enter cross-modal interaction and 、 and Multiply respectively to make the two-modal features fuse each other's attention information; After the interaction, the features are stabilized by addition & normalization training, and the original features are retained. Finally, the two-way processing results are merged to output the feature aggregation result C. The cross-focus linear attention formula is: ; ; ; ; ; Among them, Is the global context vector of visible light; Is the key vector of the visible light image for attention calculation; Is the transpose calculation; Is the value vector of the visible light image, also used for attention calculation; Is the global context vector of the depth image; Is the key vector of the depth image; Is the value vector of the depth image; Is the attention calculation result of visible light; Is the query vector of the visible light image; Is the input visible light feature of the cross-modal aggregation module; Is the attention calculation result of the depth image; Is the query vector of the depth image; Is the SoftMax activation function; Is the input depth image feature of the cross-modal aggregation module; Is the feature aggregation result after passing through the cross-focus linear attention module, including the feature aggregation result 、the feature aggregation result 、the feature aggregation result and the feature aggregation result ; Is to merge the two features in the specified dimension.

[0030] Such as Figure 4As shown, the multi-scale aggregated features are input into the feature fusion module to obtain the fused features. In the upper part, the input features enter the CGA (Collaborative Guidance Attention) module. First, the spatial attention focuses on the key spatial regions of the features, and the channel attention mines the important information in the channel dimension. After the outputs of the two are fused and combined by addition, the pixel attention is further used to refine the details to generate the attention weight W. In the lower part, the two new inputs are first fused preliminarily by addition and then fed into the CGA module. The W and complementary weight (1 - W) produced in the upper part are used to weight and regulate the two-branch features respectively. After the weighted results are added and fused, the channel dimension is compressed by a convolution with a size of (1x1), and finally the feature fusion result is output. The feature fusion module includes Fusion Module 1, Fusion Module 2, and Fusion Module 3 for fusing multi-scale features: ; ; Among them, represents the feature fusion results at each stage; is a convolution operation with a convolution kernel size of 1×1; and are features from different levels of the high layer and the low layer respectively; is the weight obtained through content attention calculation; is the pixel attention; is the merging of two features in the specified dimension; is the spatial attention; is the channel attention.

[0031] The fused features are input into the decoder to obtain the segmentation result.

[0032] The registered color image and the registered depth image are input into the trained segmentation network for segmentation to obtain the segmentation result; the segmentation result includes the complete leaf and the lesion area.

[0033] In some embodiments, it further includes evaluating the severity of the leaf lesions for the segmentation result by using a data set; the data set is a data set related to leaf lesion images. Specifically, the severity of the lesions is evaluated for the segmentation result through a comprehensive scoring model, and the comprehensive scoring model is a comprehensive evaluation model based on multi-dimensional features such as lesion type, area, and depth offset value proposed based on the constructed leaf lesion image data set and its segmentation result, aiming to achieve a quantitative analysis of the degree of leaf pathological damage. The comprehensive scoring model includes: Construct a weighted scoring model based on lesion indicators; the lesion indicators include lesion type, lesion area, and depth offset characteristics.

[0034] The area formula for different types of lesions is as follows: First, count all non-background pixels in the mask (i.e., the leaf area, including healthy and diseased areas): ; Among them, is a non-background pixel; i and j are the horizontal and vertical pixel variables of the mask respectively; is an indicator function, which is 1 when the condition is met and 0 when not met; is 's mask.

[0035] Secondly, count the number of pixels of the rd type of lesion: ; Among them, is the number of pixels of the kth type of lesion; k is the lesion type variable; is the mask type k.

[0036] Calculate the area ratio of a single type of lesion: ; Among them, is the area ratio of the kth type of lesion.

[0037] Calculate the comprehensive ratio of all lesions to the leaf area: ; Among them, is the comprehensive ratio of all lesions to the leaf area.

[0038] Calculate the depth offset. First, calculate the average depth of the healthy leaf: ; Among them, is the average depth of the healthy leaf; is 's depth value; is the mask type 1.

[0039] Secondly, calculate the average depth of each type of lesion area: ; Among them, is the average depth of the kth type of lesion area.

[0040] Calculate the depth offset of each type of lesion: ; Among them, is the depth offset of the k-th type of lesion.

[0041] Calculate the depth offset percentage of each type of lesion and the comprehensive depth offset percentage of all lesions: ; ; Among them, is the depth offset percentage of the k-th type of lesion; is the comprehensive depth offset percentage of all lesions.

[0042] The weighted scoring model is: ; Among them, is the score; , and are the first scoring weight, the second scoring weight and the third scoring weight respectively, which can be determined by the analytic hierarchy process, comparing the importance of features pairwise according to expert experience or literature (scale 1 - 9); is the lesion type variable; n is the total number of lesion types; is the type hazard severity parameter (set according to expert experience or literature); is the number of pixels of the k-th type of lesion; is the total number of pixels; is the comprehensive proportion of all lesions in the leaf area; is the comprehensive depth offset percentage of all lesions.

[0043] Construct a judgment matrix based on the importance among lesion indicators. For example, pairwise comparison of three indicators (lesion type, lesion area and depth offset) to construct the judgment matrix as: ; Among them, is the importance of indicator 1 relative to indicator 2; is the importance of indicator 1 relative to indicator 3; is the importance of indicator 2 relative to indicator 1; is the importance of indicator 2 relative to indicator 3; is the importance of indicator 3 relative to indicator 1; is the importance of indicator 3 relative to indicator 2; Calculate the eigenvector of the judgment matrix to determine the weights of the weighted scoring model; The weighted scoring model with determined weights is used as the comprehensive scoring model, and the score is calculated through the comprehensive scoring model, and the score is graded to obtain the evaluation result. The evaluation result is: ; where I is mild damage; II is moderate damage; III is severe damage.

[0044] In summary, through key technologies such as RGB-D cross-modal fusion, Focus Linear Attention mechanism, and three-dimensional evaluation of disease spots, the accurate segmentation of the disease spot area of crop leaves and the objective evaluation of the disease severity are realized. RGB-D cross-modal fusion fully combines the texture and color features of visible light RGB images with the spatial morphological features of depth maps, enabling this method to maintain stable and reliable disease spot recognition performance in complex field scenarios such as strong light changes and leaf occlusion; the Focus Linear Attention mechanism makes the attention of the segmentation network more focused on the key disease spot feature areas, effectively improving the segmentation accuracy of weakly dominant or interfered disease spots, and significantly enhancing the robustness of the model in real field scenarios; the three-dimensional evaluation method of disease spots based on depth information breaks through the limitation of traditional disease assessment relying only on two-dimensional images, and more objectively and accurately quantifies the disease severity and its spatial distribution trend by comprehensively considering the disease spot area ratio and three-dimensional deformation features. It provides a new technical means for the monitoring and control of agricultural pests and diseases, and can provide real-time and reliable decision-making basis for intelligent agricultural applications such as precise variable spraying in the field.

[0045] As Figure 5 shown, the plant leaf disease spot segmentation system based on RGB-D cross-modal fusion includes a data processing module, a data acquisition module, a network training module, a network segmentation module, and an evaluation module; The data acquisition module is configured to acquire the color image and depth image of the plant leaf; The data processing module is configured to register the acquired color image and depth image to obtain the registered color image and registered depth image, preparing for model training; The network training module is configured to learn the registered plant leaf disease spot image to obtain a segmentation network that can accurately identify various disease spot types; The network segmentation module is configured to accurately segment the disease spots of the plant leaf to obtain the segmentation result; The evaluation module is configured to objectively and comprehensively evaluate the severity of the plant leaf disease spots to obtain the evaluation result.

[0046] It can be understood that the plant leaf lesion segmentation system based on RGB-D cross-modal fusion provided in this embodiment and the plant leaf lesion segmentation method based on RGB-D cross-modal fusion provided in the above embodiment are based on the same concept. For the more specific working principles of each module in the embodiments of the present invention, refer to the above embodiments and will not be elaborated in the embodiments of the present invention.

[0047] The above are the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for segmenting plant leaf lesions based on RGB-D cross-modal fusion, characterized in that Including: Collecting color images and depth images of crops by using a synchronous acquisition device; the synchronous acquisition device includes a visible light camera and a depth sensor; Registering the color image and the depth image to obtain a registered color image and its corresponding registered depth image; Constructing an initial segmentation network and training the model of the initial segmentation network; the initial segmentation network includes a feature aggregation module based on cross-attention and an adaptive feature fusion module guided by attention; Inputting the registered color image and the registered depth image into the trained segmentation network for segmentation to obtain a segmentation result; the segmentation result includes complete leaves and lesion areas.

2. The method for segmenting plant leaf lesions based on RGB-D cross-modal fusion according to claim 1, wherein It also includes evaluating the severity of leaf lesions of the segmentation result by using a data set to obtain an evaluation result; the data set is an image data set related to leaf lesions.

3. The method for segmenting plant leaf disease spots based on RGB-D cross-modal fusion according to claim 2, wherein Evaluating the severity of lesions of the segmentation result through a comprehensive scoring model to obtain an evaluation result, including: Constructing a weighted scoring model based on lesion indicators; the lesion indicators include lesion type, lesion area, and depth offset feature; Constructing a judgment matrix based on the importance degree among lesion indicators; Calculating the eigenvector of the judgment matrix to determine the weight of the weighted scoring model; Taking the weighted scoring model with determined weight as the comprehensive scoring model, and calculating a score through the comprehensive scoring model, and grading the score to obtain an evaluation result.

4. The method for segmenting plant leaf lesions based on RGB-D cross-modal fusion according to claim 3, characterized in that, The weighted scoring model is: ； wherein, is the score; , and are the first scoring weight, the second scoring weight and the third scoring weight respectively; i is the lesion type variable; n is the total number of lesion types; is the type hazard severity parameter; is the number of pixels of the i-th type of lesion; is the total number of pixels; is the comprehensive ratio of all lesions to the leaf area; is the comprehensive depth offset percentage of all lesions; The judgment matrix is: ； Among them, is the importance of Index 1 relative to Index 2; is the importance of Index 1 relative to Index 3; is the importance of Index 2 relative to Index 1; is the importance of Index 2 relative to Index 3; is the importance of Index 3 relative to Index 1; is the importance of Index 3 relative to Index 2; The evaluation result is: ； Wherein, I is mild damage; II is moderate damage; III is severe damage.

5. The method for segmenting plant leaf disease spots based on RGB-D cross-modal fusion according to claim 1, wherein, Collecting training samples for the model training, including: Using a crop canopy acquisition device to synchronously obtain color images and depth images of crops; the crop canopy acquisition device integrates a high-resolution visible light camera and a depth sensor; Aligning the space of the visible light camera and the depth sensor through the checkerboard calibration method to ensure pixel-level coordinate consistency; Annotating the color image to make label data to obtain a label image; the label image is used for model training; the label data is created by a crop annotation tool and includes crop part areas, crop lesion areas, and background areas.

6. The method for segmenting plant leaf lesions based on RGB-D cross-modal fusion according to claim 1, wherein The initial segmentation network adopts a semantic segmentation framework, including an encoder and a decoder; the encoder includes a double-branch feature extraction module, a cross-modal aggregation module, and a feature fusion module; the double-branch feature extraction module includes a first extraction branch and a second extraction branch; Inputting the color image and the depth image into the first extraction branch and the second extraction branch respectively to obtain multi-scale color features and depth features; Inputting the multi-scale color features and their corresponding depth features into the cross-modal aggregation module respectively to obtain multi-scale aggregated features; Inputting the multi-scale aggregated features into the feature fusion module to obtain fused features; Inputting the fused features into the decoder to obtain a segmentation result.

7. The method for segmenting plant leaf lesions based on RGB-D cross-modal fusion according to claim 6, wherein Both the first extraction branch and the second extraction branch include a plurality of extraction blocks; each extraction block includes overlapping block embedding, a transformer, and downsampling; The overlapping block embedding is used to map the input image into initial transformation features through a convolution operation with overlap: ； Among them, is the initial transformation feature; is the operation of dividing the input image into blocks with overlap; input is the input image; and are the height and width of the input image respectively; is the number of feature channels; R is the real number space; The converter is used to perform transformation operations on the transformed features of each scale, including aggregated linear attention and a hybrid feed-forward network: ； Each transformer block in the transformer consists of a stack of multiple sub-modules: ； ； Among them, is the transformed feature output at the i-th scale; i is the scale variable; is the change operation at the i-th scale; is the transformed feature output at the (i - 1)-th scale; R is the real number space; is the output feature of the n-th sub-module in the i-th scale; is the output feature of the (n - 1)-th sub-module in the i-th scale; is the focused linear attention function; is the hybrid feed-forward network function; N is the total number of sub-modules stacked in each transformer block; The downsampling is used to perform overlapping downsampling on the hybrid features: ； Among them, is the attention feature output by downsampling; is the downsampling process.

8. The method for segmenting plant leaf disease spots based on RGB-D cross-modal fusion according to claim 7, wherein, The focused linear attention function is: ； ； Among them, is the output linear attention feature; , , are the query vector, key vector, and value vector used for attention calculation, is the mapping function; represents element-wise power operation, and P is a hyperparameter that controls the degree of feature focusing; represents the similarity calculation after being mapped by the focusing function; represents depth convolution; represents the transpose operation on K; x represents the independent variable of the function, which here refers to Q and ; ReLU(x) represents the ReLU activation function; represents the power normalization function; represents the L2 norm operation.

9. The method for segmenting plant leaf lesions based on RGB-D cross-modal fusion according to claim 6, wherein The cross-modal aggregation module is a cross-focused linear attention feature aggregation module, which is used to aggregate the features of two modalities within each stage. The cross-focused linear attention formula is: ； ； ； ； ； Among them, is the global context vector of visible light; is the key vector of the visible light image for attention calculation; is the transpose calculation; is the value vector of the visible light image, also used for attention calculation; is the global context vector of the depth image; is the key vector of the depth image; is the value vector of the depth image; is the attention calculation result of visible light; is the query vector of the visible light image; is the input visible light feature of the cross-modal aggregation module; is the attention calculation result of the depth image; is the query vector of the depth image; is the SoftMax activation function; is the input depth image feature of the cross-modal aggregation module; is the feature aggregation result after passing through the cross-focus linear attention module; is to merge the two features in the specified dimension; The feature fusion module is used to fuse the features of each stage: ； ； Among them, represents the feature fusion result of each stage; is a convolution operation with a convolution kernel size of 1×1; and are features of different levels from the high layer and the low layer respectively; is the weight calculated through content attention; is pixel attention; is the merging of two features in a specified dimension; is spatial attention; is channel attention.

10. A plant leaf lesion segmentation system using the plant leaf lesion segmentation method based on RGB-D cross-modal fusion according to any one of claims 1-9, characterized in that, It includes a data processing module, a data acquisition module, a network training module, a network segmentation module, and an evaluation module; The data acquisition module is configured to obtain the color image and depth image of the plant leaf; The data processing module is configured to register the obtained color image and depth image to obtain the registered color image and registered depth image, so as to prepare for model training; The network training module is configured to learn the registered plant leaf lesion image, so as to obtain a segmentation network for identifying the lesion type; The network segmentation module is configured to perform lesion segmentation on the plant leaf to obtain a segmentation result; The evaluation module is configured to evaluate the severity of the plant leaf lesion to obtain an evaluation result.

Images