A method for detecting grape diseases based on image segmentation and registration fusion

Abstract

The application provides a grape disease detection method based on image segmentation and registration fusion, comprising the following steps: acquiring a grape multispectral image; performing artificial marking on the acquired grape multispectral disease image after registration, segmenting a disease spot image and marking a disease spot category; establishing a grape disease sample set and dividing the grape disease sample set into a training sample set and a test sample set; constructing a semantic segmentation disease detection model based on visible light and near-infrared light; training the constructed grape disease detection model by using the training sample set and test sample set respectively and analyzing model error; directly detecting the acquired actual grape disease image by using the trained grape disease detection model, and segmenting a disease spot image; and fusing the disease spot images under the multispectrum to obtain a final disease image. The grape disease detection precision is improved, grape disease states can be grasped in time, and the method has a wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image registration in agricultural scenarios and the field of segmented image fusion, and in particular to a method for semantic segmentation of aerial images in a vineyard scenario, classifying specific regions, and then obtaining the disease situation of the grapevine through multispectral image fusion. BACKGROUND

[0002] One of the main goals of future agriculture is to increase productivity while ensuring production quality and reducing inputs. Achieving this goal is highly technologically challenging, and intelligent agriculture can bring interesting solutions to vineyard management and reduce the environmental impact. Automatic detection of grapevine diseases can improve the efficiency and flexibility of crop management in vineyards, while reducing chemical inputs. In addition, the use of pesticides is becoming increasingly regulated, so the demand for intelligent agriculture is imminent.

[0003] Current methods for detecting grapevines mainly fall into two categories: expert manual identification and automatic detection. Early grape leaf diseases are mainly detected by experienced grape leaf disease detection experts through manual detection. This expert manual identification method is time-consuming and labor-intensive and cannot achieve automatic disease detection and processing. Automatic detection methods train the detection target in advance using detection algorithms, then select the best model after training to encapsulate the system, and finally use machine vision and sensors for automatic disease detection. These methods can achieve automatic detection, but have problems such as low detection accuracy and long detection time.

[0004] In summary, the accuracy and duration of automatic detection of grapevine leaves are difficult and important in the detection process. The present application aims to draw a map of the diseased area in the vineyard to quickly and accurately process diseased plants and ensure the health of the plants, which is very important for yield management. To solve this problem, the present application discloses a deep learning segmentation method based on unmanned aerial vehicle images for detecting grapevine mildew. SUMMARY

[0005] The present application proposes a segmentation and image registration fusion method based on unmanned aerial vehicle images to achieve disease detection in vineyards, aiming to solve the above-mentioned technical problems. This method combines visible light and infrared images obtained from two different sensors, uses a fully convolutional neural network method to classify each pixel according to different conditions (i.e. shadow, ground, healthy and diseased), and then performs image fusion on the visible light and near-infrared image segmentation results, achieving the fusion of multispectral information from two sensors, which is beneficial to obtaining more accurate grapevine disease areas and achieving disease detection.

[0006] First, the multispectral image data of the vineyard is collected by the unmanned aerial vehicle and transmitted to the server, and the server processes and configures the image data to obtain a training set. Based on the idea of deep learning, a semantic segmentation network model is built and trained, which integrates a double attention mechanism module. Then, multispectral data is collected from the unmanned aerial vehicle, registered and processed to obtain a test set, which is put into the model to obtain a better segmentation effect, and then image fusion is performed to realize disease detection of the vineyard, reduce unnecessary loss, and improve agricultural economic benefits.

[0007] The technical scheme of the present application is a method for disease detection of grapes based on image segmentation and registration fusion, which specifically comprises the following steps:

[0008] Step 1, collect original visible light and near-infrared light image data of the vineyard;

[0009] Step 2, pre-process the original visible light and near-infrared light image data to obtain pre-processed visible light data set and registered near-infrared light data set, and train and validate the two data sets respectively;

[0010] Step 3, construct a semantic segmentation network model, which includes a backbone feature extraction network, a double attention mechanism module, a spatial pyramid pooling module with a hollow convolution, and an up-sampling module;

[0011] Step 4, train and validate the constructed semantic segmentation network model using the training set in the visible light data set and the registered near-infrared light data set respectively;

[0012] Step 5, pre-process the visible light image and near-infrared image to be tested to obtain the corresponding test set;

[0013] Step 6, input the test set into the corresponding spectral semantic segmentation network model to generate respective semantic segmentation results, and use the discrete wavelet transform method to perform segmentation fusion on the respective segmentation maps to obtain the disease detection map of the vineyard.

[0014] Further, the specific implementation of step 2 includes the following sub-steps:

[0015] Step 2.1, image registration is performed on two photos taken by the visible light sensor and the near-infrared light sensor at the same time to obtain a registered near-infrared light image B, and the original visible light image is denoted as image A;

[0016] Step 2.2, label image A using the labelme tool, the label categories of image A include shadow, ground, healthy grapevine and diseased grapevine, each category is assigned a pixel value, the pixel points in the shadow area are assigned a pixel value of 0, the pixel points in the ground area are assigned a pixel value of 1, the pixel points in the healthy grapevine area are assigned a pixel value of 2, and the pixel points in the diseased grapevine area are assigned a pixel value of 3, and the corresponding label image C is generated after the labeling is completed;

[0017] Step 2.3, label image B using the labelme tool, the label categories of image B also include shadow, ground, healthy grapevine and diseased grapevine, each category is assigned a pixel value, the pixel points in the shadow area are assigned a pixel value of 0, the pixel points in the ground area are assigned a pixel value of 1, the pixel points in the healthy grapevine area are assigned a pixel value of 2, and the pixel points in the diseased grapevine area are assigned a pixel value of 3, and the corresponding label image D is generated after the labeling is completed;

[0018] Step 2.4, cut image A, B and the corresponding label images C, D using the sliding window method, and cut them into multiple images of a certain size;

[0019] Step 2.5, traverse the label images C and D, delete images that do not contain labeled areas and images whose labeled areas are greater than P%, and their corresponding original images, so that all images can retain sufficient context information while reducing some redundant information, allowing the network to learn sufficient information;

[0020] Step 2.6, traverse the images C and D processed in step 2.5, calculate the total number of labeled pixels for each category, and then calculate the respective proportion, for images with a certain category label proportion exceeding P1%, randomly delete them, only keep P2% of the image number, to obtain the visible light dataset E and the registered near-infrared light dataset F;

[0021] Step 2.7, divide the processed datasets E and F into training set and validation set according to a certain proportion, and the training set and the validation set have corresponding label images.

[0022] Further, the specific implementation of image registration in step 2.1 is as follows:

[0023] First, extract the green channel G from the visible image and the near-infrared channel NIR from the infrared image; second, normalize the two spectral channels by using the histogram normalization method to improve their contrast; third, use the AKAZE algorithm to extract interest points from the two channels and calculate their features; fourth, based on the features of the interest points, map each interest point extracted in the G channel to the corresponding point in the NIR channel; fifth, remove some outliers.

[0024] Further, the backbone feature extraction network is constructed based on an Xception structure and comprises a convolution layer, an activation layer and a depth separable convolution.

[0025] Further, the dual attention mechanism module comprises a position attention module and a channel attention module.

[0026] The processing procedure of the position attention module is as follows: the feature map A is firstly subjected to three convolution layers to obtain three feature maps B, C and D, then the dimensions of B, C and D are converted, the transpose of the dimension-converted B is multiplied by the dimension-converted C, the position attention feature map S is obtained through a softmax function, then the matrix multiplication is performed between the dimension-converted D and the transpose of S, multiplied by a scale coefficient α, then the dimension is converted to the original shape, and finally added with A to obtain E1.

[0027] The processing procedure of the channel attention module is as follows: the dimension conversion operation and the dimension conversion plus transpose operation are performed on the feature map A to obtain feature Figure 1 and feature Figure 2 The two obtained feature maps are multiplied, and the channel attention feature map X is obtained through a softmax function; then the transpose of X is subjected to the matrix multiplication with feature Figure 1 , multiplied by a scale coefficient β, then the dimension is converted to the original shape, and finally added with A to obtain E2.

[0028] The element sum of E1 and E2 is obtained to obtain the output E, and E is taken as the final output of the dual attention mechanism module and input to the dilated spatial pyramid pooling module.

[0029] Further, the dilated spatial pyramid pooling module is composed of a 1×1 convolution, a 3×3 dilated convolution with an expansion rate of 6, a 3×3 dilated convolution with an expansion rate of 12, a 3×3 dilated convolution with an expansion rate of 16 and a global average pooling, and the extraction of multi-scale features is realized; then a 1×1 convolution is connected after the dilated pyramid module to adjust the channel number, and the feature map containing high-level semantics is obtained.

[0030] Further, the processing procedure of the up-sampling module is that the low-level semantic feature map obtained by the backbone feature extraction network and the high-level semantic feature map obtained by the dilated spatial pyramid pooling module are concatenated, and then a 3x3 convolution is performed to refine the features, thereby improving the segmentation accuracy of different scale regions; finally, bilinear up-sampling is performed to obtain the predicted result and generate a segmentation image.

[0031] Further, in step 4, a multi-classification cross-entropy loss function is used to calculate the loss between the predicted semantic segmentation map and the manually labeled true value semantic segmentation map, and the parameters of the semantic segmentation network model are optimized through a back propagation algorithm, SGD is used as the optimization algorithm, the initial learning rate is 0.01, the minimum learning rate is 0.0001, the batch size is 4, and a total of 200,000 iterations are trained, and the trained model parameters are saved.

[0032] Further, in step 5, the two photos taken by the visible light sensor and the near-infrared light sensor at the same time are first image registered to obtain a registered near-infrared light image B, and the original visible light image is denoted as image A, and then image A and B are cropped to a certain size to obtain test set A and B.

[0033] Further, in step 6, the grapevine disease map obtained by fusion includes three cases.

[0034] The first case is when the pixel is classified as "sick" in the infrared range and as healthy in the visible light range, in which case the disease has not yet affected the visible range through leaf discoloration; the second case is when the pixel is classified as healthy in the infrared range and as "sick" in the visible light range, in which case the symptoms can be distinguished by the naked eye under visible light; the third case is when the pixel is classified as "sick" in both the infrared range and the visible light range, indicating that the grapevine at this location has been detected as diseased under both visible light and near-infrared light, and needs close attention from the user.

[0035] The advantages of the present application mainly embody: the deep learning method is used to realize the semantic segmentation of the multispectral aerial image in the agricultural scene, and the image registration and the segmentation image fusion method are used to realize the detection of the grapevine disease. The method fully utilizes the multispectral remote sensing information in the original image, so that the disease detection of the grapevine is more reliable. In addition, the channel attention module in the double attention mechanism uses the correlation between different channel features to strengthen the correlation between different channel features, and the position attention module simulates the correlation between different local features, so as to promote the classification accuracy between different local features. Considering the actual demand of the method in the agricultural scene, a lightweight network module is used to extract the main features, which is fast and has fewer parameters. Finally, in view of the annotation area imbalance problem between each class in the data set, the data set downsampling method and the focal entropy loss function are used to improve the segmentation performance of the low proportion class. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The flow chart of the real-time semantic segmentation method of the remote sensing image in the agricultural scene is implemented by the present application.

[0037] Figure 2 The network structure diagram of the semantic segmentation is implemented by the present application.

[0038] Figure 3 The data set generation flow chart is implemented by the present application.

[0039] Figure 4 The flow chart of the multispectral image registration is implemented by the present application.

[0040] Figure 5 The attention mechanism structure diagram of the semantic segmentation is implemented by the present application. DETAILED DESCRIPTION

[0041] The present application proposes a segmentation and image registration fusion method based on unmanned aerial vehicle image to realize the disease detection of the vineyard in view of the above-mentioned technical problems. The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application, including the following steps:

[0042] Step 1: the visible light sensor and the infrared light sensor carried by the unmanned aerial vehicle are used to collect image data in the target vineyard area, and the specific image collection path needs to be planned and designed in advance. The unmanned aerial vehicle flies at a height of 25 meters above the plot at an average speed of 10 kilometers / hour, automatically takes a picture every 2 seconds, and uses wireless network to transmit the visible light image and the near infrared image taken by the unmanned aerial vehicle to the server in real time. The server saves the image data;

[0043] Step 2: The server processes the obtained multispectral image data, and then performs image registration on the near-infrared light image and the visible light image to obtain a registered near-infrared light image. The visible light image and the registered near-infrared light image together constitute a multispectral image, thereby obtaining a training set and a verification set required by the network;

[0044] Furthermore, step 2 includes the following sub-steps:

[0045] Step 2.1: Perform image registration on two photos taken by the visible light sensor and the near-infrared light sensor at the same time to obtain a registered near-infrared light image B, and the original visible light image is denoted as image A. During registration, the first step is to extract the green channel (G) from the visible image and the near-infrared channel (NIR) from the infrared image. The second step is to normalize the two spectral channels by using a histogram normalization method to improve their contrast. The third step is to extract interest points from the two channels and calculate their features by using the AKAZE algorithm. The fourth step is to map each interest point extracted in the G channel to the corresponding point in the NIR channel based on the features of the interest points. The fifth step is to remove some outliers.

[0046] Step 2.2: Label image A using the labelme tool. The label categories of image A include shadow, ground, healthy grapevine, and diseased grapevine. These categories are mainly used to determine whether the grapevine is diseased based on the yellow or brown color in the visible light range, which can guide the user's decision-making in the next step. Each category is assigned a pixel value, with pixel points in the shadow region assigned a pixel value of 0, pixel points on the ground assigned a pixel value of 1, pixel points in the healthy grapevine region assigned a pixel value of 2, and pixel points in the diseased grapevine region assigned a pixel value of 3. After labeling is completed, a corresponding label image C is generated;

[0047] Step 2.3: Label image B using the labelme tool. The label categories of image B also include shadow, ground, healthy grapevine, and diseased grapevine. In the near-infrared range, the symptom category is determined by the change in reflectivity between the leaf and its adjacent leaves. From a visual feature perspective, this provides a special texture characterized by a significant change in reflectivity. Each category is assigned a pixel value, with pixel points in the shadow region assigned a pixel value of 0, pixel points on the ground assigned a pixel value of 1, pixel points in the healthy grapevine region assigned a pixel value of 2, and pixel points in the diseased grapevine region assigned a pixel value of 3. After labeling is completed, a corresponding label image D is generated;

[0048] Step 2.4: Cut images A, B and corresponding label images C, D into multiple 512x512 size images using a sliding window method;

[0049] Step 2.5: Traverse the label maps C and D, delete images that do not contain labeled regions and images whose labeled regions are greater than 90%, and their corresponding original images, so that all images can retain sufficient context information while reducing some redundant information, allowing the network to learn sufficient information;

[0050] Step 2.6: Traverse images C and D, calculate the total number of pixels labeled for each class, and then calculate the respective proportions. For images with a class label proportion greater than 30%, randomly delete them, retaining only 80% of the image quantity. Obtain the visible light dataset E and the registered near-infrared light dataset F. Due to the long-tail problem in the dataset, the network can only learn the information of the class with a large proportion, resulting in poor segmentation results for classes with a small proportion. This step is a data resampling method to alleviate the extreme imbalance of classes;

[0051] Step 2.7: Divide the processed datasets E and F into training and validation sets in a 7:3 ratio, respectively. Both the training and validation sets have corresponding label maps.

[0052] Step 3: To address the problems in the grapevine disease detection process, a semantic segmentation network model is constructed. The network model mainly consists of a backbone feature extraction network, a spatial pyramid pooling module with a dilated convolution, and an upsampling module. The network input is a 512x512 image, and the output is a segmentation result map.

[0053] The backbone feature extraction network is based on the Xception structure and contains convolution layers, activation layers, and depthwise separable convolutions. For the input image, the backbone feature extraction network obtains a feature map containing low-level semantic information and a feature map containing high-level semantic information. For an input image of 512x512, the backbone feature extraction network can obtain two effective feature layers. One effective feature layer is the result of compressing the input image height and width twice, containing low-level semantic information. The other effective feature layer is the result of compressing the input image height and width four times, containing high-level semantic information.

[0054] The high-level semantic feature map obtained in the previous step is passed into the double attention mechanism module (DAMM module). This module first performs a 3x3 convolution operation with an expansion rate of 2 on the feature map obtained by the backbone network, then sends it to the channel attention module and the position attention module for processing, and finally performs element summation on the processed feature maps to complete feature fusion. The channel attention module uses the correlation between different channel features to enhance different class features and improve classification accuracy. The position attention module simulates the relationship between different local features, which can promote the classification accuracy of different local features.

[0055] The principle of the positional attention module is as follows: The feature map A has a size of CxHxW, where C is the number of channels in an image; H is the image height, representing the number of pixels in the vertical dimension of the image; and W is the image width, representing the number of pixels in the horizontal dimension of the image. A first passes through three convolutional layers to obtain three feature maps B, C, and D. Then, B, C, and D are reshaped to a size of CxN, where N = HxW. Next, the transpose of the reshaped B (NxC) is multiplied by the reshaped C (CxN), and then softmax is applied to obtain the positional attention feature map S (NxN). Then, matrix multiplication is performed between the reshaped D (CxN) and the transpose of S (NxN), multiplied by a scaling factor α, and then reshaped back to its original shape. Finally, this is added to A to obtain the final output E1.

[0056] The channel attention module works as follows: Reshape and reshape with transpose are performed on feature map A to obtain feature maps of size CxN and NxC. These two feature maps are multiplied together and then passed through softmax to obtain the channel attention feature map X(CxC). Next, the transpose of X (CxC) is multiplied by the CxN feature map, multiplied by a scaling factor β, and then reshaped back to its original shape. Finally, this result is added to A to obtain the final output E2.

[0057] The element-wise summation of E1 and E2 yields the output E. This feature map E is then fed into the dilated spatial pyramid pooling module, which uses dilated convolution to increase the receptive field and better capture the image's contextual information, thereby improving the final segmentation accuracy. The dilated spatial pyramid pooling module consists of five parts: a 1×1 convolution, a 3×3 dilated convolution with a dilation rate of 6, a 3×3 dilated convolution with a dilation rate of 12, a 3×3 dilated convolution with a dilation rate of 16, and global average pooling, enabling multi-scale feature extraction. Finally, a 1×1 convolution is appended to the dilated pyramid module to adjust the number of channels, resulting in a feature map containing high-level semantics.

[0058] The low-level semantic feature map obtained by the backbone feature extraction network and the high-level semantic feature map obtained by the hollow pyramid module are concatted, and then a 3x3 convolution is performed to refine the features, thereby improving the segmentation accuracy of regions at different scales. Finally, bilinear upsampling is performed to obtain the predicted result and generate the segmented image.

[0059] Step 4: Input the processed dataset E under visible light into the model and train the built semantic segmentation network model A; at the same time, input the registered and processed dataset F under near-infrared light into the model and train the built semantic segmentation network model B.

[0060] The loss function is used to calculate the loss between the predicted semantic segmentation map and the ground truth semantic segmentation map annotated by human, and the parameters of the semantic segmentation network model are optimized by the back propagation algorithm. During the training process, the regression problem of the predicted semantic segmentation map can be regarded as a multi-classification problem of background and foreground, and the multi-classification cross-entropy loss function can be used for the multi-classification of each pixel of the predicted semantic segmentation map. Among them, the class weight coefficients of different classes in the training set are calculated first, and then the corresponding weight cross-entropy loss function is established for training. Then the training set and the validation set of the image under visible light and near infrared are respectively input into two network models, wherein SGD is used as the optimization algorithm, the initial learning rate is 0.01, the minimum learning rate is 0.0001, the batch size is 4, and a total of 200,000 iterations are trained, and the trained model parameters are saved.

[0061] Step 5: The unmanned aerial vehicle collects test image data in the actual farm, including visible light images and near-infrared images, and then transmits the images back to the server, and the image under visible light is denoted as A. Repeat step 2.1 to perform image registration to obtain image B. Then, image A and B are cropped to a size of 512x512 to obtain test set A and B;

[0062] Step 6: The test set A and B are input into the corresponding semantic segmentation network model A and B to generate two semantic segmentation result images A and B. Then, the discrete wavelet transform method is used to perform image fusion on the two segmentation result images to obtain the final grapevine disease image. The user analyzes the segmentation result and makes a decision. The fusion of segmentation is performed to obtain a disease image with more robust results. In order to generate the disease image, each pixel of the image in the visible range is compared with the pixel at the same position in the infrared range. Here, three main cases are considered. The first case is when the pixel is classified as "sick" in the infrared range and as healthy in the visible light range. In this case, it is possible that the disease has not yet affected the visible range through leaf discoloration. The second case is when the pixel is classified as healthy in the infrared range and as "sick" in the visible light range, and the symptoms are visible to the naked eye under visible light. The third case is when the pixel is classified as "sick" in the infrared range and in the visible light range, indicating that the grapevine at this position has been detected with disease under both visible light and near-infrared light, and the user needs to pay close attention.

[0063] The specific embodiments described herein are merely illustrative of the present application. Various modifications or supplements can be made to the described specific embodiments or similar ways can be substituted without departing from the spirit of the present application or exceeding the scope defined by the appended claims.

Claims

1. A method for detecting diseases in grapes based on image segmentation and registration fusion, characterized in that, Includes the following steps: Step 1: Acquire raw vineyard image data in visible and near-infrared light; Step 2: Preprocess the original visible light and near-infrared light image data to obtain a preprocessed visible light dataset and a registered near-infrared light dataset, and use the two datasets as training and validation sets respectively. Step 3: Construct a semantic segmentation network model, which includes a backbone feature extraction network, a dual attention mechanism module, a spatial pyramid pooling module with dilated convolution, and an upsampling module. The backbone feature extraction network is built on the Xception structure and includes convolutional layers, activation layers and depthwise separable convolutions. For the input image, the backbone feature extraction network obtains feature maps containing low-level semantic information and feature maps containing higher-level semantic information. The dual attention mechanism module includes a positional attention module and a channel attention module. This module first performs a 3×3 convolution operation with an expansion rate of 2 on the feature map extracted from the backbone network, and then feeds it into the channel attention module and the positional attention module for processing respectively. Finally, the processed feature maps are summed element by element to complete feature fusion. The processing procedure of the position attention module is as follows: Feature map A is first processed by three convolutional layers to obtain three feature maps B, C, and D. Then, B, C, and D are dimensionally transformed. Next, the transpose of the dimensionally transformed B is multiplied with the dimensionally transformed C. Then, the position attention feature map S is obtained by passing it through the softmax function. Then, matrix multiplication is performed between the dimensionally transformed D and the transpose of S. Then, the matrix is ​​multiplied by the scale coefficient α. The dimensions are then transformed back to the original shape. Finally, it is added to A to obtain E1. The processing steps of the channel attention module are as follows: Perform dimensional transformation and dimensional transformation plus transpose operation on feature map A to obtain feature map 1 and feature map 2. Multiply the two feature maps and then use softmax to obtain channel attention feature map X. Next, perform matrix multiplication between the transpose of X and feature map 1, multiply by the scaling factor β, convert the dimensions back to the original shape, and finally add it to A to obtain E2. The elements of E1 and E2 are summed to obtain the output E. E is used as the final output of the dual attention mechanism module and input into the void space pyramid pooling module. The hollow spatial pyramid pooling module consists of five parts: 1×1 convolution, 3×3 dilated convolution with an inflation rate of 6, 3×3 dilated convolution with an inflation rate of 12, 3×3 dilated convolution with an inflation rate of 16, and global average pooling, which enables the extraction of multi-scale features. Then, a 1×1 convolution is appended to the hollow pyramid module, and the number of channels is adjusted to obtain a feature map containing high-level semantics; The upsampling module processes the following steps: it concatenates the low-level semantic feature map obtained from the backbone feature extraction network with the high-level semantic feature map obtained from the dilated spatial pyramid pooling module, then refines the features through a 3x3 convolution to improve the segmentation accuracy of regions at different scales; finally, it performs bilinear upsampling to obtain the predicted result and generate the segmented image. Step 4: Train and validate the constructed semantic segmentation network model using the training sets in the visible light dataset and the registered near-infrared light dataset, respectively. Step 5: Preprocess the visible light and near-infrared images to be tested to obtain the corresponding test set; Step 6: Input the test set into the semantic segmentation network model of the corresponding spectrum to generate their respective semantic segmentation results. Use the discrete wavelet transform method to segment and fuse the respective segmentation maps to obtain the disease detection map of the vineyard.

2. The method for detecting grape diseases based on image segmentation and registration fusion as described in claim 1, characterized in that: Step 2 includes the following sub-steps; Step 2.1: Perform image registration on two photos taken by the visible light sensor and the near-infrared light sensor at the same time to obtain the registered near-infrared light image B, and the original visible light image is denoted as image A; Step 2.2: Use the labelme tool to annotate image A. The annotation categories of image A include four categories: shadow, ground, healthy grapevines, and diseased grapevines. Each category is assigned a pixel value: pixels in the shadow area are assigned pixel value 0, pixels in the ground area are assigned pixel value 1, pixels in the healthy grapevine area are assigned pixel value 2, and pixels in the diseased grapevine area are assigned pixel value 3. After the annotation is completed, the corresponding label image C will be generated. Step 2.3: Use the labelme tool to annotate image B. Image B also includes four categories: shadow, ground, healthy grapevines, and diseased grapevines. Each category is assigned a pixel value: pixels in the shadow area are assigned pixel value 0, pixels in the ground area are assigned pixel value 1, pixels in the healthy grapevine area are assigned pixel value 2, and pixels in the diseased grapevine area are assigned pixel value 3. After the annotation is completed, the corresponding label image D will be generated. Step 2.4: Use a sliding window to cut images A and B and their corresponding label images C and D into multiple images of a certain size; Step 2.5: Traverse label images C and D, delete images that do not contain labeled regions and images with labeled regions greater than P%, as well as their corresponding original images, so that all images can retain sufficient contextual information, while reducing some redundant information, so that the network can learn sufficient information. Step 2.6: Traverse the images C and D after processing in step 2.5, calculate the total number of labeled pixels for each category, and then calculate their respective proportions. For images whose labeled proportion for a certain category exceeds P1%, randomly delete them and keep only P2% of the images to obtain the visible light dataset E and the registered near-infrared light dataset F. Step 2.7: Divide the processed datasets E and F into training and validation sets according to a certain ratio, with corresponding label images for both sets.

3. The method for detecting grape diseases based on image segmentation and registration fusion as described in claim 2, characterized in that: The specific implementation method of image registration in step 2.1 is as follows; The first step is to extract the green channel G from the visible image and the near-infrared channel NIR from the infrared image; the second step is to normalize the two spectral channels by using histogram normalization to improve their contrast. The third step is to use the AKAZE algorithm to extract interest points from the two channels and calculate their features; The fourth step is to map each point of interest extracted from the G channel to the corresponding point in the NIR channel based on the features of the points of interest. The fifth step is to remove some outliers.

4. The method for detecting grape diseases based on image segmentation and registration fusion as described in claim 1, characterized in that: In step 4, the multi-class cross-entropy loss function is used to calculate the loss between the predicted semantic segmentation map and the manually labeled ground truth semantic segmentation map. The parameters of the semantic segmentation network model are optimized by the backpropagation algorithm. SGD is used as the optimization algorithm, with an initial learning rate of 0.01, a minimum learning rate of 0.0001, a batch size of 4, and a total of 200,000 iterations. The trained model parameters are then saved.

5. The method for detecting grape diseases based on image segmentation and registration fusion as described in claim 1, characterized in that: In step 5, the two photos taken by the visible light sensor and the near-infrared light sensor at the same time are first registered to obtain the registered near-infrared light image B. The original visible light image is recorded as image A. Then, images A and B are cropped to a certain size to obtain test sets A and B.

6. The method for detecting grape diseases based on image segmentation and registration fusion as described in claim 1, characterized in that: Step 6 involves fusing the data to obtain a grapevine disease map, which includes three scenarios. The first scenario is when a pixel is classified as "sick" in the infrared range and as healthy in the visible light range. In this case, the disease has not yet affected the visible range through leaf discoloration. The second scenario is when a pixel is classified as healthy in the infrared range and as "sick" in the visible light range. The symptoms are visible to the naked eye in the visible light range. The third scenario is when a pixel is classified as "sick" in both the infrared and visible light ranges. This indicates that the grapevine has been found to have disease in both visible and near-infrared light, requiring close monitoring by the user.

Images