A method and system for identifying termite indicators using optical and infrared drone data

Abstract

The invention discloses a method and system for identifying termite indicators based on optical and infrared unmanned aerial vehicle data, comprising the following steps: planning a flight path of an unmanned aerial vehicle and setting shooting parameters for the unmanned aerial vehicle to take pictures, taking pictures to obtain infrared images and optical images; acquiring a termite indicator image in an actual area, performing image enhancement processing, and vector calibration thereof to obtain a classified and calibrated termite indicator image; inputting the infrared image and the optical image and the classified and calibrated termite indicator image into a MaskRCNN image segmentation and recognition model for training to obtain a trained MaskRCNN image segmentation and recognition model; acquiring a newly shot image of a study area and inputting the trained MaskRCNN image segmentation and recognition model for recognition to obtain an infrared image and an optical image with the outline and category of a termite indicator marked, and calculating the damage of termites in the study area according to the area of ​​the outline of the termite indicator; the method improves the accuracy and speed of identifying termite indicators in the study area.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent management of water conservancy projects, and in particular to a method and system for identifying termite indicators using optical and infrared drone data. Background Art

[0002] With the climate change in recent years, the damage caused by termites in reservoirs and embankments with earth and stone as the main dam building materials has become increasingly serious. Termites' activities are usually hidden, covered by mud, weeds, etc., and their activities last for a long time, even lasting for ten years, forming anthills with a diameter of up to 2 meters, which can even cause the embankment to be penetrated, posing a huge safety threat to water conservancy projects.

[0003] At present, the inspection method for termite damage is usually to find surface indicators such as mud, mud lines, and chicken mushrooms. Among them, manual inspection is the main means, and some methods are completed in combination with termite automatic monitoring instruments. However, due to the huge number of dams, the cost of comprehensive coverage deployment is expensive and lacks scientificity, resulting in low efficiency in identifying termite indicators. There are still some problems in directly applying it to termite damage on dams. Therefore, there is an urgent need for an information-based method for large-scale and efficient termite indicator identification. At present, visual segmentation and recognition technology represented by artificial intelligence methods such as deep learning, reinforcement learning, transfer learning, and enhanced learning has made great progress and can be applied to the recognition of people and animals with good results. However, when directly applied to termite damage on dams, there are problems such as difficulty in extracting dam photos, low accuracy in identifying termite indicators, and slow recognition. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for identifying termite indicators based on optical and infrared drone data. The optical image and infrared image of the study area are acquired through optical lens and infrared lens, and are identified by a segmentation and recognition model based on deep learning technology, thereby realizing rapid and high-precision identification of termite indicators, so as to quickly determine the location and severity of termite damage and ensure the safety of dam and water conservancy projects in the study area.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A method for identifying termite indicators using optical and infrared drone data comprises the following steps:

[0007] S1, planning the flight path of the drone and setting the shooting parameters of the drone, and taking photos to obtain infrared images and optical images;

[0008] S2, obtaining a termite indicator image of the actual area, performing image enhancement processing, and performing vector calibration on the enhanced termite indicator image to obtain a classified and calibrated termite indicator image;

[0009] S3, inputting the infrared image and optical image obtained in step S1 and the classified and calibrated termite indicator image obtained in step S2 into the MaskRCNN image segmentation and recognition model for training, thereby obtaining a trained MaskRCNN image segmentation and recognition model;

[0010] S4. Obtain newly captured images of the study area and input them into the MaskRCNN image segmentation and recognition model trained in step S3 to identify termite indicators, obtain infrared images and optical images with the outlines and categories of termite indicators marked, and calculate the damage of termites in the study area based on the area of ​​the outlines of termite indicators.

[0011] Furthermore, step S1 specifically includes:

[0012] S11. Set the altitude and speed of the drone, and respectively set the focal lengths of the infrared lens and the optical lens, and calculate the minimum and maximum resolutions of the infrared lens and the optical lens, namely:

[0013]

[0014] Among them, a represents the pixel size, GSD represents the resolution of the lens, h represents the relative altitude of the drone, and f represents the focal length of the lens;

[0015] S12, obtaining an average resolution of the infrared lens according to the lowest resolution and the highest resolution of the infrared lens in step S11;

[0016] S13, obtaining an average resolution of the optical lens according to the lowest resolution and the highest resolution of the optical lens in step S11;

[0017] S14, according to the minimum resolution and maximum resolution of the infrared lens and the optical lens calculated in step S11 and the set focal lengths of the infrared lens and the optical lens, the flight route altitude of the UAV is calculated, and it satisfies the relationship corresponding to the following formula, that is:

[0018] L≥GSD avg

[0019] Among them, L represents the flight altitude of the drone, GSD avg Indicates the average resolution of the lens;

[0020] S15. Plan the flight path of the UAV, and according to the flight altitude of the UAV calculated in step S14 and the focal lengths of the infrared lens and the optical lens respectively set in step S11, use the UAV to conduct a flight inspection of the embankments in the study area, and simultaneously take photos to obtain infrared images and optical images.

[0021] Furthermore, step S2 specifically includes:

[0022] S21, obtaining the actual damage situation of termites in the area and the maximum area, minimum area and average area of ​​termite indicators, setting the minimum scale, area ratio and pixel resolution for identifying the area of ​​termite indicators in the image, and obtaining a termite indicator image;

[0023] S22, performing image enhancement processing on the termite indicator image obtained in step S21 to obtain an enhanced termite indicator image;

[0024] S23, using vector calibration software to mark the range and type of termite indicators in the enhanced termite indicator image obtained in step S22, to obtain a classified and calibrated termite indicator image.

[0025] Furthermore, step S3 specifically includes:

[0026] S31, using the infrared image and the optical image obtained in step S1 as raw data, using the classified and calibrated termite indicator image obtained in step S2 as a label for the raw data, and inputting the labeled raw data as an image data set into the MaskRCNN image segmentation and recognition model for training;

[0027] S32. Adjust the parameters of the model by improving the total loss function of the MaskRCNN image segmentation and recognition model to obtain suitable training parameters and a trained MaskRCNN image segmentation and recognition model.

[0028] Furthermore, the training process of the MaskRCNN image segmentation and recognition model in step S31 is:

[0029] Firstly, the image dataset is input into the convolution backbone structure to extract its features and output a feature map. Meanwhile, a region proposal network is introduced after the convolution backbone structure to select a candidate region in the feature map belonging to the image dataset. The candidate window of the candidate region adjusts the aspect ratio according to the type of termite indicator, and the feature map is aligned with the region of interest of a candidate region input to obtain a global feature map.

[0030] Secondly, a binary classification network consisting of a three-layer convolutional neural network and a one-layer fully connected layer is introduced after the region of interest, and the obtained global feature map is input into the binary classification network for termite indicator feature recognition. If the input global feature map contains termite indicators, the branch network output is true, and it is input into a branch network consisting of two fully connected layers and two convolutional layers for classification, bounding box regression and mask generation, and the loss function is used to adjust the weights of the fully connected layer and the convolutional layer respectively to obtain appropriate training parameters and a trained MaskRCNN image segmentation and recognition model; if the input global feature map does not contain termite indicators, it returns to the input end of the MaskRCNN image segmentation and recognition model.

[0031] Furthermore, the total loss function of the MaskRCNN image segmentation and recognition model in step S32 is:

[0032]

[0033] Among them, L represents the total loss function of the MaskRCNN image segmentation and recognition model, where w1, w2, w3, w4, and w5 represent weights, L class represents the loss of optical image classification, L box represents the loss of the proposed bounding box, L mask represents the segmentation loss, L class1 represents the loss of infrared image classification and is calculated in the same way as the loss of optical image classification. kps represents the loss of key positions, N cls Indicates the number of anchor boxes, L cls (·) indicates that the intermediate parameter has no practical meaning, N box Indicates the number of detection boxes, p i represents the accuracy of target prediction, Indicates the recognition result and both 0 and 1 indicate the correct result. It represents the mean absolute error between the true value and the predicted value, t i represents the vector anchor box, Indicates that i The vector anchor box of the same dimension, log(·) represents the logarithmic function, m represents the image dimension, y ij represents the mask label value of the image pixel in row i and column j, Indicates that the k-th channel image pixel is the mask label value of the i-th row and j-th column, k represents the channel, i and j represent the row and column of the pixel respectively.

[0034] Furthermore, step S4 specifically includes:

[0035] S41, obtaining newly captured images of the study area, and inputting the images into the trained MaskRCNN image segmentation and recognition model to identify termite indicators, thereby obtaining infrared images and optical images with outlines and categories of termite indicators marked;

[0036] S42, calculating the area of ​​the termite indicator contour in the infrared image and the optical image according to the infrared image and the optical image marked with the contour and category of the termite indicator obtained in step S41;

[0037] S43. Count the damage caused by termites in the study area according to the areas of the outlines of the termite indicators in the infrared image and the optical image calculated in step S42.

[0038] Furthermore, in step S42, the calculation formula for the area of ​​the termite indicator contour in the infrared image and the optical image is:

[0039]

[0040] Where S represents the area of ​​the termite indicator contour, w represents the image resolution, n represents the number of boundary points in the termite indicator contour, and x i ,y i They represent the positions of the pixel coordinates of the i-th boundary point in the termite indicator contour on the X and Y axes, respectively.

[0041] Furthermore, the calculation formula for the damage of termites in the study area in step S43 is:

[0042] S ′ =w1S l +w2S h

[0043] Among them, S ′ represents the damage of termites in the study area, w1 and w2 represent the weights of optical recognition results and infrared recognition results in the overall evaluation, respectively, and S l represents the area of ​​the termite indicator contour in the optical image, S h Represents the area of ​​the termite indicator outline in the infrared image.

[0044] A termite indicator recognition system for optical and infrared drone data, comprising:

[0045] The UAV image data acquisition module is used to plan the flight path of the UAV and set the shooting parameters of the UAV to obtain infrared images and optical images;

[0046] The image data preprocessing and sample calibration module is used to perform image enhancement processing on the acquired termite indicator image of the actual area, and perform vector calibration on the enhanced termite indicator image to obtain a classified and calibrated termite indicator image;

[0047] The indicator image segmentation and recognition module is used to input the infrared image and optical image obtained by the drone image data acquisition module and the classified and calibrated termite indicator image obtained by the image data preprocessing and sample calibration module into the MaskRCNN image segmentation and recognition model for training, thereby obtaining a trained MaskRCNN image segmentation and recognition model;

[0048] The module for rapid identification and calculation of termite indicators is used to obtain newly captured images of the study area and input the trained MaskRCNN image segmentation and recognition model to identify termite indicators, obtain infrared images and optical images with the outlines and categories of termite indicators marked, and calculate the damage of termites in the study area based on the area of ​​the termite indicator outlines.

[0049] The present invention has the following beneficial effects:

[0050] The present invention proposes a method and system for identifying termite indicators based on optical and infrared drone data, which utilizes the MaskRCNN image segmentation and recognition model to identify termite indicators in collected infrared images and optical images, thereby improving the accuracy and speed of termite indicator recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the process of a method for identifying termite indicators using optical and infrared drone data proposed in the present invention;

[0052] Figure 2 This is a schematic diagram of the MaskRCNN image segmentation and recognition model structure;

[0053] Figure 3 This is a schematic diagram of mud identification results;

[0054] Figure 4 This is a schematic diagram of the structure of a termite indicator recognition system based on optical and infrared drone data proposed in the present invention. DETAILED DESCRIPTION

[0055] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0056] like Figure 1 As shown, a method for identifying termite indicators using optical and infrared drone data includes the following steps S1-S4:

[0057] S1. Plan the flight path of the drone and set the shooting parameters of the drone, and take photos to obtain infrared images and optical images.

[0058] In this embodiment, by planning a reasonable UAV flight path and setting reasonable photo shooting parameters, the UAV is used to collect infrared images and optical images of the dike in the study area. Among them, the UAV is equipped with two lenses, namely an infrared lens and an optical lens, and the shooting parameters of the two lenses are set the same. Since the two lenses are different, the corresponding focal lengths are different, and the spatial resolution of the obtained images is also different. Since the ground of the dike in the study area is undulating, the relative height is set, and during the actual flight process, the flight route height of the UAV may change. Therefore, when flying and shooting, the flight route height of the UAV should be greater than or equal to the average resolution of the lens.

[0059] Specifically, step S1 includes S11-S15:

[0060] S11. Set the altitude and speed of the drone, and respectively set the focal lengths of the infrared lens and the optical lens, and calculate the minimum and maximum resolutions of the infrared lens and the optical lens, namely:

[0061]

[0062] Among them, a represents the pixel size, GSD represents the resolution of the lens, h represents the relative altitude of the drone, and f represents the focal length of the lens.

[0063] S12. Obtain an average resolution of the infrared lens according to the lowest resolution and the highest resolution of the infrared lens in step S11.

[0064] S13, obtaining an average resolution of the optical lens according to the lowest resolution and the highest resolution of the optical lens in step S11.

[0065] S14, according to the minimum resolution and maximum resolution of the infrared lens and the optical lens calculated in step S11 and the set focal lengths of the infrared lens and the optical lens, the flight route altitude of the UAV is calculated, and it satisfies the relationship corresponding to the following formula, that is:

[0066] L≥GSD avg

[0067] Among them, L represents the flight altitude of the drone, GSD avg Indicates the average resolution of the lens.

[0068] S15. Plan the flight path of the UAV, and according to the flight altitude of the UAV calculated in step S14 and the focal lengths of the infrared lens and the optical lens respectively set in step S11, use the UAV to conduct a flight inspection of the embankments in the study area, and simultaneously take photos to obtain infrared images and optical images.

[0069] S2. Acquire a termite indicator image of the actual area, perform image enhancement processing, and perform vector calibration on the enhanced termite indicator image to obtain a classified and calibrated termite indicator image.

[0070] In this embodiment, the image with termite indicators is subjected to image enhancement processing, wherein the image enhancement processing includes binarization, contrast enhancement, etc., and its purpose is to ensure that the effects of all images are close, so that the contrast, saturation, and clarity of the image are close to the same under cloudy days and conditions of particularly strong light.

[0071] Specifically, step S2 specifically includes S21-S23:

[0072] S21, obtaining the actual termite damage situation in the area and the maximum area, minimum area and average area of ​​the termite indicator, setting the minimum scale, area ratio and pixel resolution for identifying the termite indicator area in the image, and obtaining the termite indicator image.

[0073] In this embodiment, according to the actual damage situation of engineering termites in the area, combined with the maximum area, minimum area and average area required by the types of termite indicators such as Alisma oleifera, mud blanket, mud line, and fly hole, the minimum scale length for identifying the termite indicator in the image is set to 1 cm, the proportion of the area in an image is r, and the corresponding pixel resolution p is determined to obtain the termite indicator image.

[0074] S22, performing image enhancement processing on the termite indicator image obtained in step S21 to obtain an enhanced termite indicator image.

[0075] S23, using vector calibration software to mark the range and type of termite indicators in the enhanced termite indicator image obtained in step S22, to obtain a classified and calibrated termite indicator image.

[0076] In this embodiment, vector calibration software is used to mark the range and type of termite indicators. The range usually accurately covers the boundaries of termite indicators, and the types of termite indicators are calibrated, including Alstroemeria, Mud Quilt, Mud Line, Red Dusk, etc. The calibrated images and results are uniformly stored in corresponding categories with the file names.

[0077] S3. Input the infrared image and optical image obtained in step S1 and the classified and calibrated termite indicator image obtained in step S2 into the MaskRCNN image segmentation and recognition model for training to obtain a trained MaskRCNN image segmentation and recognition model.

[0078] like Figure 2 As shown, Figure 2 This is a schematic diagram of the MaskRCNN image segmentation and recognition model structure; Figure 2 The structure of the MaskRCNN image segmentation and recognition model is clearly given. In this model, the candidate window changes the traditional fixed ratio to the aspect ratio that needs to be adjusted according to the category of termite indicators. The purpose is to meet the aspect ratio of common termite indicators and increase the original anchor box by about 20% to improve the accuracy of model recognition. At the same time, the branch network is connected to the global feature map. The branch network is a two-classification network composed of three layers of convolutional neural network and one fully connected layer. It is used to determine whether the global feature map contains termite indicators; if the branch network output is true, then Figure 2 If there is an edge marked as yes in the image, it is determined that the global feature map contains a termite indicator object area, and it is input into a branch network containing two fully connected layers and two convolutional layers to identify the location of the area in the global feature map and perform segmentation. Otherwise, it returns to the input end of the model. In this embodiment, in order to improve the accuracy of the model, other positions of the model remain unchanged, and the total loss function of the MaskRCNN image segmentation and recognition model is adjusted to screen out suitable training parameters. The MaskRCNN image segmentation and recognition model adopted by the present invention obtains a trained model by selecting suitable training parameters, and under the same conditions, it is faster and consumes less resources.

[0079] Specifically, step S3 specifically includes S31-S32:

[0080] S31. The infrared image and the optical image obtained in step S1 are used as raw data, the classified and calibrated termite indicator image obtained in step S2 is used as a label for the raw data, and the labeled raw data is input as an image data set into the MaskRCNN image segmentation and recognition model for training.

[0081] Specifically, the training process of the MaskRCNN image segmentation and recognition model in step S31 is:

[0082] Firstly, the image dataset is input into the convolution backbone structure for feature extraction and outputting feature maps. Meanwhile, a region proposal network is introduced after the convolution backbone structure to select a class of candidate regions in the feature map belonging to the image dataset. The candidate window of the candidate region adjusts the aspect ratio according to the type of termite indicator, and the feature map is aligned with the region of interest of a class of candidate regions input to obtain a global feature map.

[0083] Secondly, a binary classification network consisting of a three-layer convolutional neural network and a one-layer fully connected layer is introduced after the region of interest, and the obtained global feature map is input into the binary classification network for termite indicator feature recognition. If the input global feature map contains termite indicators, the branch network output is true, and it is input into a branch network consisting of two fully connected layers and two convolutional layers for classification, bounding box regression and mask generation, and the loss function is used to adjust the weights of the fully connected layer and the convolutional layer respectively to obtain appropriate training parameters and a trained MaskRCNN image segmentation and recognition model; if the input global feature map does not contain termite indicators, it returns to the input end of the MaskRCNN image segmentation and recognition model.

[0084] S32. Adjust the parameters of the model by improving the total loss function of the MaskRCNN image segmentation and recognition model to obtain suitable training parameters and a trained MaskRCNN image segmentation and recognition model.

[0085] Specifically, the total loss function of the MaskRCNN image segmentation and recognition model in step S32 is:

[0086]

[0087] Among them, L represents the total loss function of the MaskRCNN image segmentation and recognition model, where w1, w2, w3, w4, and w5 represent weights, L class represents the loss of optical image classification, L box represents the loss of the proposed bounding box, L mask represents the segmentation loss, L class1 represents the loss of infrared image classification and is calculated in the same way as the loss of optical image classification. kps represents the loss of key positions, N cls Indicates the number of anchor boxes, L cls (·) indicates that the intermediate parameter has no practical meaning, N box Indicates the number of detection boxes, p i represents the accuracy of target prediction, Indicates the recognition result and both 0 and 1 indicate the correct result. It represents the mean absolute error between the true value and the predicted value, t i represents the vector anchor box, Indicates that i The vector anchor box of the same dimension, log(·) represents the logarithmic function, m represents the image dimension, y ij represents the mask label value of the image pixel in row i and column j, Indicates that the k-th channel image pixel is the mask label value of the i-th row and j-th column, k represents the channel, i and j represent the row and column of the pixel respectively.

[0088] In this embodiment, the weight of the model is selected according to the actual situation, and its reasonable value is determined through multiple experiments or least squares method.

[0089] S4. Obtain newly captured images of the study area and input them into the MaskRCNN image segmentation and recognition model trained in step S3 to identify termite indicators, obtain infrared images and optical images with the outlines and categories of termite indicators marked, and calculate the damage of termites in the study area based on the area of ​​the outlines of termite indicators.

[0090] In this embodiment, the newly captured images are infrared images and optical images of the study area captured by the infrared lens and optical lens of the drone. The trained MaskRCNN image segmentation and recognition model is used to identify termite indicators on the newly input images to obtain images marked with termite indicators. According to the outline and category of the identified termite indicators, the area is calculated to obtain the termite damage situation in the study area.

[0091] Specifically, step S4 includes S41-S43:

[0092] S41. Acquire newly captured images of the study area and input them into the trained MaskRCNN image segmentation and recognition model to identify termite indicators, thereby obtaining infrared images and optical images with outlines and categories of termite indicators marked.

[0093] S42. Calculate the area of ​​the termite indicator outline in the infrared image and the optical image based on the infrared image and the optical image marked with the outline and category of the termite indicator obtained in step S41.

[0094] Specifically, the calculation formula for the area of ​​the termite indicator contour in the infrared image and the optical image in step S42 is:

[0095]

[0096] Where S represents the area of ​​the termite indicator contour, w represents the image resolution, n represents the number of boundary points in the termite indicator contour, and x i ,y i They represent the positions of the pixel coordinates of the i-th boundary point in the termite indicator contour on the X and Y axes, respectively.

[0097] S43. Count the damage caused by termites in the study area according to the areas of the outlines of the termite indicators in the infrared image and the optical image calculated in step S42.

[0098] Specifically, the calculation formula for the damage of termites in the study area in step S43 is:

[0099] S ′ =w1S l +w2S h

[0100] Among them, S ′ represents the damage of termites in the study area, w1 and w2 represent the weights of optical recognition results and infrared recognition results in the overall evaluation, respectively, and S l represents the area of ​​the termite indicator contour in the optical image, S h Represents the area of ​​the termite indicator outline in the infrared image.

[0101] like Figure 3 As shown, Figure 3 This is a schematic diagram of mud identification results; Figure 3 The part framed by the black solid line is the specific location of the mud blanket in the termite indicator in the study area. In this embodiment, the study area is identified using the optical and infrared drone data termite indicator identification method proposed in the present invention, and mud blankets are found on the tree trunks in the study area. Therefore, the termite indicator can be accurately identified using this method.

[0102] like Figure 4 As shown, a termite indicator recognition system based on optical and infrared drone data includes:

[0103] The UAV image data acquisition module is used to plan the UAV flight path and set the shooting parameters of the UAV to obtain infrared images and optical images.

[0104] The image data preprocessing and sample calibration module is used to perform image enhancement processing on the acquired termite indicator image of the actual area, and to perform vector calibration on the enhanced termite indicator image to obtain a classified and calibrated termite indicator image.

[0105] The indicator image segmentation and recognition module is used to input the infrared image and optical image obtained by the drone image data acquisition module and the classified and calibrated termite indicator image obtained by the image data preprocessing and sample calibration module into the MaskRCNN image segmentation and recognition model for training, so as to obtain a trained MaskRCNN image segmentation and recognition model.

[0106] The module for rapid identification and calculation of termite indicators is used to obtain newly captured images of the study area and input the trained MaskRCNN image segmentation and recognition model to identify termite indicators, obtain infrared images and optical images with the outlines and categories of termite indicators marked, and calculate the damage of termites in the study area based on the area of ​​the termite indicator outlines.

[0107] In the present embodiment, a termite indicator recognition system based on optical and infrared drone data is provided. Optical images and infrared images of the study area are acquired through the optical lens and infrared lens of the drone, and are recognized using a segmentation recognition model based on deep learning technology. A fast and high-precision recognition system for termite indicators is provided, and fast recognition of termite damage indicators in earth-rock structure type embankments, dams, slopes and other water conservancy projects is achieved, so that the location and severity of termite damage can be quickly surveyed and determined, thereby ensuring the safety of embankment water conservancy projects in the study area.

[0108] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0109] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for identifying termite indicators using optical and infrared drone data, characterized in that: The following steps are involved: S1, planning the flight path of the drone and setting the shooting parameters of the drone, and taking photos to obtain infrared images and optical images; S2, obtaining a termite indicator image of the actual area, performing image enhancement processing, and performing vector calibration on the enhanced termite indicator image to obtain a classified and calibrated termite indicator image; S3, inputting the infrared image and optical image obtained in step S1 and the classified and calibrated termite indicator image obtained in step S2 into the MaskRCNN image segmentation and recognition model for training, thereby obtaining a trained MaskRCNN image segmentation and recognition model; Wherein, step S3 specifically includes: S31, using the infrared image and the optical image obtained in step S1 as raw data, using the classified and calibrated termite indicator image obtained in step S2 as a label for the raw data, and inputting the labeled raw data as an image data set into the MaskRCNN image segmentation and recognition model for training; S32. Adjust the parameters of the model by improving the total loss function of the MaskRCNN image segmentation and recognition model to obtain appropriate training parameters and a trained MaskRCNN image segmentation and recognition model; Among them, the total loss function of the improved MaskRCNN image segmentation and recognition model is: in, Represents the total loss function of the MaskRCNN image segmentation and recognition model, where , , , , Represent weights, represents the loss of optical image classification, represents the loss of the proposed bounding box, represents the segmentation loss, represents the loss of infrared image classification and is calculated in the same way as the loss of optical image classification. represents the loss at the key location, represents the number of anchor boxes, Indicates that the intermediate parameters have no practical meaning. Indicates the number of detection boxes. represents the accuracy of target prediction, Indicates the recognition result and both 0 and 1 indicate the correct result. It represents the mean absolute error between the true value and the predicted value. represents the vector anchor box, Representation and Vector anchor boxes of the same dimension, represents the logarithmic function, represents the image dimension, Indicates that the image pixel is Line and The mask label value of the column, Indicates Channel image pixels are Line and The mask label value of the column, Indicates the channel, , Represent the rows and columns of pixels respectively; S4, obtaining a newly captured image of the study area and inputting it into the MaskRCNN image segmentation and recognition model trained in step S3 to identify termite indicators, obtaining infrared images and optical images with outlines and categories of termite indicators marked, and calculating the damage of termites in the study area according to the area of ​​the outlines of termite indicators; Wherein, step S4 specifically includes: S41, obtaining newly captured images of the study area, and inputting the images into the trained MaskRCNN image segmentation and recognition model to identify termite indicators, thereby obtaining infrared images and optical images with outlines and categories of termite indicators marked; S42, according to the infrared image and the optical image marked with the outline and category of the termite indicator obtained in step S41, the area of ​​the outline of the termite indicator in the infrared image and the optical image is calculated, that is: in, represents the area of ​​the termite indicator outline, Indicates the image resolution, represents the number of boundary points in the termite indicator contour, , Respectively represent the first The pixel coordinates of the boundary points on the X and Y axes; S43, according to the area of ​​the outline of the termite indicator in the infrared image and the optical image calculated in step S42, the damage of termites in the study area is counted, that is: in, Indicates the damage caused by termites in the study area. , Respectively represent the weights of optical recognition results and infrared recognition results in the overall evaluation, represents the area of ​​the termite indicator contour in the optical image, Represents the area of ​​the termite indicator outline in the infrared image.

2. The method for identifying termite indicators using optical and infrared drone data according to claim 1, characterized in that: Step S1 specifically includes: S11. Set the altitude and speed of the drone, and respectively set the focal lengths of the infrared lens and the optical lens, and calculate the minimum and maximum resolutions of the infrared lens and the optical lens, namely: in, represents the pixel size, Indicates the resolution of the lens. Indicates the relative altitude of the drone. Indicates the focal length of the lens; S12, obtaining an average resolution of the infrared lens according to the lowest resolution and the highest resolution of the infrared lens in step S11; S13, obtaining an average resolution of the optical lens according to the lowest resolution and the highest resolution of the optical lens in step S11; S14, according to the minimum resolution and maximum resolution of the infrared lens and the optical lens calculated in step S11 and the set focal lengths of the infrared lens and the optical lens, the flight route altitude of the UAV is calculated, and it satisfies the relationship corresponding to the following formula, that is: in, Indicates the flight altitude of the drone. Indicates the average resolution of the lens; S15. Plan the flight path of the UAV, and according to the flight altitude of the UAV calculated in step S14 and the focal lengths of the infrared lens and the optical lens respectively set in step S11, use the UAV to conduct a flight inspection of the embankments in the study area, and simultaneously take photos to obtain infrared images and optical images.

3. The method for identifying termite indicators using optical and infrared drone data according to claim 1, characterized in that: Step S2 specifically includes: S21, obtaining the actual damage situation of termites in the area and the maximum area, minimum area and average area of ​​termite indicators, setting the minimum scale, area ratio and pixel resolution for identifying the area of ​​termite indicators in the image, and obtaining a termite indicator image; S22, performing image enhancement processing on the termite indicator image obtained in step S21 to obtain an enhanced termite indicator image; S23, using vector calibration software to mark the range and type of termite indicators in the enhanced termite indicator image obtained in step S22, to obtain a classified and calibrated termite indicator image.

4. The method for identifying termite indicators using optical and infrared drone data according to claim 1, characterized in that: The training process of the MaskRCNN image segmentation and recognition model in step S31 is: Firstly, the image dataset is input into the convolution backbone structure to extract its features and output a feature map. Meanwhile, a region proposal network is introduced after the convolution backbone structure to select a candidate region in the feature map belonging to the image dataset. The candidate window of the candidate region adjusts the aspect ratio according to the type of termite indicator, and the feature map is aligned with the region of interest of a candidate region input to obtain a global feature map. Secondly, a binary classification network consisting of a three-layer convolutional neural network and a one-layer fully connected layer is introduced after the region of interest, and the obtained global feature map is input into the binary classification network for termite indicator feature recognition. If the input global feature map contains termite indicators, the branch network output is true, and it is input into a branch network consisting of two fully connected layers and two convolutional layers for classification, bounding box regression and mask generation, and the loss function is used to adjust the weights of the fully connected layer and the convolutional layer respectively to obtain appropriate training parameters and a trained MaskRCNN image segmentation and recognition model; if the input global feature map does not contain termite indicators, it returns to the input end of the MaskRCNN image segmentation and recognition model.

5. A termite indicator recognition system based on optical and infrared drone data, characterized in that: include: The UAV image data acquisition module is used to plan the flight path of the UAV and set the shooting parameters of the UAV to obtain infrared images and optical images; The image data preprocessing and sample calibration module is used to perform image enhancement processing on the acquired termite indicator image of the actual area, and perform vector calibration on the enhanced termite indicator image to obtain a classified and calibrated termite indicator image; The indicator image segmentation and recognition module is used to input the infrared image and optical image obtained by the drone image data acquisition module and the classified and calibrated termite indicator image obtained by the image data preprocessing and sample calibration module into the MaskRCNN image segmentation and recognition model for training to obtain a trained MaskRCNN image segmentation and recognition model, which specifically includes: The infrared image and the optical image are used as the raw data, the classified and calibrated termite indicator image is used as the label of the raw data, and the labeled raw data is used as the image dataset to be input into the MaskRCNN image segmentation and recognition model for training; By improving the total loss function of the MaskRCNN image segmentation and recognition model, the model parameters are adjusted to obtain appropriate training parameters and a trained MaskRCNN image segmentation and recognition model. Among them, the total loss function of the improved MaskRCNN image segmentation and recognition model is: in, Represents the total loss function of the MaskRCNN image segmentation and recognition model, where , , , , Represent weights, represents the loss of optical image classification, represents the loss of the proposed bounding box, represents the segmentation loss, represents the loss of infrared image classification and is calculated in the same way as the loss of optical image classification. represents the loss at the key location, represents the number of anchor boxes, Indicates that the intermediate parameters have no practical meaning. Indicates the number of detection boxes. represents the accuracy of target prediction, Indicates the recognition result and both 0 and 1 indicate the correct result. It represents the mean absolute error between the true value and the predicted value. represents the vector anchor box, Representation and Vector anchor boxes of the same dimension, represents the logarithmic function, represents the image dimension, Indicates that the image pixel is Line and The mask label value of the column, Indicates Channel image pixels are Line and The mask label value of the column, Indicates the channel, , Represent the rows and columns of pixels respectively; The module for rapid identification and calculation of termite indicators is used to obtain newly captured images of the study area and input the trained MaskRCNN image segmentation and recognition model to identify termite indicators, obtain infrared images and optical images with the outlines and categories of termite indicators marked, and calculate the damage of termites in the study area based on the area of ​​the outlines of termite indicators, including: Get newly taken images of the study area and input them into the trained MaskRCNN image segmentation and recognition model to identify termite indicators, and obtain infrared images and optical images with the outlines and categories of termite indicators marked; According to the infrared image and optical image with the outline and category of the termite indicator marked, the area of ​​the outline of the termite indicator in the infrared image and the optical image is calculated, that is: in, represents the area of ​​the termite indicator outline, Indicates the image resolution, represents the number of boundary points in the termite indicator contour, , Respectively represent the first The pixel coordinates of the boundary points on the X and Y axes; According to the area of ​​termite indicator contours in infrared images and optical images, the damage of termites in the study area is statistically analyzed, namely: in, Indicates the damage caused by termites in the study area. , Respectively represent the weights of optical recognition results and infrared recognition results in the overall evaluation, represents the area of ​​the termite indicator contour in the optical image, Represents the area of ​​the termite indicator outline in the infrared image.

Images