A method for detecting forest baldness based on artificial intelligence and drone remote sensing

Abstract

The present invention discloses a method for detecting bare forest land based on artificial intelligence and unmanned aerial vehicle remote sensing, which comprises: (S1) using an unmanned aerial vehicle to patrol a target area and collect position information and image information in real time; (S2) comparing the position information with the planning range line of a forestry class, judging whether the position of the unmanned aerial vehicle and the scene in the image information are within the range line, using a convolutional neural network to identify the image information within the range line, and obtaining the bare forest land area; (S3) calculating the canopy density of each forestry class according to the bare forest land area. This method can identify the bare forest land in real time. Compared with hyperspectral equipment, visible light cameras are cheaper and more commonly used. Therefore, the relevant deep learning algorithms can be deployed on a large scale, saving costs. Compared with satellite remote sensing, this solution is more flexible, more accurate, and more suitable for Shanghai's climatic conditions, which is conducive to timely law enforcement by the forestry department.

Description

Technical Field

[0001] The invention belongs to the field of forestry remote sensing, and in particular relates to a forest baldness detection method based on artificial intelligence and unmanned aerial vehicle remote sensing. Background Art

[0002] There are two main ways to obtain data for monitoring traditional forest bareness. One is manual inspection, and the other is satellite or remote sensing hyperspectral image collection. Both require human labor for data processing or image interpretation at a later stage.

[0003] Manual inspections usually require a lot of manpower and take a long time, while satellite and aerial remote sensing are greatly affected by weather factors such as cloud cover, have low flexibility and low resolution. According to data provided by the National Meteorological Science Data Center, the annual average total cloud cover in Shanghai is 60%, and the number of days with an annual average total cloud cover greater than 80% (cloudy days) is about 120 days, accounting for one-third of the year, which creates certain difficulties for remote sensing observations. UAVs are not affected by cloud cover.

[0004] At present, the resolution of aerial images commonly used in forestry in Shanghai is 0.25 meters, which is difficult to meet the needs of refined operations, especially for monitoring young forests, where the resolution of 0.25 meters is difficult to accurately distinguish tree crowns. The resolution of drone images is generally at the centimeter level. Drones equipped with multiple sensors and equipment such as RTK GNSS can achieve more functions and more accurate monitoring.

[0005] The remote sensing image acquisition cycle in Shanghai is only four times a year, and the monitoring frequency and flexibility need to be improved. Drones are relatively flexible and can conduct targeted monitoring of key areas periodically as needed.

[0006] In terms of data processing, the commonly used manual remote sensing image interpretation often involves operations such as orthophoto processing and hyperspectral filtering processing, and the data post-processing cycle is relatively long. In addition, hyperspectral equipment is easily affected by light images, which has certain restrictions on the lighting conditions of the operation. In general, the existing technology for judging the emptiness of forest land is low in real time, and there is a lot of manual intervention, which relies on manpower. Summary of the invention

[0007] The purpose of the present invention is to provide a method for detecting bare forest land based on artificial intelligence and unmanned aerial vehicle remote sensing. The method uses real-time video streams from unmanned aerial vehicles and real-time two-dimensional orthophoto models to efficiently identify bare forest land, calculate the density, and design backup plans to deal with emergencies.

[0008] The technical solution of the present invention is a method for detecting bald forest land based on artificial intelligence and unmanned aerial vehicle remote sensing, which comprises:

[0009] (S1) Use drones to patrol the target area and collect location information and image information in real time;

[0010] (S2) comparing the location information with the range line of the forestry class planning, determining whether the location of the drone and the scene in the image information are within the range line, and using a convolutional neural network to identify the image information within the range line to obtain the area of ​​the bald area of ​​the forest land;

[0011] (S3) Calculate the density of each forestry group based on the area of ​​bare areas.

[0012] The present invention further lies in that the image information includes video, and the transmission and recognition of the video to the recognition device adopts one or more combinations of the following methods:

[0013] (1) The drone uses RTMP streaming to transmit the video to the stream processing server. After the cloud pulls the stream, the convolutional neural network deployed on the recognition device identifies the vacant areas in real time.

[0014] (2) An embedded device is deployed on the drone as a recognition device. The recognition device uses a convolutional neural network to perform real-time recognition of the video and transmits back the recognition results of the empty area;

[0015] (3) The drone stores the recorded video in an onboard storage device. After the drone is recovered, a recognition device equipped with a convolutional neural network is used to recognize the video.

[0016] The present invention further lies in that, in step (S2), the recognition result of the convolutional neural network includes the number of pixels of the forestry class and the number of pixels of the empty and bald area; in the process of calculating the empty and bald area, the empty and bald area is determined according to the number of pixels of the empty and bald area and the ground resolution of the video image; wherein the step of obtaining the ground resolution of the video image includes:

[0017] (S201) performing distortion correction according to camera parameters;

[0018] (S202) Calculate the ground resolution (GSD) of the video image, which is expressed as:

[0019] H=(imW×GSD×Fr) / (Sw×100)

[0020] Where: H is the relative altitude (meters), Fr is the focal length of the camera lens (millimeters), Sw is the actual sensor width (millimeters), imW is the image width (pixels), and GSD is the ground resolution of the image (cm / pixel). The relative altitude is determined based on the location information of the drone.

[0021] The present invention further lies in that the image information includes a photo, and the transmission and recognition of the photo to the recognition device adopts one or a combination of two of the following methods:

[0022] (1) After receiving the photos sent back by the drone, the recognition device performs real-time two-dimensional orthophoto processing on the photos. Based on the orthophotos, a convolutional neural network is used to identify the bare areas of the forest land, and the canopy density of each forestry group is calculated based on the area of ​​the bare areas.

[0023] (2) The drone stores the photos in an onboard storage device. After the drone is recovered, the recognition device performs two-dimensional orthophoto processing on the photos. Based on the orthophotos, a convolutional neural network is used to identify the bare areas of the forest land, and the density of each forestry group is calculated based on the area of ​​the bare areas.

[0024] The present invention further lies in that the convolutional neural network is SOLOv2.

[0025] The present invention further lies in that the training process of the convolutional neural network includes:

[0026] 1) Collect sample photos of bare forests;

[0027] 2) Manually identify and annotate photos;

[0028] 3) Perform data augmentation on the annotated photos to obtain a data set;

[0029] 4) Divide the data set into a training set and a test set, and use the training set to train the convolutional neural network;

[0030] 5) Use the test set to verify the trained convolutional neural network.

[0031] The present invention further lies in that the data augmentation process comprises the following steps:

[0032] a) Take photos from multiple angles and mirror, flip, translate, zoom, rotate, stretch and deform the photos;

[0033] b) Add noise to the photo;

[0034] c) Randomly adjust and change the brightness, contrast, and saturation of the photo;

[0035] d) Enhance the background of the photo;

[0036] e) Add orthophoto processed images;

[0037] f) Use the adversarial generative network (DAGAN-GP) to amplify samples in the bald area and generate bald images.

[0038] The beneficial effects of the present invention are as follows: the method can identify the situation of forest baldness in real time. Compared with hyperspectral equipment, visible light cameras are cheaper and more commonly used. Therefore, relevant deep learning algorithms can be deployed on a large scale, saving costs. Compared with satellite remote sensing, this solution is more flexible, more accurate, and more suitable for Shanghai's climatic conditions, which is conducive to timely law enforcement by the forestry department. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the forest baldness detection method based on artificial intelligence and drone remote sensing. DETAILED DESCRIPTION

[0040] Example 1: Figure 1 As shown, this embodiment provides a method for detecting bald forest land based on artificial intelligence and drone remote sensing, which includes:

[0041] (S1) Use a drone to patrol the target area and collect location information and image information in real time. The location information is obtained by the RTK module of the drone, and the image information includes video and images, which are taken by the visible light camera mounted on the drone.

[0042] (S2) Compare the location information with the range line of the forestry class planning, determine whether the location of the drone and the scene in the image information are within the range line, use a convolutional neural network to identify the image information within the range line, and obtain the area of ​​the vacant area of ​​the forest land.

[0043] (S3) Calculate the density of each forestry group based on the area of ​​bare areas.

[0044] In this embodiment, different processing methods are used for videos and photos.

[0045] The transmission and recognition of the video to the recognition device adopts one or more of the following methods:

[0046] (1) The drone uses RTMP streaming to transmit the video to the stream processing server. After the cloud pulls the stream, the convolutional neural network deployed on the recognition device identifies the vacant areas in real time.

[0047] (2) In order to reduce the amount of data transmission, an embedded device is deployed on the drone as a recognition device. The recognition device uses a convolutional neural network to perform real-time recognition of the video and transmits back the recognition results of the empty area;

[0048] (3) In case of loss of image transmission, the drone stores the recorded video in an onboard storage device. After the drone is recovered, a recognition device equipped with a convolutional neural network is used to identify the video.

[0049] In step (S2), the recognition result of the convolutional neural network for the video includes the number of pixels of the forestry class and the number of pixels of the empty and bald area; in the process of calculating the empty and bald area, the empty and bald area is determined according to the number of pixels of the empty and bald area and the ground resolution of the video image; wherein the step of obtaining the ground resolution of the video image includes:

[0050] (S201) performing distortion correction according to camera parameters;

[0051] (S202) Calculate the ground resolution (GSD) of the video image, which is expressed as:

[0052] H=(imW×GSD×Fr) / (Sw×100)

[0053] Where: H is the relative altitude (meters), Fr is the focal length of the camera lens (millimeters), Sw is the actual sensor width (millimeters), imW is the image width (pixels), and GSD is the ground resolution of the image (cm / pixel); the relative altitude is determined based on the position information of the drone. When the camera parameters are determined and the relative altitude is known, the image ground resolution (GSD) can be calculated according to the above formula. Image ground resolution (GSD) represents the actual size represented by each pixel in the video.

[0054] The transmission and recognition of the photo to the recognition device adopts one or a combination of the following methods:

[0055] (1) After receiving the photos sent back by the drone, the recognition device performs real-time two-dimensional orthophoto processing on the photos. Based on the orthophotos, a convolutional neural network is used to identify the bare areas of the forest land, and the canopy density of each forestry group is calculated based on the area of ​​the bare areas.

[0056] (2) The drone stores the photos in an onboard storage device. After the drone is recovered, the recognition device performs two-dimensional orthophoto processing on the photos. Based on the orthophotos, a convolutional neural network is used to identify the bare areas of the forest land, and the density of each forestry group is calculated based on the area of ​​the bare areas.

[0057] The canopy density is the ratio of the projected area of ​​the canopy to the forest area. The criterion for judging severe forest baldness is that the tree crown projection accounts for less than 20% of the forest area. If it is higher than 70%, the canopy is likely to be connected, which is not conducive to the healthy development of the forest ecology. Therefore, if the canopy density is greater than 0.7 or less than 0.2, that is, the canopy density exceeds the limit or does not meet the standard, a work order will be automatically generated.

[0058] In this embodiment, the convolutional neural network is SOLOv2. The training process of the convolutional neural network includes:

[0059] 1) Collect sample photos of bare forests;

[0060] 2) Manually identify and annotate photos;

[0061] 3) Perform data augmentation on the annotated photos to obtain a data set;

[0062] 4) Divide the data set into a training set and a test set, and use the training set to train the convolutional neural network;

[0063] 5) Use the test set to verify the trained convolutional neural network.

[0064] In this embodiment, the data augmentation process includes the following steps:

[0065] a) Taking photos from multiple angles, mirroring, translating, scaling, rotating, stretching and deforming the photos; the shooting process is performed by drones, and the photos taken include bare forest land;

[0066] b) Add noise to the photo;

[0067] c) Randomly adjust and change the brightness, contrast, and saturation of the photo;

[0068] d) Enhance the background of the photo;

[0069] e) Add orthophoto images; orthophoto images usually cover a larger area and need to be sliced ​​to form orthophotos of smaller areas in order to be used together with photographs;

[0070] f) Use the Generative Adversarial Network (DAGAN-GP) to amplify samples in the bald area and generate bald images. The Generative Adversarial Network uses orthophotos and processed photos as input. The amplified samples can be used as training data for the convolutional neural network through manual annotation.

[0071] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for detecting bald forest land based on artificial intelligence and unmanned aerial vehicle remote sensing, comprising: (S1) Use drones to patrol the target area and collect location information and image information in real time; (S2) comparing the location information with the range line of the forestry class planning, determining whether the location of the drone and the scene in the image information are within the range line, and using a convolutional neural network to identify the image information within the range line to obtain the area of ​​the bald area of ​​the forest land; (S3) Calculate the canopy density of each forestry group based on the area of ​​bare areas; The image information includes video, and the transmission and recognition of the video to the recognition device adopts one or more of the following methods: (1) The drone uses RTMP streaming to transmit the video to the stream processing server. After the cloud pulls the stream, the convolutional neural network deployed on the recognition device identifies the vacant areas in real time. (2) An embedded device is deployed on the drone as a recognition device. The recognition device uses a convolutional neural network to perform real-time recognition of the video and transmits back the recognition results of the empty area; (3) The drone stores the recorded video in an onboard storage device, and after the drone is recovered, a recognition device deployed with a convolutional neural network is used to recognize the video; In step (S2), the recognition result of the convolutional neural network includes the number of pixels of the forestry class and the number of pixels of the empty and bald area; In the process of calculating the bald area, the bald area is determined according to the number of pixels in the bald area and the ground resolution of the video image; wherein the steps of obtaining the ground resolution of the video image include: (S201) performing distortion correction according to camera parameters; (S202) Calculate the ground resolution of the video image, which is expressed as: H=(imW×GSD×Fr) / (Sw×100) Where: H is the relative altitude, Fr is the focal length of the camera lens, Sw is the actual sensor width, imW is the image width, and GSD is the ground resolution of the image; the relative altitude is determined according to the position information of the UAV.

2. The method for detecting forest baldness based on artificial intelligence and drone remote sensing according to claim 1 is characterized in that: The image information includes a photo, and the photo is transmitted to the recognition device and recognized by one or a combination of the following methods: (1) After receiving the photos sent back by the drone, the recognition device performs real-time two-dimensional orthophoto processing on the photos. Based on the orthophotos, a convolutional neural network is used to identify the bare areas of the forest land, and the canopy density of each forestry group is calculated based on the area of ​​the bare areas. (2) The drone stores the photos in an onboard storage device. After the drone is recovered, the recognition device performs two-dimensional orthophoto processing on the photos. Based on the orthophotos, a convolutional neural network is used to identify the bare areas of the forest land, and the density of each forestry group is calculated based on the area of ​​the bare areas.

3. The method for detecting forest baldness based on artificial intelligence and unmanned aerial vehicle remote sensing according to any one of claims 1 to 2, characterized in that: The convolutional neural network is SOLOv2.

4. The method for detecting bald forest land based on artificial intelligence and drone remote sensing according to claim 3 is characterized in that: The training process of the convolutional neural network includes: 1) Collect sample photos of bare forests; 2) Manually identify and annotate photos; 3) Perform data augmentation on the annotated photos to obtain a data set; 4) Divide the data set into a training set and a test set, and use the training set to train the convolutional neural network; 5) Use the test set to verify the trained convolutional neural network.

5. According to claim 4, a method for detecting forest baldness based on artificial intelligence and drone remote sensing is characterized in that: The data augmentation process includes the following steps: a) Take photos from multiple angles and mirror, flip, translate, zoom, rotate, stretch and deform the photos; b) Add noise to the photo; c) Randomly adjust and change the brightness, contrast, and saturation of the photo; d) Enhance the background of the photo; e) Add orthophoto processed images; f) Use the adversarial generative network to amplify samples in the empty area and generate empty images.

Images