Method of marking bark beetle infested trees and means for executing this method

Abstract

Method of marking bark beetle infested trees, and means for executing this method, comprises acquiring a map of a forest and selecting the GNSS coordinates of potential infested areas and possible entry points into the forest for an autonomous UAV. The method further comprises deployment of the autonomous UAV in the nearest entry point with GNSS localization, and integration of localization and mapping information from onboard SLAM algorithms with the GNSS coordinates of its entry point, to enter and navigate the forest and avoid obstacles while reaching the coordinates of the possible bark beetle infestation. The method further comprises inspection of the trees by the UAV using onboard cameras and CNN computer vision algorithms, to identify characteristic marks of bark beetle infestation and verification of the infestation, followed by marking the infested tree by UAV using marking means, preferably with paint dispensed from an onboard paint dispenser, and verification of the marking using its onboard cameras and CNN algorithms.

Description

Title: Method of marking bark beetle infested trees and means for executing this methodTECHNICAL FIELD

[0001] The invention relates to a method of searching and marking bark beetle infested trees using at least one UAV equipped with means adapted to execute required steps and comprising hardware and software specifically designed for this method,BACKGROUND ART

[0002] Current methods for detecting bark beetle infestations suffer from inherent limitations, primarily relying on either aerial or static surveillance techniques (CN108921330A or CN111669723A), While these methods provide valuable insights into the overall state of the forest, they lack the capability to closely mark or definitively confirm infestations on a granular level. Their distant vantage point restricts their ability to gather detailed information up close. To address this gap, some existing approaches involve direct tree examination, wherein forest field workers are tasked with locating suspected trees and conducting inspections. These inspections can be performed using the workers’ visual assessment or aided by a camera connected to a computer, which employs sophisticated computer-vision algorithms for analysis (CZ35641U1). However, these methods present their own set of challenges and limitations.

[0003] One major drawback is the time-consuming nature of the process. Forest field workers must physically enter the forest and navigate through potentially difficult or even impassable terrains to access the trees. This not only hampers efficiency but also poses safety risks for the workers involved. Additionally, covering a large area using this approach requires a significant workforce, resulting in substantial costs and resource allocation.

[0004] Moreover, the reHance on manual inspections and human judgment introduces subjectivity and potential inconsistencies in identifying and confirming bark beetle infestations. The accuracy and reliability of the assessments heavily depend on the expertise and experience of the field workers, which can vary from person to person. This lack of standardized and objective evaluation methods undermines the effectiveness of the detection process.

[0005] In light of these challenges, there is a need for an innovative and efficient solution that overcomes the limitations of current approaches. Such a solution should enable close-range monitoring and accurate identification of bark beetle infestations while minimizing the reliance on manual labor and mitigating the difficulties associated with challenging terrains. By doing so, it would offer a more comprehensive and reliable means of detecting and managing these detrimental infestations, ultimately benefiting forest conservation efforts and minimizing economic losses caused by bark beetle damage.SUMMARY OF THE INVENTION

[0006] Present invention relates to a method of searching and marking bark beetle infested trees, which Is a method based on the use of UAVs to locate and mark infested trees. The method removes the disadvantages of the current state of the art by being both much faster than if the trees were searched by a forest field worker, and by making it possible to locate bark beetle infested trees even in very inaccessible terrain. A common problem with the use of UAVs in forest vegetation is the high risk of collision with tree branches or other natural obstacles, which has not yet been reliably addressed. Another problem in complicated locations is the possible unavailability of GPS or GNSS signal to detect the movement of the UAV, which Is otherwise essential for its orientation. Thus, It is essential for the present method that the UAV can move Independently of the GPS or GNSS signa! in the forest vegetation and that it can enter this vegetation in a suitably selected location that allows the UAV to fly in between the trees. However, the first prerequisite for the successful use of UAVs is a high level of reliability of the detection method, which will be used to identify suitable areas with the presence of bark beetleinfested trees. Another problem to be solved is the ability of the UAV to navigate through the undergrowth to the destination and remember this path even in reduced visibility conditions. Last but not least, the UAV must be able to reliably identify and mark an infested tree, for which it must be equipped with means to identify the infested tree using common markers or signs found usually in bark beetle infested trees, as well as be able to mark the tree appropriately for further recognition of the marked tree by the field worker.

[0007] For the purpose of further description, the UAV is also hereafter described as a vehicle or drone and these terms are considered equivalent.[0O0S] The present invention solves the above problems comprehensively and efficiently, advantageously using at least two UAVs, wherein one UAV surveys the terrain and identifies suitable locations for marking and for entry of the second UAV into the forest stand, wherein this UAV may also, be used to transport a smaller UAV to the area, This significantly increases the operational range of this group of UAVs, and the second UAV focuses on carrying out the steps of traversing the forest stand to the destination, locating the infested individuals and reliably marking them for further intervention by an authorized officer who is usually in charge of the forest stand intervention or conducting further reconnaissance of the infested area.

[0009] In one embodiment of the method, information on the state of the forest obtained from earlier measurements may already be employed, and therefore no further survey is necessary, provided that this information is valid in terms of the time frame in which changes to the stand due to the infesting pest may occur.

[0010] The technical problem to be solved by the present invention is to provide a readable, efficient, robust and swift method of searching and marking bark beetle infested trees, which may be utilized in the most variable and complex terrain conditions, in particular in areas with no or unreliable GPS or GNSS signal, and allows for prompt and targeted intervention for pest eradication.

[0011] The goal of the present invention is achieved by the technical solution described in this section.

[0012] As suggested above, the presented invention relates to a unique method for marking bark beetle infested trees in a forest, which utilizes an autonomous unmanned aerial vehicle (UAV) equipped with advanced navigation and sensing technologies to identify and mark the infested trees accurately. GNSS receivers do not operate correctly inside dense forest, since the signal is blocked by the trees. This prevents humans from easily reaching and inspecting pre-designated GNSS coordinates with possible bark beetle infestations, which were acquired by e.g. aerial mapping. The presented process overcomes this issue by utilizing an autonomous UAV that can accurately navigate through the GNSS-denied dense forest, combining data from its onboard sensors and algorithms to reach the specified GNSS coordinates and inspect for infestations. The invention comprises the following basic steps, which will be further elaborated in their advantageous embodiments, in particular by including additional advantageous sub-steps or conditions for their implementation, consisting for example in the use of specific hardware and / or software means:

[0013] The method comprises following essential steps:Step A: Acquiring or using already acquired map of the forest and selecting the GNSS coordinates of potential infested areas and possible entry points into the forest for the UAV;Step B: Deployment of autonomous UAV in the nearest entry point with GNSS localization and integration of localization and mapping information from onboard SLAM algorithms with the GNSS coordinates of its entry point, to enter and navigate the dense forest and avoid obstacles while reaching the coordinates of the possible bark beetle infestation;Step C: Inspection of the trees by UAV using onboard cameras and CNN computer vision algorithms, to identify characteristic marks of bark beetle infestation and verification of the infestation;Step D: Marking the infested tree by UAV using marking means, preferably with paint dispensed from an onboard paint dispenser, but any other suitable method of marking can be used, such as RFID tagging, etc. This is followed by verification of the marking by using its onboard cameras and CNN algorithms.

[0014] By employing the above method, an efficient, accurate and cost-effective solution for detecting and marking bark beetle infestation in forests is achieved.

[0015] In a more specific embodiment of this invention, the method employs in step A use of specific map data obtained by scanning the forest from a height using equipment that only scans a certain spectrum, wherein, for the purpose of the present method, only the part of the spectrum that is between 0.1 μm and 1 m is used in the step of obtaining a relevant map base for the identification of infested trees, because the atmosphere absorbs and scatters electromagnetic radiation. In the above part of the spectrum, the following radiations are used: a. Visible radiation - wavelength 0,4 to 0,7 μm, which is usable in daylight and also under good atmospheric conditions. This radiation penetrates the aquatic environment and can be used in applications on water bodies; b. Near infrared (NIR) radiation = wavelength 0.7 to 1.4 μm, litle absorbed by the atmosphere, which can be used to obtain better quality images suitable for mapping. Water here behaves as a completely black body. NIR is suitable for imaging vegetation; c. Mid-infrared (MIR) radiation = wavelength 1.4 to 3 μm. This wavelength can distinguish different vegetation.

[0016] Advantageously, the device for performing image capturing is Agrowing A7RivA Sextuple muitispectral camera, which is 14-channei and works with six wavelength filters.

[0017] By using multispectral camera imaging, healthy plants can be well identified from the images from pest-infested plants by the changes in plant chlorophyll colour in the false colour photograph in the multispectral camera images.

[0018] Advantageously, the step A is performed by an UAV which is different to the UAV for performing steps B to D. Therefore, the method advantageously employs use of at least two different types of UAVs, each of which are adapted to execute different tasks of the method.

[0019] The UAV for executing step A (further also as “mapping UAV”) is adapted to cover large areas and perform longer flight time. For this purpose, the UAV is provided with an onboard gasoline-powered generator, which supplements the standard lithium batteries used to power the UAV. Gasoline has much higher energy density than lithium batteries. Preferably, the UAV has large 30-inch propellers to support the high payload capacity needed to carry the gasoline-powered generator. For the purposes of scanning and creating the map of the forest, the mapping UAV is using an accurate GNSS receiver and a multi-spectral camera described above, coupled with advanced onboard computer vision algorithms. These are used to generate a map of the forest with coordinates of possible bark beetle infestations and convenient entry points. The mapping UAV is configured to transmit he coordinates to the marking UAV for executing following steps.

[0020] In another preferred embodiment, the mapping UAV is provided with means for transporting the UAV which is adapted for executing steps B to D (further also as “marking UAV”) and wherein the mapping UAV has substantially smaller size in order to safely navigate through the forest. The combination of these two types of UAV results in more efficient execution of the method due to large covered area by the mapping UAV and better environment permeability by the marking UAV.

[0021] In a more specific embodiment of this invention, the method employs in step B use of data obtained from 3D scanning lidar, which are coupled with onboard SLAM algorithm to provide localization for the UAV and construct an accurate 3D map of the surroundingenvironment. Path planning algorithms are used by the UAV to autonomously navigate through the forest white avoiding obstacles. By combining the GNSS coordinates of its entry point and the information provided by the SLAM algorithms, the UAV is able to navigate through the GNSS denied environment to the pre-designated area of possible bark beetle infestation.

[0022] It is essential that the marking UAV is provided with a GNSS receiver, 3D scanning lidar for localization and mapping, an RGB camera, an onboard computer and a marking means. The UAV can also be equipped with an infrared camera or with onboard lights, to enable imaging of the trees even in low-light conditions.

[0023] More advantageously, the UAV for performing step B is different to the UAV performing previous step A. However, it shall be understood that all steps A to D can be carried out by a single UAV which is provided with required means for performing the method according to this invention.

[0024] In a more specific embodiment of this invention, the method employs in step C includes autonomous imaging of the suspected trees using onboard RGB camera and CNN-based computer vision algorithms to detect characteristic marks of bark beetle infestation. The RGB camera is supplemented with onboard lights, which illuminate the trees if the natural conditions are too dark. The camera uses an onboard exposure and gain setting algorithm to produce a good quality image. If the required exposure is too long, onboard lights are used to illuminate the scene. The RGB camera can be also complemented with an infrared camera, which can produce good quality images in the dark even without the usage of onboard lights.

[0025] In a more specific embodiment of this Invention, the method employs in step D use of paint dispenser in a form of a refill spray can serving as a marking means, which is configured to allow variable application of a variety of spraying and marking fluids and with repeated use thereof, wherein the. volume of waste produced is minimized compared to disposable sprays. The refill spray can is activated by a signal from the drone that controlsthe serve. This servo also uses a rod to mechanically trigger the marking process. The pressure vessel is pressurized by a valve using a compressor at a pressure of 10 bar. Thanks to the possibility of re-pressurization, the marking fluid can be fully utilized.

[0026] Furthermore, with regards to step D, the visual servoing techniques to align its paint dispenser with the tree is employed. The infested tree is identified in the camera image with computer vision algorithms and the UAV switches to a visual-based control scheme to align its paint dispenser with the tree precisely.

[0027] Steps C and D are preferably performed repeatedly in the same location until all suspicious trees are checked and marked, if the infestation is visually confirmed.

[0028] Once it is verified that each of the infested tree is correctly marked by onboard camera and CNN algorithms and the whole area is inspected for potentially infested trees, the UAV will fly out of the forest using its previously generated map according to step B. Once the UAV reaches the forest entry point, it will switch back to GNSS localization and return to its base.DETAILED DESCRIPTION OF THE INVENTION

[0029] The present invention is further described by the following examples, which should not be construed as limiting the scope of the invention.

[0030] The example of the method employs two unmanned aerial vehicles (UAVs), each designed for specific subtasks of the application. Both the UAVs are quadcopters, powered by four motors with propellers, using lithium batteries as their power sources.

[0031] The first vehicle is used to acquire the map of the forest and identify possible areas of infestation as well as entry points into the forest (clearings or areas with less dense tree concentrations). The UAV needs a long flight time to map large areas of the forest efficiently. This is facilitated by an onboard gasoline-powered generator, whichsupplements the standard lithium batteries used to power the UAV. Gasoline has much higher energy density than lithium batteries, which enables longer flight time. The UAV has large 30-inch propellers to support the high payload capacity needed to carry the gasoline-powered generator. The power produced by the generator is used to directly power the motors of the UAV and to recharge the onboard lithium bateries. The batteries can cover power spikes caused by fast maneuvers of the UAV.

[0032] For the purposes of scanning and creating the map of the forest, the UAV is using an accurate GNSS receiver and a multi-spectral camera coupled with advanced onboard computer vision algorithms. These are used to generate a map of the forest with coordinates of possible bark beetle infestations and convenient entry points.

[0033] These coordinates are then transmitted to a smaller UAV which is optimized to fly directly into the forest, confirm the bark beetle infestation hypothesis and mark the infested trees with paint. This smaller UAV is using regular lithium batteries as its power source and has smaller 15-inch propellers. It carries a GNSS receiver, 3D scanning lidar for localization and mapping, an RGB camera, a powerful onboard computer and a lightweight paint dispenser. The smaller dimensions of the UAV are advantageous in the constrained area of the forest.

[0034] The UAV is using its GNSS receiver for operations outside of the forest, where GNSS signal is still available. When transitioning into the GNSS denied forest environment, the UAV is using data from its 3D scanning lidar, which are coupled with onboard SLAM algorithm to provide localization for the UAV and construct an accurate 3D map of the surrounding environment.

[0035] The small UAV is deployed from its base near the forest once it receives the coordinates of the possible infestation and the forest entry point from the large mapping UAV. The small UAV navigates to the entry point using GNSS localization. It then enters the forest and switches to GNSS-denied localization. The small UAV is using an onboard 3D scanning lidar and SLAM algorithms to localize itself and construct an accurate mapof the surrounding environment. Path planning algorithms are used by the UAV to autonomously navigate through the forest while avoiding obstacles. By combining the GNSS coordinates of its entry point and the information provided by the SLAM algorithms, the UAV is able to navigate through the GNSS denied environment to the pre-designated area of possible bark beetle infestation.

[0036] Once the designated area is reached, the UAV will autonomously image the suspected trees with its onboard RGB camera and use CNN-based computer vision algorithms to detect characteristic marks of bark beetle infestation. If these marks are detected, the UAV will employ visual servoing techniques to align its paint dispenser with the tree and proceed to mark the tree with paint. Onboard camera and CNN algorithms are then used to verify that the tree was marked correctly. After the whole area is inspected and infested trees are marked, the UAV will fly out of the forest using its previously generated map. Once the UAV reaches the forest entry point, it will switch back to GNSS localization and return to its base.INDUSTRIAL UTILIZATION

[0037] The present invention is industrially utilizable particularly in the field of forestry and forestry plant care, while allowing to perform fast and effective intervention against tree pests, especially in locations with reduced or no availability of satellite signal or otherwise complicated environment making it difficult for forest worker to navigate through.

Claims

AMENDED CLAIMS received by the International Bureau on10 October 2024 (10.10.2024)CLAIMS - AMENDED UNDER ART 19 PCT / 30.08.20241. A method of searching and marking bark beetle infested trees which comprises following steps:A) acquiring or using already acquired map of a forest and selecting a GNSS coordinates of potential infested areas and possible entry points into the forest for an autonomous marking UAV;B) deployment of the autonomous marking UAV in the nearest entry point with GNSS localization and integration of localization and mapping information from onboard SLAM algorithms with the GNSS coordinates of its entry point, to enter and navigate the forest and avoid obstacles while reaching the coordinates of the possible bark beetle infestation in the potential infested areas;C) inspection of the trees by marking UAV using onboard cameras and CNN computer vision algorithms, to identify characteristic marks of bark beetle infestation and verification of the infestation;D) marking the infested tree by marking UAV using marking means, preferably with paint dispensed from an onboard paint dispenser, and verification of the marking using its onboard cameras and CNN algorithms, characterized in that mapping of the forest and selection of infested areas and entry points according to step A are performed by a separate mapping UAV specifically designed for mapping purposes.

2. The method of claim 1 , wherein acquiring map of the forest is performed by scanning the forest from a height using equipment that only scans a part of the spectrum that is between 0.1 μm and 1 m, wherein in visible radiation spectrum the range of wavelength 0,4 to 0,7 μm is used and in near infrared radiation spectrum the range of wavelength 0.7 to 1.4 μm is used and in mid-infrared radiation spectrum the range of wavelength 1.4 to 3 μm is used.

3. The method of any of claims 1 to 2, wherein the mapping UAV is equipped with a gasoline-powered generator, a multi-spectral camera, and advanced computer vision algorithms.

4. The method of claims 1 , wherein the marking UAV used for navigation and inspection is equipped with a GNSS receiver, 3D scanning lidar, RGB camera, onboard computer running CNN algorithms and means for marking a tree.

5. The method of claim 4, wherein the marking UAV is further equipped with infrared cameras and / or onboard lights for imaging trees in low-light conditions.

6. The method of any of claims 1 , 4 and 5, wherein the inspection of the trees is performed using onboard RGB cameras and CNN-based computer vision algorithms.

7. The method of claim 1 , further comprising using visual servoing techniques to align the paint dispenser with the infested tree for precise marking.

8. The method of any of claims 1 to 7, wherein the steps of inspecting and marking are repeated in the same location until all suspicious trees are checked and marked.

9. An UAV having means adapted to execute the steps B, C and D of the method of claims 1.

10. The UAV according to claim 9, characterized in that the UAV is equipped with a GNSS receiver, 3D scanning lidar, RGB camera, onboard computer running CNN algorithms and means for marking a tree.11 . The UAV according to claim 10, characterized in that the UAV is further equipped with infrared cameras and / or onboard lights for imaging trees in low-light conditions.

12. An UAV having means adapted to execute the mapping of the forest and selection of infested areas and entry points of the method of claim 1 .

13. The UAV according to claim 12, characterized in that the UAV is equipped with a gasoline-powered generator, a multi-spectral camera, and advanced computer vision algorithms.

14. A computer program comprising instructions to cause the devices of claims 9 to 13 to execute the steps of the method of claims 1 to 8.