System and method for automated timber inventory

Abstract

A system and method for automated timber inventory. The system and method use photogrammetry and an artificial neural network to produce digital maps and models of highly accurate tree canopy delineation, species prediction, tree height modeling, and subsequent merchantable timber values. In one aspect, a method for automated timber inventory involves acquiring a plurality of aerial images for a timber stand of interest, photogrammetrically generating a three-dimensional representation of the timber stand of interest (e.g., an orthomosaic map, a reflectance map, a digital terrain model, a digital surface model, or any combination of the same) from the plurality of aerial images, identifying one or more tree canopies in the three-dimensional representation using an iteratively-improving artificial intelligence convolutional neural network, and generating a digital canopy map of the timber stand of interest using the convolutional neural network, where the digital canopy map includes labels for the tree canopies.

Description

SYSTEM AND METHOD FOR AUTOMATED TIMBER INVENTORYCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 510,650 filed on June 28, 2023, and incorporates the provisional application by reference in its entirety into this document as if fully set out at this point.BACKGROUND OF THE INVENTION1. Field of the Invention.

[0002] This invention generally relates to a system and method for automated timber inventory and, more particularly, to a system and method for automated timber inventory using photogrammetry and an artificial neural network to produce digital maps and models of highly accurate tree canopy delineation, species prediction, tree height modeling, and subsequent merchantable timber values.2. Description of the Related Art.

[0003] Proper management and quantification of forests and timber stands are essential for sustainable ecosystems, carbon modeling, and accurate marketing in harvest efforts. Traditional timber cruising is excessively laborious, susceptible to human errors, and results in only a small subsample of inventory being quantified. Furthermore, manual cruising evaluations are expensive, limiting inventory’ efforts to final harvesting or thinning events, which can justify their cost or grant funding.SUMMARY OF THE INVENTION

[0004] It is, therefore, desirable to provide an improved system and method for automated timber inventory.

[0005] It is further desirable to provide a system and method for automated timber inventory that is more efficient, more comprehensive, less labor intensive, and more economical in its workflow for properly quantifying forest stands.

[0006] According to one aspect, the invention relates to a method for automated timber inventory that includes the steps of acquiring a plurality of aerial images for a timber stand of interest and photogrammetrically generating a three-dimensional representation of the timber stand of interest from the plurality of aerial images, where the three-dimensional representation is an orthomosaic map, a reflectance map, a digital terrain model, a digital surface model, or any combination of the same. The method further includes the steps of identifying one or more tree canopies in the three-dimensional representation using an iteratively-improving artificial intelligence convolutional neural network, generating a digital canopy map of the timber stand of interest with labels for each of the one or more tree canopies using the convolutional neural network, and outputting the digital canopy map to an electronic display for a user.

[0007] In an embodiment, the labels for each of the one or more tree canopies on the digital canopy map provide tree canopy delineation, species prediction, tree height modeling, merchantable timber values, or a combination thereof.

[0008] In an embodiment, the step of identifying the one or more tree canopies in the three-dimensional representation further includes traversing the three-dimensional representation with the convolutional neural network using predetermined pixel frames, predicting the presence of each of the one or more tree canopies in the three-dimensional representation with an associated confidence value, and annotating the three-dimensional representation to place a bounding box over each of the one or more tree canopies.

[0009] In an embodiment, the step of generating the digital canopy map of the timber stand of interest further involves filtering out the bounding boxes for each of the one or more tree canopies for which the associated confidence value is below a threshold confidence level.

[0010] In an embodiment, the method further involves the step of overlaying the aerial images and the digital canopy map to inspect the identification of the one or more tree canopies by the convolutional neural network for accuracy.

[0011] In an embodiment, the method further involves the step of pre-training the convolutional neural network using a deep learning instance segmentation technique that performs pixel-level segmentation on detected timber stands.

[0012] In an embodiment, the plurality7of aerial images includes a first set of aerial images in leaf-off conditions and a second set of aerial images in leaf-on conditions, and the method further involves the steps of using the first set of aerial images in leaf-off conditions to produce a digital terrain model of the forest floor, using the second set of aerial images in leaf- on conditions to produce a digital surface model of the one or more tree canopies, and calculating tree height estimates for the one or more tree canopies using the digital terrain model and the digital surface model.

[0013] In an embodiment, the step of calculating tree height estimates further involves calculating the difference between the digital surface model and the digital terrain model as raster data and identifying the highest pixel values in the raster data as the highest elevations for the one or more tree canopies.

[0014] According to another aspect, the invention relates to a method for automated timber inventory that includes the steps of training a convolutional neural network using a deep learning instance segmentation technique that performs pixel-level segmentation on detected timber stands, acquiring aerial images for a timber stand of interest, and photogrammetricallygenerating a three-dimensional representation of the timber stand of interest from the aerial images, where the three-dimensional representation is an orthomosaic map. a reflectance map, a digital terrain model, a digital surface model, or any combination of the same. The method traverses the three-dimensional representation with the convolutional neural network using predetermined pixel frames, predicts the presence of each of the one or more tree canopies in the three-dimensional representation with an associated confidence value, annotates the three- dimensional representation to place a bounding box over each of the one or more tree canopies, and generates a digital canopy map of the timber stand of interest using the convolutional neural network, where the digital canopy map includes labels for each of the one or more tree canopies. The method then includes outputting the digital canopy map to an electronic display for a user.

[0015] According to another aspect, the invention relates to a system for automated timber inventory that includes a computer with a processer and a memory' having a software module stored therein. The software module includes executable instructions that, when executed by the processor, cause the processor to acquire a plurality of aerial images for a timber stand of interest, photogrammetrically generate a three-dimensional representation of the timber stand from the plurality of aerial images, and identifies one or more tree canopies in the three-dimensional representation using an iteratively-improving artificial intelligence convolutional neural network (CNN). The three-dimensional representation is an orthomosaic map, a reflectance map, a digital terrain model, a digital surface model, or any combination of the same. The executable instructions further cause the processor to generate a digital canopy map of the timber stand with labels for the one or more tree canopies using the convolutional neural network, and output the digital canopy map to an electronic display for a user.

[0016] In an embodiment, the labels for each of the one or more tree canopies on the digital canopy map provide tree canopy delineation, species prediction, tree height modeling, merchantable timber values, or any combination thereof.

[0017] In an embodiment, the processor identifies the one or more tree canopies in the three-dimensional representation by traversing the three-dimensional representation with the convolutional neural network using predetermined pixel frames, predicting the presence of each of the one or more tree canopies in the three-dimensional representation with an associated confidence value, and annotating the three-dimensional representation to place a bounding box over each of the one or more tree canopies.

[0018] In an embodiment, the processor generates the digital canopy map by filtering out the bounding boxes for each of the one or more tree canopies for which the associated confidence value is below a threshold confidence level.

[0019] In an embodiment, the executable instructions further cause the processor to overlay the plurality of aerial images and the digital canopy map to inspect the identification of the one or more tree canopies by the convolutional neural network for accuracy.

[0020] In an embodiment, the system also includes an aerial remote sensing unit that is configured to acquire the plurality of aerial images that is acquired by the processor, optionally by traversing a flight path above the timber stand and collecting the aerial images at a plurality of photographic positions along the flight path.

[0021] In an embodiment, each of the plurality' of photographic positions is equally spaced apart and the aerial images that are collected from the plurality of photographic positions overlap.

[0022] In an embodiment, the plurality of aerial images includes a first set of aerial images in leaf-off conditions and a second set of aerial images in leaf-on conditions.

[0023] In an embodiment, the executable instructions further cause the processor to photogrammetrically generate the three-dimensional representation by using the first set of aerial images in leaf-off conditions to produce a digital terrain model of the forest floor, using the second set of aerial images in leaf-on conditions to produce a digital surface model of the one or more tree canopies, and calculating tree height estimates for the one or more tree canopies using the digital terrain model and the digital surface model.

[0024] In an embodiment, the step of calculating tree height estimates includes calculating the difference between the digital surface model and the digital terrain model as raster data and identifying the highest pixel values in the raster data as the highest elevations for the one or more tree canopies.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The implementations disclosed herein are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0026] Figure 1 is a flow chart illustrating an operating logic suitable for use in a system and method in accordance with an illustrative embodiment of the invention disclosed herein.

[0027] Figure 2 is overlapping digital aerial imagery of tree stand of interest having key and / or tie points of common objects or locations within the overlapping areas between adjacent digital aerial images in accordance with an illustrative embodiment of the invention disclosed herein.

[0028] Figure 3 is an example for a plurality of aerial images for a timber stand as displayed by a data visualization component in accordance with an illustrative embodiment of the invention disclosed herein.

[0029] Figure 4 is a three-dimensional orthomosaic map that has been photogrammetrically produced from key and / or tie points of the digital aerial imagery of Figure 2 in accordance with an illustrative embodiment of the invention disclosed herein.

[0030] Figures 5A and 5B are manually annotated tree canopies for pre-training a convolutional neural network in accordance with an illustrative embodiment of the invention disclosed herein.

[0031] Figure 6 is a graphical illustration of a custom Mask Region-based Convolutional Neural Network (Mask R-CNN) object detection model trained in a data visualization component via transfer learning in accordance with an illustrative embodiment of the invention disclosed herein.

[0032] Figures 7A and 7B depict a digital canopy map of a timber stand of interest with labeled and georeferenced polygons (Fig. 7A) and a corresponding excerpt from a table of confidence values representing individual tree canopy delineations (Fig. 7B) in accordance with an illustrative embodiment of the invention disclosed herein.

[0033] Figure 8 is a digital canopy map of a timber stand of interest with labeled and georeferenced polygons representing individual tree species in accordance with an illustrative embodiment of the invention disclosed herein.

[0034] Figure 9 is a digital canopy map with labeled and georeferenced polygons representing individual tree heights in accordance with an illustrative embodiment of the invention disclosed herein.

[0035] Figure 10 is a block diagram illustrating components of a machine, according to some example embodiments, able to read instructions from a machine-readable medium and perform any one or more of the methodologies discussed herein.DETAILED DESCRIPTION OF THE INVENTION

[0036] The description that follows describes systems, methods, techniques, instruction sequences, and computing machine program products that illustrate exemplary' embodiments of the present subject matter. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the present subject matter. It will be evident, however, to those skilled in the art that embodiments of the present subject matter may be practiced without some or other of these specific details. Examples merely typify possible variations. Unless explicitly stated otherwise, structures (e.g., structural components, such as modules) are optional and may be combined or subdivided, and operations (e.g., in a procedure, algorithm, or other function) may vary in sequence or be combined or subdivided.

[0037] Example methods (e.g., algorithms) and systems (e.g., special-purpose machines) facilitate the automated timber cruising efforts (data collection), individual tree counts, species classifications, and models of timber volumes. In particular, example embodiments provide mechanisms and logic that generate one or more interactive maps or other representations for automated timber inventory. Accordingly, the mechanisms and logic of example systems and methods acquire digital aerial images of a tree stand of interest from one or more data sources (e.g., manned or unmanned aerial vehicles / systems or high-resolution satellite imagery). The mechanisms and logic then photogrammetrically transform the acquired tree stand-related data into orthomosaic and / or reflectance maps. The mechanism and logic then algorithmically using an iteratively -improving artificial intelligence convolutional neural network (CNN) to generate the interactive maps or other representations from the trained data representing individual tree canopies, height data, and predicted species for the timber stand.Suitable timber stands include evergreen stands and deciduous stands.

[0038] As a result, one or more of the methodologies described herein facilitate solving the technical problem of providing an interactive harvest / navigation / logistics map for each stand to intelligently inform harvest efforts and minimize damage to non-target trees. As such, one or more of the methodologies described herein may obviate a need for certain efforts or resources that otherwise would be involved in traditional manual timber cruising in order to determine the number of harvestable trees, species-specific predictions of stands, and predictions on total volumes.

[0039] The information produced through the inventive system and method provides an accurate visual and numerical representation of the tree stand from numerical computation and spatial modeling using algorithms based on state-of-the-art machine learning and computational statistics techniques. The algorithms composing this inventive system and method automatically find patterns in the tree canopies, which are subsequently used to predict individual tree canopies, height data, and predicted species for the timber stand. Moreover, the algorithms leverage modem computational resources and / or architectures, like distributed computing environments. The inventive system and method produce timber stand maps for large areas without the high cost associated with fieldw ork associated with traditional manual timber cruising. The use of machine learning and computational statistics algorithms allows for the precise modeling of timber stands under the computational process. In addition, byproducts of the system and method can be used to produce individual tree log and / or speciesspecific predictions and can be used to inform automated harvest machinery. Additionally, the inventive system and process support integrative environmental resource management and sustainable ecosystem management by providing precise timber stand estimates across space, which can be used to inform carbon models, quantify merchantable timber, or model ecosystems.

[0040] Figure 1 is a block diagram illustrating an operating logic and components of the inventive system (e.g., machine 1000) and method 100 for automated timber inventory . The method 100 initially involves a step 102 of taking, collecting, accessing, or otherwise acquiring a plurality of aerial images for a timber stand of interest, where the plurality of aerial images may include one or more digital images, still frames, video images, or any combination of the same. The aerial images may be manually taken, collected, accessed, or acquired using an aerial remote sensing unit, such as a manned aerial vehicle, an unmanned aerial vehicle (UAV), an unmanned aircraft system (UAS), or a satellite. If the method 100 is utilized with video images of the timber stand, video frames can be extracted and processed as provided herein. Preferably, the images will be high-definition digital images (e.g., 24MB digital images), but other arrangements are certainly possible (e.g., 4k or higher video images).

[0041] As is generally indicated in Figure 2, each of the photographic positions may be equally spaced apart. The series of aerial images are then taken above the timber stand of interest, such as by a flight path over the area of interest that is traversed using an aerial remote sensing unit with a red-green-blue (RGB) or multispectral sensor collecting overlapping raw imagery. To increase accuracy, step 102 of the method 100 can include acquiring raw digital images with substantial overlap between successive images, e.g., as much as a 90% overlap.

[0042] After the aerial images have been acquired (or, in some cases, during their acquisition), the plurality of aerial images will be made accessible to a system (e g., machine 1 00). Depending on the camera and the equipment associated with it, each aerial image might be transferred wirelessly as it is required to the cloud where it can be accessed by the system later. In other cases, the aerial images might be copied from the camera’s memory card to the server wirelessly or by placing the memory card in direct communication with the system. Theaerial images are optionally displayed separately or in combination to a user (see Figure 3) using a data visualization component of the system.

[0043] After the aerial images have been made accessible to the server, the system processes the aerial images and can prepare them for use in determining the edges of the identified timber stand. The method 100 can further include electronically transforming, calibrating, and correcting the aerial images for tilt, exposure, distortion, making, color, jittering, shape, size, or a combination thereof.

[0044] Turning to Figure 1, the system uses the aerial images to photogrammetrically generate one or more three-dimensional representations of the timber stand under step 104, step 106, or both (in sequence or simultaneously). In step 104, the system executes photogrammetric software to process the plurality' of aerial images into an orthomosaic map, a reflectance map, or both (the “map layer(s)'’). As shown in Figure 4, in one embodiment, the raw aerial imagery from the flight path of Figure 2 is orthorectified into a single seamless orthomosaic map layer with perspective distortion removed. This orthorectification is completed by using a photogrammetric process, such as a scale-invariant feature transform (SIFT) algorithm or similar algorithm, that identifies features and selects key points shared between adjacent aerial images to reconstruct each in 3D space as a three-dimensional orthomosaic or reflectance map. The resulting single map layer is an accurate nadir representation of field objects that are georeferenced with real-time kinematic (RTK)-level precision. It will be appreciated that the aerial images acquired in step 102 may be orthophotos for which distortions have been previously removed, or the aerial images may be converted to orthophotos after they are acquired in step 102.

[0045] The map layer(s) generated at step 102 are used to perform tree canopy analysis at step 108 by executing either step 108a or step 108b. In step 108a, objects of interest(individual tree canopies) are identified in the single map layer by using the iteratively- improving artificial intelligence CNN. The CNN traverses over the single map layer using predetermined pixel frames (e.g.. 256 x 256, 1024 x 1024) placing bounding boxes over inferred objects with associated confidence values with each prediction. It will be appreciated that the confidence values may be provided proximate to the respective bounding boxes or within a separate table. The CNN is pre-trained using a Mask Region-based Convolutional Neural Network (Mask R-CNN) deep learning instance segmentation technique that performs pixel-level segmentation on detected timber stands (e.g., mixed and planted timber stands of similar row' spacing and growth stage) (Figure 6). Alternatively, the CNN is pre-trained using manually annotated tree canopies, as generally shown in Figures 5A and 5B. Results from the customized CNN consist of a layer of labeled and georeferenced 2D bounding boxes (e.g., polygons) (Figure 7A) with confidence values representing individual tree canopy delineations (Figure 7B) (a “digital canopy map”). Overlaying the aerial imagery map layer(s), the predictions for the digital canopy map can be inspected for accuracy. Filtering by confidence values can adjust the sensitivity of the digital canopy map layer to reduce the number of false positives or to include true positives of low' confidence. For example, bounding boxes on the digital canopy map that have an associated confidence value below a threshold confidence level may be filtered out. In alternative step 108b, a first customized CNN is used for individual tree canopy identification, while a second customized CNN can be used to label the delineated tree canopies with species attributes or to otherwise provide species prediction for the individual tree canopies. For example, as shown in the digital canopy map of Figure 8, red and white oak species are differentiated by labeling red oak species with a red 2D bounding box, while white oak species are delineated with a white 2D bounding box.

[0046] Turning back to step 106. in another embodiment, two flights of the aerial remote sensing unit are used to collect two sets of aerial images for the tree stand of interest, one in leaf-off conditions to produce a digital terrain model of the forest floor and the other in leaf-on conditions to produce a digital surface model of the tree canopy. As shown in Figure 9, the dual set of raw aerial imagery' can be photogrammetrically processed at step 106, resulting in a raster layer of tree canopy heights. More particularly, the raster data may represent the difference between the digital surface model and the digital terrain model. At step 110, tree height estimates can be calculated and used in combination with CNN model outputs or solely to make timber estimates. More particularly, the raster layer may be masked with labeled and georeferenced 2D bounding boxes (e.g., polygons) that represent individual tree canopies, and the highest pixel values in the raster data (e.g., within each of the 2D bounding boxes) are identified as the highest elevations for the respective tree canopies. It will be appreciated that, where both steps 108 and 110 are performed, the 2D bounding boxes that are produced in step 108 may be used for the raster data evaluation in step 110. Alternatively, 2D bounding boxes may be generated separately for the raster layer according to the labeling process outlined above for step 108.

[0047] For an abundance of clarity, where both step 104 and step 106 are performed in method 100, steps 108 and 1 10 are both performed (simultaneously or in sequence). Steps 104 and 108 may be performed before step 106 commences, steps 106 and 110 may be performed before step 104 commences, or steps 104 and 106 may be performed before either step 108 or 1 10 commences. Where step 104 is performed but step 106 is not, only step 108 is subsequently performed. Where step 106 is performed but step 104 is not, step 110 is subsequently performed without step 108. The map layer(s) from steps 104 / 108 and the raster layer from steps 106 / 110 are optionally combined and annotated for a single digital canopy map. Alternatively, thebounding box-labeled map layer(s) and raster layer are maintained separately and may be annotated and analyzed as separate digital canopy maps.

[0048] Once the digital canopy map(s), with or without species attribute, and with or without height prediction, are verified and accepted, timber inventory is performed at step 112 and the digital canopy map is optionally output to an electronic display for a user. In one embodiment, a predictive linear regression model is used to correlate bounding boxes (representing delineated surface area for individual tree canopies, predicted species attribute, and / or tree height modeling, merchantable timber values) to inform carbon models, quantify merchantable timber, or model ecosystems. The canopy area attributes can be used in combination with published formulas (e.g., “canopy area to trunk volume calculations”) to predict diameter at breast height (DBH) and other dendrometric measurements. The results of the step 112 analyses may be used to product individual tree logs and / or species-specific predictions reports. The georeferenced information contained in the digital canopy map(s) may also be used to produce a harvest / navigation / logistics map to intelligently inform harvest efforts and minimize damage to non-target trees. Using the delineations of Figure 8 as an example, the harvest / navigation / logistics map might indicate a route to harvest red oak species (target trees) while avoiding damage to white oak species (non-target trees). The digital canopy map or harvest / navigation / logistics map is optionally used to inform either manual or automated harv est machinery.

[0049] It is noted that the inventive system and method 100 shown in Figure 1 is merely an example. Any of the components (e.g., modules, engines, data source) shown in Figure 1 may be, include, or otherwise be implemented in a special-purpose specialized or otherwise non-generic computer that has been modified (e.g., configured or programmed by software, such as one or more software modules of an application, operating system, firmware,middleware, or other program) to perform one or more of the functions described herein for that component, system, or machine. For example, a special-purpose computer system able to implement any one or more of the methodologies described herein is discussed below with respect to Figure 10. and such a special-purpose computer may, accordingly, be a means for performing any one or more of the methodologies discussed herein. Within the technical field of such special-purpose computers, a special-purpose computer that has been modified by the structures discussed herein to perform the functions discussed herein is technically improved compared to other special-purpose computers that lack the structures discussed herein or are otherwise unable to perform the functions discussed herein. Accordingly, a special-purpose machine configured according to the systems and methods discussed herein provides an improvement to the technology7of similar special-purpose machines.

[0050] Any one or more of the components (e.g., modules or engines) described herein may be implemented using hardware alone (e.g., one or more processors of a machine) or a combination of hardware and software. For example, any component described herein may physically include an arrangement of one or more of the processors or configure a processor (e.g., among one or more processors of a machine) to perform the operations described herein for that module. Accordingly, different components described herein may include and configure different arrangements of the processors at different points in time or a single arrangement of the processors at different points in time. Each component (e.g., module or engine) described herein is an example of a means for performing the operations described herein for that component. Moreover, any two or more of these components may be combined into a single component, and the functions described herein for a single component may be subdivided among multiple components. Furthermore, according to various exampleembodiments, components described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0051] Figure 10 illustrates components of a machine 1000, according to some example embodiments, that is able to read instructions from a machine-readable medium (e.g., a machine-readable storage device, a non-transitory machine-readable storage medium, a computer-readable storage medium, or any suitable combination thereol) and perform any one or more of the methodologies discussed herein. Specifically, Figure 10 shows a diagrammatic representation of the machine 1000 in the example form of a computer device (e.g., a computer) and within which instructions 1024 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 1000 to perform any one or more of the methodologies discussed herein may be executed, in whole or in part.

[0052] For example, the instructions 1024 may cause the machine 1000 to execute one or more steps from the flow diagram of Figure 1. In one embodiment, the instructions 1024 can transform the general, non-programmed machine 1000 into a particular machine (e.g., specially configured machine) programmed to carry out the described and illustrated functions in the manner described.

[0053] In alternative embodiments, the machine 1000 operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 1000 may operate in the capacity of a server machine or a client machine in a serverclient network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1000 may be a server computer, a client computer, a personal computer (PC), a tablet computer, alaptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 1024(sequentially or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine’' shall also be taken to include a collection of machines that individually or jointly execute the instructions 1024 to perform any one or more of the methodologies discussed herein.

[0054] The machine 1000 includes a processor 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), or any suitable combination thereof), a main memory 1004, and a static memory 1006, which are configured to communicate with each other via a bus 1008. The processor 1002 may contain microcircuits that are configurable, temporarily, or permanently, by some or all of the instructions 1024 such that the processor 1002 is configurable to perform any one or more of the methodologies described herein, in whole or in part. For example, a set of one or more microcircuits of the processor 1002 may be configurable to execute one or more modules (e.g., software modules) described herein.

[0055] The machine 1000 may further include a graphics display 1010 (e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT), or any other display capable of displaying graphics or video). The machine 1000 may also include an alphanumeric input device 1012 (e.g., a keyboard), a cursor control device 1014 (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instrument), a storage unit 1 1 , a signal generation device 1018 (e.g., a sound card, an amplifier, a speaker, a headphone jack, or any suitable combination thereof), and a network interface device 1020.

[0056] The storage unit 1016 includes a machine-readable medium 1022 (e.g., a tangible machine-readable storage medium) on which is stored the instructions 1024 (e.g..software) embodying any one or more of the methodologies or functions described herein. The instructions 1024 may also reside, completely or at least partially, within the main memory 1004, within the processor 1002 (e.g., within the processor’s cache memory), or both, before or during execution thereof by the machine 1000. Accordingly, the main memory 1004 and the processor 1002 may be considered as machine-readable media (e.g., tangible and non- transitory machine-readable media). The instructions 1024 may be transmitted or received over a network 1026 via the network interface device 1020.

[0057] In some example embodiments, the machine 1000 may be a portable computing device and have one or more additional input components (e.g., sensors or gauges). Examples of such input components include an image input component (e.g., one or more cameras), an audio input component (e.g., a microphone), a direction input component (e.g., a compass), a location input component (e.g., a global positioning system (GPS) receiver), an orientation component (e.g., a gyroscope), a motion detection component (e.g., one or more accelerometers), an altitude detection component (e.g., an altimeter), and a gas detection component (e.g., a gas sensor). Inputs harvested by any one or more of these input components may be accessible and available for use by any of the modules described herein.

[0058] As used herein, the term “memory” refers to a machine-readable medium 1022 able to store data temporarily or permanently and may be taken to include, but not be limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. While the machine-readable medium 1022 is shown in an example embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store instructions (e.g., instructions 1024). The term “machine-readable medium” shall also be taken to include any medium, or combination ofmultiple media, that is capable of storing instructions (e.g., software, a program, an application, an applet, an app, or other executable code) for execution by the machine (e g., machine 1000), such that the instructions, when executed by one or more processors of the machine (e.g., processor 1002), cause the machine to perform any one or more of the methodologies described herein. Accordingly, a “machine-readable medium” refers to a single storage apparatus or device, as well as cloud-based storage systems or storage networks that include multiple storage apparatus or devices. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, one or more data repositories in the form of a solid-state memory, an optical medium, a magnetic medium, or any suitable combination thereof. In some embodiments, a “machine-readable medium” may also be referred to as a “machine-readable storage device” or a “hardware storage device.”

[0059] Furthermore, the machine-readable medium 1022 is non-transitory in that it does not embody a propagating or transitory signal except in jurisdictions where such signals are eligible for protection. However, labeling the machine-readable medium 1022 as “non- transitory” should not be construed to mean that the medium is incapable of movement; the medium should be considered as being transportable from one physical location to another. Additionally, since the machine-readable medium 1022 is tangible, the medium may be considered to be a tangible machine-readable storage device.

[0060] In some example embodiments, the instructions 1024 for execution by the machine 1000 may be communicated by a carrier medium. Examples of such a carrier medium include a storage medium (e.g., a non-transitory machine-readable storage medium, such as a solid-state memory, being physically moved from one place to another place) and a transient medium (e.g., a propagating signal that communicates the instructions 1024).

[0061] The instructions 1024 may further be transmitted or received over a communications network 1026 using a transmission medium via the network interface device 1020 and utilizing any one of a number of well-known transfer protocols, e.g., HTTP). Examples of communication networks 1026 include a local area network (LAN), a wide area network (WAN), the Internet, mobile telephone networks, plain old telephone service (POTS) networks, and wireless data networks (e.g., WiFi, LTE, and WiMAX networks). The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding, or carrying instructions 1024 for execution by the machine 1000, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

[0062] Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

[0063] Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute either software modules (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware modules. A “hardware module” is a tangible unit capable of performing certain operations and may be configured or arranged in a certain physical manner. In various example embodiments.one or more computer systems (e.g.. a standalone computer system, a client computer system, or a server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

[0064] In some embodiments, a hardw are module may be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware module may include dedicated circuitry or logic that is permanently configured to perform certain operations. For example, a hardware module may be a special-purpose processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware module may also include programmable logic or circuitry7that is temporarily configured by softw are to perform certain operations. For example, a hardware module may include software encompassed within a general-purpose processor or other programmable processor. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.

[0065] Accordingly, the term “hardware module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently' configured (e.g., hardwired), or temporarily' configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. As used herein, “hardware-implemented module” refers to a hardware module. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, w'here the hardware modules comprise a general-purpose processor configured by software to become a special-purpose processor, the general-purpose processor may be configured as respectively different hardware modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardw are module at one instance of time and to constitute a different hardw are module at a different instance of time.

[0066] Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) betw een or among two or more of the hardw are modules. In embodiments in w hich multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardw are modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory' device to w hich it is communicatively coupled. A further hardw are module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).

[0067] The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions described herein. As used herein, “processor-implemented module” refers to a hardware module implemented using one or more processors.

[0068] Similarly, the methods described herein may be at least partially processor- implemented, a processor being an example of hardware. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. Moreover, the one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an application program interface (API)).

[0069] The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the one or more processors or processor- implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the one or more processors or processor-implemented modules may be distributed across a number of geographic locations.

[0070] Some portions of this specification may be presented in terms of algorithms or symbolic representations of operations on data stored as bits or binary digital signals within a machine memon (e g., a computer memory). These algorithms or symbolic representations are examples of techniques used by those of ordinary skill in the data processing arts to convey the substance of their work to others skilled in the art. As used herein, an “algorithm” is a self- consistent sequence of operations or similar processing leading to a desired result. In this context, algorithms and operations involve the physical manipulation of physical quantities. Typically, but not necessarily, such quantities may take the form of electrical, magnetic, oroptical signals capable of being stored, accessed, transferred, combined, compared, or otherwise manipulated by a machine. It is convenient at times, principally for reasons of common usage, to refer to such signals using words such as “data,” “content,” “bits,” “values,” “elements,” “symbols,” “characters,” “terms,” “numbers,” “numerals,” or the like. These words, however, are merely convenient labels and are to be associated with appropriate physical quantities.

[0071] Unless specifically stated otherwise, discussions herein using words such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” and the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory', or any suitable combination thereof), registers, or other machine components that receive, store, transmit, or display information. Furthermore, unless specifically stated otherwise, the terms “a” or “an” are herein used, as is common in patent documents, to include one or more than one instance. Finally, as used herein, the conjunction “or” refers to a non-exclusive “or,” unless specifically stated otherwise.

[0072] Although an overview' of the present subject matter has been described with reference to specific example embodiments, various modifications and changes may' be made to these embodiments without departing from the broader scope of embodiments of the present invention. For example, various embodiments or features thereof may be mixed and matched or made optional by a person of ordinary' skill in the art. Such embodiments of the present subject matter may be referred to herein, individually or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or present concept if more than one is, in fact, disclosed.

[0073] The embodiments illustrated herein are believed to be described in sufficient detail to enable those skilled in the art to practice the teachings disclosed. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. The Detailed Description, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

[0074] Moreover, plural instances may be provided for resources, operations, or structures described herein as a single instance. Additionally, boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary', and particular operations are illustrated in a context of specific illustrative configurations. Other allocations of functionality' are envisioned and may fall within a scope of various embodiments of the present invention. In general, structures and functionality' presented as separate resources in the example configurations may be implemented as a combined structure or resource. Similarly, structures and functionality presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within a scope of embodiments of the present invention as represented by the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

WHAT IS CLAIMED IS:

1. A method for automated timber inventory, the method comprising the steps of: a. acquiring a plurality of aerial images for a timber stand of interest; b. photogrammetrically generating a three-dimensional representation of the timber stand of interest from the plurality of aerial images, wherein the three- dimensional representation is an orthomosaic map, a reflectance map, a digital terrain model, a digital surface model, or any combination of the same; c. identifying one or more tree canopies in the three-dimensional representation using an iteratively-improving artificial intelligence convolutional neural network; and d. generating a digital canopy map of the timber stand of interest using the convolutional neural network, wherein the digital canopy map comprises labels for each of the one or more tree canopies.

2. The method of claim 1, wherein the labels for each of the one or more tree canopies on the digital canopy map provide tree canopy delineation, species prediction, tree height modeling, merchantable timber values, or a combination thereof.

3. The method of claim 1 , wherein the step of identifying the one or more tree canopies in the three-dimensional representation further comprises: traversing the three-dimensional representation with the convolutional neural network using predetermined pixel frames; predicting the presence of each of the one or more tree canopies in the three- dimensional representation with an associated confidence value; andannotating the three-dimensional representation to place a bounding box over each of the one or more tree canopies.

4. The method of claim 3, wherein the step of generating the digital canopy map of the timber stand of interest further comprises filtering out the bounding boxes for each of the one or more tree canopies for which the associated confidence value is below a threshold confidence level.

5. The method of claim 1, further comprising the step of overlaying the plurality of aerial images and the digital canopy map to inspect the identification of the one or more tree canopies by the convolutional neural network for accuracy.

6. The method of claim 1, further comprising the step of pre-training the convolutional neural network using a deep learning instance segmentation technique that performs pixel-level segmentation on detected timber stands.

7. The method of claim 1, wherein the plurality of aerial images comprises a first set of aerial images in leaf-off conditions and a second set of aerial images in leaf-on conditions, the method further comprising the steps of: using the first set of aerial images in leaf-off conditions to produce a digital terrain model of the forest floor; using the second set of aerial images in leaf-on conditions to produce a digital surface model of the one or more tree canopies; and8 calculating tree height estimates for the one or more tree canopies using the9 digital terrain model and the digital surface model.1 8. The method of claim 7, wherein the step of calculating tree height estimates further2 comprises:3 calculating the difference between the digital surface model and the digital terrain4 model as raster data; and5 identifying the highest pixel values in the raster data as the highest elevations for the6 one or more tree canopies.1 9. A method for automated timber inventory7, the method comprising the steps of:2 training a convolutional neural network using a deep learning instance3 segmentation technique that performs pixel-level segmentation on detected timber4 stands;5 acquiring aerial images for a timber stand of interest;6 photogrammetrically generating a three-dimensional representation of the7 timber stand of interest from the aerial images, wherein the three-dimensional8 representation is an orthomosaic map, a reflectance map, a digital terrain model, a9 digital surface model, or any combination of the same; 0 traversing the three-dimensional representation with the convolutional neural1 network using predetermined pixel frames; 2 predicting the presence of each of the one or more tree canopies in the three-3 dimensional representation with an associated confidence value;annotating the three-dimensional representation to place a bounding box over each of the one or more tree canopies; generating a digital canopy map of the timber stand of interest using the convolutional neural network, wherein the digital canopy map comprises labels for each of the one or more tree canopies: and outputting the digital canopy map to an electronic display for a user.

10. A system for automated timber inventory', the system comprising: a computer having a processor and a memory'; and a software module stored in the memory7, comprising executable instructions that when executed by the processor cause the processor to: a. acquire a plurality' of aerial images for a timber stand; b. photogrammetrically generate a three-dimensional representation of the timber stand from the plurality of aerial images, wherein the three-dimensional representation is an orthomosaic map, a reflectance map, a digital terrain model, a digital surface model, or any combination of the same; c. identify one or more tree canopies in the three-dimensional representation using an iteratively-improving artificial intelligence convolutional neural network; and d. generate a digital canopy map of the timber stand using the convolutional neural network, wherein the digital canopy map comprises labels for the one or more tree canopies.

11. The system of claim 10, wherein the labels for each of the one or more tree canopies on the digital canopy map provide tree canopy delineation, species prediction, tree height modeling, merchantable timber values, or any combination thereof.

12. The system of claim 10, wherein the processor identifies the one or more tree canopies in the three-dimensional representation by: traversing the three-dimensional representation with the convolutional neural network using predetermined pixel frames; predicting the presence of each of the one or more tree canopies in the three- dimensional representation with an associated confidence value; and annotating the three-dimensional representation to place a bounding box over each of the one or more tree canopies.

13. The system of claim 12, wherein the processor generates the digital canopy map by filtering out the bounding boxes for each of the one or more tree canopies for which the associated confidence value is below a threshold confidence level.

14. The system of claim 10, wherein the executable instructions further cause the processor to overlay the plurality of aerial images and the digital canopy map to inspect the identification of the one or more tree canopies by the convolutional neural network for accuracy.

15. The system of claim 10 further comprising an aerial remote sensing unit configured to acquire the plurality of aerial images that is acquired by the processor.

16. The system of claim 15, wherein the aerial remote sensing unit is configured to acquire the plurality of aerial images by traversing a flight path above the timber stand and collecting the plurality of aerial images at a plurality of photographic positions along the flight path.

17. The system of claim 16, wherein each of the plurality of photographic positions is equally spaced apart and the aerial images that are collected from the plurality of photographic positions overlap.

18. The system of claim 10, wherein the plurality of aerial images comprises a first set of aerial images in leaf-off conditions and a second set of aerial images in leaf-on conditions.

19. The system of claim 18, wherein the executable instructions further cause the processor to photogrammetrically generate the three-dimensional representation by: using the first set of aerial images in leaf-off conditions to produce a digital terrain model of the forest floor; using the second set of aerial images in leaf-on conditions to produce a digital surface model of the one or more tree canopies; and calculating tree height estimates for the one or more tree canopies using the digital terrain model and the digital surface model.

20. The system of claim 19, wherein the step of calculating tree height estimates further comprises: calculating the difference between the digital surface model and the digital terrain model as raster data; and5 identifying the highest pixel values in the raster data as the highest elevations for the one or more tree canopies.

Images