Predicting and correcting vegetation status

Abstract

Methods and systems for managing vegetation include training a machine learning model based on images of training data regions before a weather event, images of the training data regions after the weather event, and information about the weather event. A risk score for a second region is generated using the trained machine learning model based on an image of the second region and predicted weather information for the second region. The risk score is used to indicate high risk vegetation in the second region. A corrective action is performed to reduce the risk of vegetation in the second region.

Description

BACKGROUND

[0001] The present application relates generally to the maintenance of vegetation, and more particularly to predicting and identifying areas where vegetation will overgrow and potentially damage useful areas and needs to be removed.

[0002] While satellite imagery can provide accurate two-dimensional imaging of current vegetation encroachment on, for example, utility, transportation infrastructure, and agriculture, it is difficult to predict how these vegetations will affect these uses during adverse weather. As a result, vegetation management can be a labor-intensive process in which an inspector physically visits the relevant assets to check the type and state of the vegetation and identify vegetation that needs to be removed. SUMMARY

[0003] A method for managing vegetation includes training a machine learning model based on images of training data areas before a weather event, images of the training data areas after the weather event, and information about the weather event. A risk score for a second area is generated using the trained machine learning model based on images of the second area and predicted weather information for the second area. The risk score is used to indicate high risk vegetation in the second area. A corrective action is performed to reduce the risk of vegetation in the second area.

[0004] A method for managing vegetation includes training a machine learning model based on images of training data areas before a weather event, images of the training data areas after the weather event, and information about the weather event, the information about the weather event including a difference between vegetation shown in the images of the training data areas before the weather event and vegetation shown in the images of the training data areas after the weather event. A risk score for a second area is generated using the trained machine learning model based on images of the second area and predicted weather information for the second area. The risk score is used to indicate high risk vegetation in the second area. A corrective action is performed to reduce the risk of vegetation in the second area.

[0005] A system for managing vegetation includes a model trainer configured to train a machine learning model based on images of training data areas before a weather event, images of the training data areas after the weather event, and information about the weather event. A vegetation manager is configured to generate a risk score for a second area using the trained machine learning model based on images of the second area and predicted weather information for the second area to determine that the risk score indicates high risk vegetation in the second area, and trigger a corrective action to reduce the risk of vegetation in the second area.

[0006] A system for managing vegetation, comprising: a model trainer configured to train a machine learning model based on images of a training data region before a weather event, images of the training data region after the weather event, and information about the weather event, the information about the weather event including a difference between vegetation shown in the images of the training data region before the weather event and vegetation shown in the images of the training data region after the weather event. A vegetation manager configured to: generate, using the trained machine learning model, a risk score for a second region based on images of the second region and predicted weather information for the second region, to determine that the risk score indicates a high risk vegetation in the second region, and to trigger a corrective action to reduce the risk of vegetation in the second region.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is achieved in connection with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0008] The following description will provide details of preferred embodiments with reference to the following drawings, in which:

[0009] Figure 1 is a diagram of a neural network model according to embodiments of the present application;

[0010] Figure 2 is a diagram of a neural network architecture according to embodiments of the present application;

[0011] Figure 3 is a block / flow diagram of a method for vegetation management according to embodiments of the present application, the method using a machine learning model to identify a high risk vegetation region;

[0012] Figure 4 is a block / flow diagram of a method for training a machine learning model to identify a high risk vegetation region according to embodiments of the present application;

[0013] Figure 5 is a block / flow diagram of a method for using a machine learning model to identify a high risk vegetation region according to embodiments of the present application;

[0014] Figure 6 is a block diagram of a vegetation management system using a machine learning model to identify and control a high risk vegetation region according to embodiments of the present application;

[0015] Figure 7 is a block diagram of a processing system for vegetation management according to embodiments of the present application; and,

[0016] Figure 8 is a diagram showing comparisons between different elevation models to identify different kinds of terrain and structures in an image according to embodiments of the present application. DETAILED DESCRIPTION

[0017] Embodiments of the present application use measurements of current vegetation conditions, as well as information related to weather and terrain data, to predict areas where vegetation growth can pose a potential danger to assets and people during periods of adverse weather. For example, during strong winds or heavy snow, tree branches can fall on power lines and railroad tracks, requiring emergency repairs and removal. Historical data of damage events, such as the location of damage, the severity of damage, and proximity to man-made structures, power lines, and prevailing weather conditions, can be considered to identify those trees that can cause damage. Additional information, such as the length of power outages, lack of communication services, or damage to houses, can be used to assess the severity of damage that can be caused by trees. Based on expert recommendations and best practices implemented by utility companies and local authorities, and taking into account local regulations, the best practices for vegetation management in a certain area are determined. Additional information, such as tree type, age, health, soil, and surrounding environment, are also integrated into the model. Thus, embodiments of the present application identify vegetation that can cause problems and remove or prune the vegetation before it can cause damage.

[0018] To this end, embodiments of the present application employ machine learning techniques to identify changes in vegetation over time using, for example, satellite images, and combine this information with terrain and weather information to identify areas where vegetation growth can cause damage. Embodiments of the present application use ranging techniques, such as Light Detection and Ranging (LIDAR), to construct a terrain map and distinguish between different kinds of features, such as trees and man-made structures. Embodiments of the present application then create a map that identifies high-risk vegetation that can cause problems under adverse weather conditions or vegetation growth conditions, and generate recommendations to avoid damage by removing the relevant vegetation.

[0019] To identify areas of vegetation that can cause damage, an artificial neural network (ANN) is trained on images that change over time (e.g., indicating the fall of a tree), and then used to identify areas that can suffer damage in the future. During the training process, the entire three-dimensional distribution information from LIDAR data is considered to identify individual trees or parts of trees (branches, crowns, trunks, etc.) that can cause damage. In this LIDAR data, such features can be identified as a reduction in the number of return points (indicating sick or damaged branches), or an excessive number of return points distributed asymmetrically around the tree body (such as overgrown branches), or only the detection of a tree body without leaves indicating a dead tree.

[0020] ANNs are information processing systems inspired by biological nervous systems, such as the brain. A key element of ANNs is the structure of the information processing system, which includes a large number of highly interconnected processing units (referred to as "neurons") working in parallel to solve a particular problem. ANNs are further trained in use through learning, which involves adjusting the weights that exist between the neurons. Through such a learning process, ANNs are configured for a particular application, such as pattern identification or data classification.

[0021] Reference is now made to Figure 1 which shows a general schematic of a neural network. ANNs demonstrate the ability to derive meaning from complex or inaccurate data, and can be used to extract patterns, as well as detect trends that are too complex to be detected by humans or other computer-based systems. Known neural network structures generally have input neurons 102 that provide information to one or more "hidden" neurons 104. The connections 108 between the input neurons 102 and the hidden neurons 104 are weighted, and these weighted inputs are then processed by the hidden neurons 104 according to some function in the hidden neurons 104 and the weighted connections 108 between the layers. There can be any number of layers of hidden neurons 104, as well as neurons that perform different functions. There are also different neural network structures, such as convolutional neural networks, maxout networks, etc. Finally, a set of output neurons 106 accept and process the weighted inputs from the last set of hidden neurons 104.

[0022] This represents a "feedforward" computation, in which information is propagated from the input neurons 102 to the output neurons 106. After the feedforward computation is complete, the output is compared to the desired output available from the training data. Errors associated with the training data are then processed in a "feedback" computation, in which the hidden neurons 104 and the input neurons 102 receive information about the error that is backpropagated from the output neurons 106. Once the backpropagation of error has been completed, weight updates are performed, and the weighted connections 108 are updated to take into account the received error. The above represents only one ANN.

[0023] Reference is now made to Figure 2 which shows an exemplary ANN structure 200. It should be appreciated that the present architecture is merely exemplary, and other architectures or types of neural networks can be used instead. In particular, while hardware embodiments of ANNs are described in the present application, it should be appreciated that neural network architectures can be implemented or simulated in software. Also, the hardware embodiments described in the present application are intended to illustrate the general principles of neural network computation in a high-level generality, and should not be understood as limiting in any way.

[0024] Furthermore, the neuron layers and the weights connecting them described below are described in a general manner and can be replaced by any type of neural network layer having any appropriate degree or type of interconnectivity. For example, the layers can include convolutional layers, pooling layers, fully connected layers, normalization exponential function (softmax) layers, or any other appropriate type of neural network layer. Furthermore, layers can be added or removed as desired, and weights can be omitted for more complex forms of interconnectivity.

[0025] During the feedforward operation, a set of input neurons 202 each provide an input voltage in parallel to a row of respective weights (W) 204. In the hardware embodiment described in this application, the weights 204 each have a settable resistance value such that a current output flows from the weight 204 to the respective hidden neuron 206 to represent the weighted input. In a software embodiment, the weights 204 can simply be represented as coefficient values that are multiplied with the relevant neuron output.

[0026] According to the hardware embodiment, the current output by a given weight 204 is determined as where V is the input voltage from the input neuron 202 and r is the set resistance of the weight 204. The current from each weight is summed column-wise and flows to the hidden neuron 206. A set of reference weights 207 have a fixed resistance and their outputs are combined into a reference current that is provided to each of the hidden neurons 206. Since conductance values can only be positive, some reference conductance is needed to encode both positive and negative values in the matrix. The current produced by the weights 204 is a continuous value and is positive, so the reference weights 207 are used to provide a reference current above which the current is considered positive and below which the current is considered negative. The use of reference weights 207 is not needed in a software embodiment, in which the values of the output and weights can be obtained exactly and directly. As an alternative to using reference weights 207, a separate array of weights 204 can be used in another embodiment to obtain negative values.

[0027] The hidden neurons 206 use the currents from the array of weights 204 and the reference weights 207 to perform certain computations. The hidden neurons 206 then output their own voltage to another array of weights 204. This array performs in the same manner, with a column of weights 204 receiving a voltage from their respective hidden neuron 206 to produce a weighted output current that is summed row-wise and provided to the output neurons 208.

[0028] It will be appreciated that any number of the above steps can be implemented by the insertion of additional array layers and hidden neurons 206. It will also be noted that some neurons can be constant neurons 209 that provide a constant output to the array. Constant neurons 209 can be present among the input neurons 202 and / or hidden neurons 206 and are used only during the forward pass operation.

[0029] During backpropagation, the output neurons 208 provide a voltage that is backpropagated across the array of weights 204. The output layer compares the generated network response to the training data and computes an error. This error is applied to the array as a voltage pulse, where the height and / or duration of the pulse is modulated in proportion to the error value. In this example, a row of weights 204 receives a voltage in parallel from a corresponding output neuron 208 and converts this voltage to a current that is summed column by column to provide input to the hidden neurons 206. The hidden neurons 206 combine the weighted feedback signal with the derivative of their forward pass calculation and store the error value before outputting the feedback signal voltage to their respective column of weights 204. This backpropagation is carried through the entire network 200 until all hidden neurons 206 and input neurons 202 have stored an error value.

[0030] During weight update, the input neurons 202 and hidden neurons 206 apply a first weight update voltage forward through the network 200, and the output neurons 208 and hidden neurons 206 apply a second weight update voltage backward. The combination of these voltages produces a state change within each weight 204, such that the weights 204 assume a new resistance value. In this way, the weights 204 can be trained to adapt the neural network 200 to errors in its processing. It will be noted that these three modes of operation, forward pass, backpropagation, and weight update, do not overlap with one another.

[0031] As mentioned above, the weights 204 can be implemented in software or hardware, for example using relatively complex weighted circuitry or using resistive cross-point devices. Such resistive devices can have non-linear switching characteristics that can be used to process data. The weights 204 can belong to a class of devices known as resistive processing units (RPUs) in that their non-linear characteristics are used to perform calculations in the neural network 200. RPU devices can be implemented with resistive random access memory (RRAM), phase change memory (PCM), programmable metallization cell (PMC) memory, or any other device with non-linear resistive switching characteristics. Such RPU devices can also be considered to be memristive systems.

[0032] Reference is now made to Figure 3which illustrates a method of detecting and removing high-risk vegetation. Block 302 performs a training phase of a machine learning model, such as training a neural network based on a set of training data that includes previously recorded vegetation measurements for a given area separated by time, and weather data for the given area. In some embodiments, the training data is labeled to identify areas of vegetation that have caused damage or have otherwise been identified as high-risk. The output of the training phase is a predictive model that takes measurements of vegetation state and weather information and generates an output indicating a risk level for the relevant area. Although it is expressly contemplated that a neural network can be used to implement the machine learning model, as described previously, it should be understood that any suitable machine learning structure can be used instead.

[0033] The inference phase 304 then takes new measurements of the area of interest and collects weather information to use as input to the trained machine learning model. In some embodiments, the inference phase assigns a risk score to areas within a given image area, identifying particular sub-areas according to their score or range of scores. In some embodiments, the risk score can be binary, simply identifying respective areas as high-risk and low-risk vegetation. In other embodiments, the risk score can take on discrete or continuous values between zero and one. In yet other embodiments, the risk score can be expressed as a number that starts at zero and can reach an arbitrarily high value.

[0034] In some embodiments, the risk score can account for a potential class of damage. For example, one risk index refers to the likelihood of a tree contacting or falling on a power line, road, or residence. This embodiment can estimate the height and crown size of a tree and calculate whether it is likely to bend or lean and come into contact with a power line based on the characteristics of the tree. The likelihood of contacting a power line would then be a risk index, where no likelihood of contact corresponds to a zero risk index, a tree trunk falling on a power line has a 100% risk index, and branches of a tree that contact a power line have a risk index between 0% and 100% based on the amount of overlap between the crown and the power line.

[0035] In another embodiment, weather data characteristics that cause a certain type of tree to cause a certain type of damage can be identified. Weather characteristics can include the amount of precipitation, snow, or strong winds that topple or damage trees. The trained model can identify characteristics such as wind, snowfall, tree height, tree age, and tree species type, in combination with distance from the asset to identify similar areas.

[0036] In another embodiment, the model can learn the types of terrain and soil types that have historically experienced damage, in combination with local weather measurements to understand how weather parameters are affected by vegetation and terrain, and how they cause dangerous conditions.

[0037] In another embodiment, the NN can learn situations where trees are uprooted or destroyed by extreme weather conditions, and can learn, through new images or areas where there is no damage data, those locations where damage is likely to occur.

[0038] Block 306 identifies vegetation risk areas. In some embodiments, this can include comparing the risk score to a threshold, such that areas with a risk score above the threshold are identified as high risk areas. The risk score can be learned locally from the training data, and then summarized by a model across the entire area of interest. Block 308 then performs a corrective action based on the determination of block 306. For example, block 308 can dispatch a team of workers to the identified area(s) to trim vegetation that poses a risk to assets and personnel. In other embodiments, block 308 can use, for example, aerial drones or other automated units to trigger automated verification for physically accessing and observing the high risk areas, and in some embodiments, also for automatically performing vegetation maintenance actions, such as trimming growth or deploying herbicides.

[0039] Reference is now made to Figure 4 which shows additional details of block 302. Block 302 accepts a set of training data as input. For example, the training data can include such as satellite images, LIDAR images, ground conditions such as ground surface normal directions, age and condition of trees, terrain surface properties, and intervening weather information. For example, age of tree, type of tree, location of tree, health of tree, and tree crown features can represent labels of the training data, and can include such features as indicating if a given tree is upright, presence of fallen branches, or presence of fallen trunk, width of trunk, asymmetry of crown relative to its trunk, and direction of trunk. Additional training data can be created around terrain, land use, building density, soil type, and weather patterns. Of particular interest is that “after” images can be taken following a certain severe weather event, such as strong winds, heavy precipitation, or heavy snow. The intervening weather information is then considered to be the cause of the vegetation change between the “before” and “after” images.

[0040] Block 402 identifies changes in plant state based on the before and after images in the training data. Detection of this change is described in more detail below, but it will be appreciated that the image data used for the training phase can include several different parts, such that it is possible to distinguish between terrain, foliage, and man-made structures. Block 402 is therefore able to detect when foliage present in the "before" image has disappeared in the "after" image. Block 404 extracts features from this detected change in vegetation. In some embodiments, each change between the "before" image and the "after" image can be treated as a fallen tree or branch, and can therefore be interpreted as a "positive" sample. Negative samples are interpreted as those regions with no change or a change below a threshold, indicating that the trees and branches of those regions have not fallen.

[0041] Accordingly, in some embodiments, positive samples indicating high-risk vegetation regions include regions where a change was detected. These changes can be particularly focused on changes in vegetation when it is possible to distinguish between vegetation and other features such as man-made structures. Changes in vegetation can then be grouped into contiguous regions, or alternatively, changes in vegetation can be divided into sub-regions, down to the level of individual trees or even branches of trees. Negative samples indicating low-risk vegetation regions can then be determined by grouping regions that have no change in features after the extreme weather event. These regions can also be divided into sub-regions, down to the level of individual trees.

[0042] Block 406 then trains a model using the extracted features and the training data. Any appropriate training process can be used, for example by separating training and test samples in the training data, using the training samples to train the network, and then using the test samples to evaluate the accuracy of the trained network. Differences between the output of the model and the expected output indicated by the training data can be corrected by, for example, backpropagation.

[0043] In some embodiments, the machine learning model can be implemented as an autoencoder neural network operating on input image data, and using k-means clustering or using principal component analysis. In some embodiments, the machine learning model can be implemented as a u-net, which is a convolutional neural network that uses upsampling operators instead of pooling operations to increase the resolution of the output.

[0044] Reference is now made to Figure 5which shows additional details of the inference phase 304. The inference phase takes as input the current image information of the relevant area and the predicted weather information. Block 502 detects vegetation in the recent image data, for example using the same techniques as in block 402, which will be described in more detail below. Block 504 can extract features, for example by grouping together regions showing vegetation, or by subdividing such regions into sub-regions corresponding to, for example, individual trees. As in block 404, various different characteristics can be used to determine the features, including satellite images, LIDAR images, ground conditions, age and condition of trees, terrain surface properties, and intervening weather information. In block 506, the extracted features are applied to a machine learning model, which generates a risk score output.

[0045] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0046] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0047] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage and / or execution.

[0048] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing / processing device, partly on the user's computing / processing device, as a stand-alone software package, partly on the user's computing / processing device and partly on a remote computing / processing device or entirely on the remote computing / processing device or server. In the latter scenario, the remote computing / processing device can be connected to the user's computing / processing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing / processing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0049] Various aspects of the present application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0050] These computer readable program instructions can be provided to a processor of a computer, or other programmable data processing apparatus, to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions for

[0051] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0052] Reference throughout this specification to "an embodiment" or "embodiments" means that a particular feature, structure, characteristic, property, or the like being referred to is included in at least one embodiment of the present invention. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" or any other variations throughout this specification and claims are not necessarily all referring to the same embodiment.

[0053] It should be appreciated that the use of any of the following 7, "and / or", and "at least one of", for example, in the items of the claims that follow, is intended to cover the selection of one of the listed options only, selected options, or a combination of options, e.g., combinations of at least one of the options. As an example, in the phrases "A and / or B" and "at least one of A and B" each of the following is contemplated: A alone and B alone; A alone and B alone; A only; B only; A and B. As an example, in the phrases "A, B, and / or C" and "at least one of A, B, and C" each of the following is contemplated: A alone and B alone and C alone; A alone, B alone, and C alone; A only; B only; C only; A and B; A and C; B and C; A and B and C. As will be apparent to those of ordinary skill in the art and relevant fields, this can be extended to any number of listed items.

[0054] The computer program product of the second aspect can include a computer readable storage medium. The computer readable storage medium can include instructions that, when executed by a processor, cause the processor to perform operations corresponding to the steps of the method of the first aspect. The computer readable storage medium can include a non-transitory computer readable storage medium. The computer readable storage medium can include a computer readable storage medium that is tangible. The computer readable storage medium can include a computer readable storage medium that does not include a signal. The computer readable storage medium can include a computer readable storage medium that is non-transitory. The computer readable storage medium can include a computer readable storage medium that is tangible.

[0055] As used herein, the term "hardware processor subsystem" or "hardware processor" can refer to a processor, memory, software, or combination thereof that coordinates to perform one or more particular tasks. In useful embodiments, a hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, and / or controllers based on individual processor or computing elements (e.g., logic gates, etc.). A hardware processor subsystem can include one or more on-board memories (e.g., cache, dedicated memory arrays, read-only memory, etc.). In some embodiments, a hardware processor subsystem can include one or more memories, which can be on-board memories or off-board memories, or can be exclusively used by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0056] In some embodiments, a hardware processor subsystem can include and execute one or more software elements. The one or more software elements can include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0057] In other some embodiments, a hardware processor subsystem can include specialized, dedicated circuitry to perform one or more electronic processing functions to achieve a specified result. These circuits can include one or more application specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0058] These and other variations of a hardware processor subsystem are contemplated in accordance with embodiments of the present application.

[0059] Reference is now made to Figure 6which shows a vegetation management system 600. The system 600 includes a hardware processor 602 and a memory 604. A network interface 606 also provides communication over any appropriate wired or wireless transmission medium and protocol. For example, the network interface 606 can access up-to-date image information for the relevant area from, for example, a satellite image database. Additionally, the network interface can communicate with other agents to coordinate management of high-risk vegetation.

[0060] A machine learning model 608 is configured to determine a risk score based on input features. These features can be, for example, images including vegetation and additional weather information, as well as various other features such as terrain, soil, and as described above. The risk score output indicates a risk level for a particular area of the input image according to the degree to which the vegetation in that particular area is likely to cause damage during adverse weather conditions. The model is trained by a model trainer 614 using a set of training data 610. The training data includes, for example, images taken before and after adverse weather conditions, as well as information about the intervening weather conditions. The model trainer 614 uses a feature extractor 612 to identify features within the training data, for example by comparing the "before" images and the "after" images and identifying differences or changes between the images as instances of damage.

[0061] A vegetation manager 616 uses the machine learning model 608 to identify risk scores for new images of areas of interest. For example, the vegetation manager 616 can use a feature extractor to identify features in new input images and provide those features as input to the machine learning model 608. For those areas with risk scores above a threshold, the vegetation manager 616 takes some corrective action, as described above.

[0062] Reference is now made to Figure 7 which shows an example processing system 700 that can represent the vegetation management system 600. The processing system 700 includes at least one processor (CPU) 704 that is operatively coupled to other components of the system via a system bus 702. A cache 706, a read-only memory (ROM) 708, a random access memory (RAM) 710, an input / output (I / O) adapter 720, a sound adapter 730, a network adapter 740, a user interface adapter 750, and a display adapter 760 are operatively coupled to the system bus 702.

[0063] A first storage device 722 is operatively coupled to the system bus 702 by the I / O adapter 720. The first storage device 722 can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid-state memory device, or the like. The first storage device 722 can be the same type of storage device or different types of storage devices.

[0064] A speaker 732 is operably coupled to system bus 702 via sound adapter 730. A transceiver 742 is operably coupled to system bus 702 via network adapter 740. A display device 762 is operably coupled to system bus 702 via display adapter 760.

[0065] A first user input device, i.e., input device 1 752, is operably coupled to system bus 702 via user interface adapter 750. Input device 1 752 can be any of the following: a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, a device incorporating functionality of at least two of the above-mentioned devices, etc. Of course, other types of input devices can be used while remaining within the spirit of the present principles. Input device 1 752 can be the same type of user input device or a different type of user input device. Input device 1 752 is used for inputting information to and Figure 7

[0066] Of course, processing system 700 can also include other elements (not shown) as readily appreciated by those of ordinary skill in the art, and certain of them can be omitted. For example, as readily appreciated by one of ordinary skill in the art, various other input devices and / or output devices can be included in processing system 700 depending upon the particular implementation of processing system 700. For example, various types of wireless and / or wired input and / or output devices can be used. Moreover, additional processors, controllers, memories, etc. in various configurations can also be utilized as readily appreciated by one of ordinary skill in the art. These and other variations of processing system 700 can be readily appreciated by one of ordinary skill in the art in light of the teachings of the present principles provided herein.

[0067] Referring now to FIG. 8, Figure 8 This figure illustrates how LIDAR information can be used to detect vegetation and other features in image data. The LIDAR information includes a set of LIDAR data points 802, which represent distances from the LIDAR camera. LIDAR operates by emitting a laser pulse and measuring the amount of time it takes for the reflected return to start point. This information is converted into height / distance information. By considering the height information across different horizontal ranges, different types of features can be resolved.

[0068] ​The digital elevation model 804 identifies the underlying terrain by smoothing the LIDAR points 802 over a relatively large radius (e.g., 100 meters). This builds a terrain profile of the ground plane. The local maximum model 806 tracks the local profile defined by the highest LIDAR points 802 within a local radius (e.g., 10 meters). A corresponding local minimum model (not shown) identifies the profile defined by the lowest LIDAR points 802 within the local radius.

[0069] By combining these models, different types of features can be resolved. For example, by identifying areas where the local minimum model and the local maximum model 806 coincide but differ from the digital elevation model 804, structures 808 can be identified. In these areas, man-made structures create a local deviation from the underlying terrain. In other areas, where there is a large difference between the local minimum model and the local maximum model 806, areas of vegetation 810 can be indicated, where the incomplete coverage allows some LIDAR pulses to reach the ground, while others are blocked by branches and leaves. Thus, in areas where the local minimum model and the local maximum model 806 differ by more than a threshold, the presence of vegetation can be indicated by the present embodiment.

[0070] By a similar combination of the local minimum model and the local maximum model 806, smaller structures 812 such as telephone lines or power lines can be identified, where other structural features such as periodically occurring telephone poles and the long and continuous structure shape can create a distinct signature. In some cases, LIDAR pulses can be completely absorbed, creating a gap in the digital elevation model 804, which can represent a body of water.

[0071] The output of this process can be, for example, in the form of a false-color satellite image, which uses different colors to identify different structural, terrain, and vegetation features. Thus, when an area where the local elevation information has changed significantly shows a substantial difference in the encoded values (e.g., a substantial color change), the change can be quickly identified by subtracting the image before the change from the image after the change.

[0072] The same steps can be followed for all data points returned from the image, and for data points returned from a height range that can represent a certain type of tree or vegetation. From small shrubs and large trees, an adjusted model is created to extract images in the model for assessing risk. In one embodiment, the images can be created for trees that are 20 meters or taller in height, and have a small crown size. This can represent, for example, pine trees, which are easily broken by strong winds. The top can also be moved a large distance by strong winds, where the location of the tree and the type of damage can be learned by the model.

[0073] The initial subtraction of one image from another produces certain artifacts due to the imprecision of the LIDAR data. In such cases, the structure can have an edge shown as new and an edge shown as disappeared because the second image is taken at a slightly different position or angle. In these cases, local averaging removes these artifacts, leaving only the true changes between the images.

[0074] When performing training, box 402 uses different models to first identify the vegetation in the previous and subsequent images, and then subtracts one image from the other to locate the areas where the vegetation has changed. During the inference phase, box 502 similarly uses different models to identify the vegetation within the new image. This information can be combined with various other types of data, particularly weather data, to identify areas of the new image that may have changed.

[0075] In some embodiments, the LIDAR information can be represented as a set of N irregular data points:

[0076]

[0077] where each data point has coordinates . The term "irregular" reflects the following definition:

[0078]

[0079] where is and placeholder, and the index pair marks the nearest adjacent coordinates. Generally speaking, the term " " can be replaced by any kind of norm on a two-dimensional vector space. For simplicity, the dimensions and are interpreted as representing longitude, latitude, and altitude respectively. Each point can have numerical properties associated with it . Such properties can include, for example, LIDAR laser reflection intensity, LIDAR laser reflection angle, etc.

[0080] To create a false color image, this embodiment can, for example, use the red channel to represent the overall global elevation at a scale parameter L . Define a sliding window of ordered pixels where 0 < r < L is the scale of the size of a single pixel of the set image raster, and the image raster has regularly spaced pixel coordinates Instead of irregularly scattered points ( ), so that You can apply any aggregation function to the data points, such as by averaging:

[0081]

[0082] in,

[0083]

[0084] It is used with coordinates The set of points related to the aggregation. Normalization constant. yes The number of midpoints, then can be... Calculate on a set of regularly spaced image grid points :

[0085]

[0086] To form a false-color image. A DEM can represent, for example... Figure 8 Curve 804 in the diagram.

[0087] The above spherical constraints, This is for illustrative purposes only and can be replaced by any suitable geometric condition. The entire aggregation process can be performed using a convolution kernel. Substitute, so that:

[0088]

[0089] in,

[0090]

[0091] For example, the Gaussian function can be used, where For wavelengths greater than 1 / L, a Fourier kernel with an appropriate low-pass filter is another option. Alternatively, any suitable nonlinear function F, such as the median, can also be applied to the z-value.

[0092]

[0093] After the global elevation model (DEM) has been identified, it can be used to remove the influence of overall terrain from other elevation models. This is particularly useful for vegetation management, distinguishing between translucent objects (such as trees) and opaque objects (such as buildings). Introducing a local scale... ,in Larger than resolution scale And smaller than the global scale The local surface model can be defined as

[0094]

[0095] In some embodiments, this can be used as the green channel of a false color image. The LEM can represent, for example Figure 8 the curve 814.

[0096] In a similar manner, the local elevation model can be defined according to

[0097]

[0098] This can be used as, for example, the blue channel of a false color image. Note that for opaque objects, approximately equal to while for semi-transparent objects, is the proxy height above the local ground.

[0099] Having described preferred embodiments for predicting and correcting vegetation status (which are intended to be illustrative and not limiting), it is noted that modifications and alterations can occur to others upon reading the attached claims. It is intended that the application be constructed as including all such modifications and alterations insofar as they come within the scope of the appended claims. Having thus described the application in terms of particular embodiments and illustrative figures, those of ordinary skill in the art will understand that the application is not limited to these examples and details of the procedures described therein. Changes in form and detail can be made without departing from the spirit of the application as expressed by the appended claims.

Claims

1. A method for managing vegetation, comprising: training a machine learning model based on images of a training data region before a weather event, images of the training data region after the weather event, and information about the weather event, the training including identifying man-made structures in the training data region by comparing a local minimum model and a local maximum model to an elevation model, wherein identifying the man-made structures includes determining that the man-made structures exist in portions of the training data region where the local minimum model and the local maximum model coincide but both differ from the elevation model; generating, using the trained machine learning model, a risk score for a second region based on images of the second region and predicted weather information for the second region; determining that the risk score indicates a high risk of vegetation in the second region; and performing a corrective action to reduce the risk of vegetation in the second region.

2. The method of claim 1, wherein training the machine learning model comprises: determining a difference between the images of the training data region before the weather event and the images of the training data region after the weather event.

3. The method of claim 2, wherein training the machine learning model further comprises: identifying changed portions of the training data region as first type samples and un-changed portions of the training data region as second type samples for use as labels during training.

4. The method of claim 1, wherein training the machine learning model further comprises: identifying vegetation in the images of the training data region before the weather event and the images of the training data region after the weather event.

5. The method of claim 4, wherein, identifying vegetation in an image includes comparing a local minimum model and a local maximum model of light detection and ranging (LIDAR) information.

6. The method of claim 5, wherein, identifying vegetation in an image further includes determining that vegetation exists in portions of the image where a difference between the local minimum model and the local maximum model exceeds a threshold.

7. The method of claim 1, further comprising: generating a recommendation for vegetation removal to minimize a risk of vegetation damage to man-made structures.

8. The method of claim 7, wherein the suggestion comprises: repeating the generating after vegetation removal has been performed to verify that the high risk of vegetation has been removed.

9. A method for managing vegetation, comprising: training a machine learning model based on images of a training data region before a weather event, images of the training data region after the weather event, and information about the weather event, the information about the weather event including a difference between vegetation shown in the images of the training data region before the weather event and the images of the training data region after the weather event, the training including identifying man-made structures in the training data region by comparing a local minimum model and a local maximum model to an elevation model, wherein identifying the man-made structures includes determining that the man-made structures exist in portions of the training data region where the local minimum model and the local maximum model coincide but both differ from the elevation model; generating, using the trained machine learning model, a risk score for a second region based on images of the second region and predicted weather information for the second region; determining that the risk score indicates high risk vegetation in the second region; and performing a corrective action to reduce the risk of vegetation in the second region.

10. A non-transitory computer-readable storage medium for managing vegetation, comprising a computer-readable program, wherein when the computer-readable program is executed on a computer, the computer is caused to perform the following steps: training a machine learning model based on images of a training data region before a weather event, images of the training data region after the weather event, and information about the weather event, the training including identifying man-made structures in the training data region by comparing a local minimum model and a local maximum model to an elevation model, wherein identifying the man-made structures includes determining that the man-made structures exist in portions of the training data region where the local minimum model and the local maximum model coincide but both differ from the elevation model; generating, using the trained machine learning model, a risk score for a second region based on images of the second region and predicted weather information for the second region; determining that the risk score indicates high risk vegetation in the second region; and performing a corrective action to reduce the risk of vegetation in the second region.

11. A system for managing vegetation, comprising: a model trainer configured to train a machine learning model based on images of a training data region before a weather event, images of the training data region after the weather event, and information about the weather event, the training including identifying man-made structures in the training data region by comparing a local minimum model and a local maximum model to an elevation model, wherein identifying the man-made structures includes determining that the man-made structures exist in portions of the training data region where the local minimum model and the local maximum model coincide but both differ from the elevation model; and a vegetation manager configured to generate, using the trained machine learning model, a risk score for a second region based on images of the second region and predicted weather information for the second region, determine that the risk score indicates high risk vegetation in the second region, and trigger a corrective action to reduce the risk of vegetation in the second region.

12. The system of claim 11, wherein the model trainer is further configured to determine differences between the images of the training data region before the weather event and the images of the training data region after the weather event.

13. The system of claim 12, wherein the model trainer is further configured to identify changed portions of the training data region as first type samples and unchanged portions of the training data region as second type samples for use as labels during training.

14. The system of claim 11, wherein the model trainer is further configured to identify vegetation in the images of the training data region before the weather event and in the images of the training data region after the weather event.

15. The system of claim 14, wherein the model trainer is further configured to compare a local minimum model of light detection and ranging (LIDAR) information and a local maximum model.

16. The system of claim 15, wherein the model trainer is further configured to determine that the vegetation is present in portions of the images where a difference between the local minimum model and the local maximum model exceeds a threshold value. the vegetation manager is further configured to generate a recommendation for vegetation removal to minimize a risk of vegetation damage to a man-made structure.

17. The system of claim 11, wherein, the vegetation manager is further configured to repeat the generation of the risk score after vegetation removal has been performed to verify that the high risk vegetation has been removed.

18. The system of claim 17, wherein, 19. A system for managing vegetation, comprising: a model trainer configured to train a machine learning model based on images of a training data region before a weather event, images of the training data region after the weather event, and information about the weather event, the information about the weather event including a difference between vegetation shown in the images of the training data region before the weather event and the images of the training data region after the weather event, the training including identifying a man-made structure in the training data region by comparing a local minimum model and a local maximum model to an elevation model, wherein identifying the man-made structure includes determining that the man-made structure is present in portions of the training data region where the local minimum model and the local maximum model coincide but both differ from the elevation model; and a vegetation manager configured to generate a risk score for a second region using the trained machine learning model based on images of the second region and predicted weather information for the second region, determine that the risk score indicates high risk vegetation in the second region, and trigger a corrective action to reduce the risk of vegetation in the second region. ​

Images