Classification method and classification device for tree species along power transmission corridor
Through the feature processing of multi-source remote sensing data and the dual-channel input of Transformer classification model, the problem of low classification accuracy of single-wood-level tree species in the transmission corridor is solved, and detailed identification of tree morphological characteristics and effective distinction between morphological differences is achieved.
Patent Information
- Application Number
- CN202411147002.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-22
AI Technical Summary
The existing tree species recognition technology has low accuracy in transmission corridor scenarios, especially the classification of single-wood-level tree species, and ignores the detailed characteristics in point cloud data, making it difficult to distinguish tree species with huge morphological differences.
Multi-source remote sensing data, including multi-spectral images, thermal infrared images and point cloud data, is used to combine feature processing and JM distance, combined with local maximum filters and watershed segmentation algorithm, to identify the treetops and canopies and cut into central tiles. and external rectangular tiles and use the improved Transformer classification model for dual-channel input parallel processing.
It improves the classification accuracy of single-wood-level tree species, can more accurately identify the morphological characteristics of trees, enhances the ability to distinguish tree species with large morphological differences, and improves the recognition accuracy.
Smart Images

Figure CN119006922A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of tree species identification, and in particular to a classification method, a classification device, a computer-readable storage medium and a computer program product for tree species along a power transmission corridor. Background Art
[0002] The area within two parallel planes formed by the transmission line extending horizontally outward and perpendicular to the ground is called the transmission corridor. In today's society, it is extremely important to ensure the safety of the transmission corridor. However, in the transmission corridor scene, overgrown or weirdly growing trees are often an important cause of power safety accidents. These trees may directly contact the wires and cause wire damage, or cover the wires so that the wires cannot dissipate heat normally and are damaged, or some deciduous tree species frequently produce falling objects that hit the wires and cause physical damage. Therefore, tree species identification plays a key role in the engineering of protecting the safety of transmission corridor facilities. By identifying the tree species in the transmission corridor, it is possible to prevent specific trees that may threaten the power system from threatening the safety of the transmission lines in advance, avoid short circuits and fires, thereby ensuring the continuous supply of electricity, supporting the normal operation of all walks of life, and reducing economic losses caused by power outages. In addition, scientific tree species management helps maintain the balance of the ecological environment, avoid forest fires and vegetation destruction, reduce energy loss, and improve energy utilization efficiency. Therefore, it is of great significance to ensure the safety of the transmission corridor through tree species identification.
[0003] Existing tree species recognition technologies still have significant deficiencies in some aspects. First, the use of LiDAR point cloud data is not comprehensive enough. Most current technologies usually only extract overall morphological feature information from point cloud data. Although this method has wide applicability, it ignores the detailed information contained in the point cloud data that can more accurately express the morphological characteristics of trees. Point cloud data contains rich information such as the height, crown width, and branch and leaf density of trees. If we only focus on the overall morphological characteristics, many important detailed features may be ignored, thus affecting the accuracy of recognition.
[0004] Secondly, in the process of tree species identification and classification, there is often a lack of distinction between tree species with significant morphological differences. For example, banyan trees and pine trees have obvious differences in tree shape. Banyan trees are characterized by multiple branches and no obvious treetops, while pine trees have fewer branches and obvious treetop characteristics. This significant morphological difference has not been fully valued and processed in existing deep learning models, resulting in a large loss of morphological characteristics of the trees themselves during the identification process. Simply mixing tree species with huge morphological differences into a deep learning model not only reduces the accuracy of identification, but may also cause the model to perform poorly when faced with tree species with high diversity. Summary of the invention
[0005] The main purpose of the present application is to provide a classification method, a classification device, a computer-readable storage medium and a computer program product for tree species along a power transmission corridor, so as to at least solve the problem of low accuracy of single-wood-level tree species classification in the prior art.
[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for classifying tree species along a power transmission corridor is provided, comprising: acquiring multi-source remote sensing data, the multi-source remote sensing data comprising multispectral images, thermal infrared images and point cloud data, the point cloud data comprising airborne laser point cloud data and backpack laser point cloud data; performing feature processing on the multi-source remote sensing data to obtain multiple features, the features comprising at least texture features, visible light band features, near infrared band features, thermal infrared band features, normalized vegetation index features and photogrammetric point cloud posture features, the photogrammetric point cloud posture features being morphological features of trees in the point cloud data; sequentially calculating the JM distance between a benchmark feature and a first feature, so as to select a plurality of the features according to the JM distance to generate a target feature combination, the benchmark feature being any one of the features, the first feature being any one of the features other than the benchmark feature, and the target feature combination being a combination of the benchmark feature and a plurality of the first features; using a local maximum filter to extract a target feature from a canopy height model. The treetops are identified to obtain a treetop image, and the outline of the treetop is delineated from the canopy height model using a watershed segmentation algorithm to obtain a canopy image, wherein the canopy height model is a digital model that describes the distribution of vegetation canopy height; the treetop image and the canopy image are cropped to obtain a plurality of central blocks and a plurality of circumscribed rectangular blocks, respectively, wherein the central block is a block cropped from the treetop image with the treetop as the center, and the circumscribed rectangular block is a block cropped from the canopy image with the circumscribed rectangle of the canopy as the edge; the first input data and the second input data are simultaneously input into an improved Transformer classification model to obtain a classification result of single-tree-level tree species along the transmission corridor, wherein the first input data is the target feature combination and the central block, and the second input data is the target feature combination and the circumscribed rectangular block, and the improved Transformer classification model is a Transformer model that obtains the classification result of the single-tree-level tree species by dual-channel input and parallel input processing.
[0007] Optionally, before performing feature processing on the multi-source remote sensing data to obtain multiple features, the method also includes: performing noise filtering on the point cloud data to obtain valid point cloud data; using a registration algorithm to register the airborne laser point cloud data and the backpack laser point cloud data in the valid point cloud data to obtain fused point cloud data; eliminating discrete abnormal points in the fused point cloud data to obtain valid fused point cloud data; distinguishing ground points and non-ground points in the valid fused point cloud data according to a set elevation threshold to obtain a ground point set and a non-ground point set; using a natural neighbor interpolation method to rasterize the ground point set into a digital elevation model, and at the same time rasterizing the non-ground point set into a digital surface model; calculating the difference between the digital elevation model and the digital surface model to obtain the canopy height model.
[0008] Optionally, feature processing is performed on the multi-source remote sensing data to obtain multiple features, including: preprocessing operations and alignment and stitching operations are performed on the multispectral image and the thermal infrared image to obtain image features, the image features include the texture features, the visible light band features, the near-infrared band features, the thermal infrared band features and the normalized vegetation index features, the alignment and stitching operations at least include orthogonal correction, radiation calibration, stitching and resampling, and the preprocessing operation is an operation of preprocessing the image; feature extraction processing is performed on the point cloud data using the canopy height model to obtain the photogrammetric point cloud posture features.
[0009] Optionally, the multispectral image and the thermal infrared image are subjected to a preprocessing operation and a registration and stitching operation to obtain image features, including: performing the preprocessing operation on the multispectral image and the thermal infrared image to enhance the quality of the images and obtain an effective multispectral image and an effective thermal infrared image, respectively; performing the registration and stitching operation on the effective multispectral image and the effective thermal infrared image and extracting a grayscale co-occurrence matrix and spectral band information at the same time; extracting the texture features from the grayscale co-occurrence matrix; extracting a visible light band, a near infrared band and a thermal infrared band from the spectral band information to obtain the visible light band features, the near infrared band features and the thermal infrared band features, respectively; and calculating the normalized vegetation index features according to a first formula, the first formula being NDVI=((NIR-R)) / ((NIR+R)), where NIR represents the near infrared band features, R represents the reflectance value of red light in the visible light band, and NDVI represents the normalized vegetation index features. Optionally, the canopy height model is used to perform feature extraction processing on the point cloud data to obtain the photogrammetric point cloud posture features, including: determining an extraction standard based on the pixel size of the multispectral image, the extraction standard being to set the maximum elevation of all pixel points in each pixel, the mean of all elevation points in each pixel, and the entropy and key percentiles of the elevation distribution in each pixel; extracting point cloud posture information that meets the extraction standard from the canopy height model to obtain the photogrammetric point cloud posture features.
[0010] Optionally, the JM distance between the first feature and the second feature is calculated in sequence, and the feature combination with the saturated JM distance is selected to be determined as the target feature combination, including: a calculation step of calculating the JM distance between the reference feature and the first feature in sequence according to a second formula, wherein the second formula is Wherein, JM represents the JM distance, d M represents the Mahalanobis distance between the reference feature and the first feature, μ1 and μ2 are the mean vectors of the benchmark feature and the first feature respectively, ∑1 and ∑2 are the covariance matrices of the benchmark feature and the first feature respectively, and T represents the transpose of the difference between the mean vectors of the benchmark feature and the first feature; a combining step, adding the current feature to the benchmark feature combination, the current feature is the first feature with the largest JM distance calculated with the benchmark feature, and the benchmark feature combination is a combination formed by the features with the largest JM distance; a screening step, screening the current feature from the first feature set, the first feature set is a set formed by all the first features; the calculating step, the combining step and the screening step are repeated at least once in sequence until the number of the features in the benchmark feature combination reaches a set number or the JM distance reaches saturation, and the benchmark feature combination is determined as the target feature combination.
[0011] Optionally, the treetop image and the crown image are cropped to obtain a plurality of center blocks and a plurality of circumscribed rectangular blocks, respectively, including: cropping the treetop image with the treetop as the center according to a set block size to obtain the center block; cropping the crown image with the circumscribed rectangle of the crown as the edge to obtain a preliminary circumscribed rectangular block; scaling the preliminary circumscribed rectangular block to be consistent with the set block size to obtain the circumscribed rectangular block.
[0012] According to another aspect of the present application, a device for classifying tree species along a power transmission corridor is provided, the device comprising: an acquisition unit, used to acquire multi-source remote sensing data, the multi-source remote sensing data comprising multispectral images, thermal infrared images and point cloud data, the point cloud data comprising airborne laser point cloud data and backpack laser point cloud data; a feature processing unit, used to perform feature processing on the multi-source remote sensing data to obtain multiple features, the features comprising at least texture features, visible light band features, near infrared band features, thermal infrared band features, normalized vegetation index features and photogrammetric point cloud posture features, the photogrammetric point cloud posture features being morphological features of trees in the point cloud data; a feature combination unit, used to sequentially calculate the JM distance between a benchmark feature and a first feature, so as to select a plurality of the features according to the JM distance to generate a target feature combination, the benchmark feature being any one of the features, the first feature being any one of the features other than the benchmark feature, and the target feature combination being a combination of the benchmark feature and a plurality of the first features; and an identification unit, used to use a local maximum filter to select a target feature combination from a plurality of the first features. The treetops are identified in the canopy height model to obtain a treetop image, and the outline of the treetop is delineated from the canopy height model using a watershed segmentation algorithm to obtain a canopy image. The canopy height model is a digital model that describes the distribution of vegetation canopy height; a cropping unit is used to crop the treetop image and the canopy image to obtain a plurality of central blocks and a plurality of circumscribed rectangular blocks, respectively. The central block is a block cropped from the treetop image with the treetop as the center, and the circumscribed rectangular block is a block cropped from the canopy image with the circumscribed rectangle of the canopy as the edge; a classification unit is used to input the first input data and the second input data simultaneously into the improved Transformer classification model to obtain the classification result of single-tree-level tree species along the transmission corridor, wherein the first input data is the target feature combination and the central block, and the second input data is the target feature combination and the circumscribed rectangular block. The improved Transformer classification model is a Transformer model that obtains the classification result of the single-tree-level tree species by dual-channel input parallel processing.
[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.
[0014] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, any one of the methods described above is implemented.
[0015] Applying the technical solution of the present application, in the classification method of tree species along the transmission corridor, multispectral images, thermal infrared images and point cloud data along the transmission corridor are collected; then the multispectral images, thermal infrared images and point cloud data are feature processed to obtain multiple features; the best feature combination, i.e., the target feature combination, is obtained by calculating the JM distance saturation between features; the treetops are identified from the canopy height model using a local maximum filter, and then the outline of the crown is delineated from the canopy height model using a watershed segmentation algorithm; with the single treetop as the center, multiple central blocks are cropped from the feature data, and then multiple circumscribed rectangular blocks are cropped with the circumscribed rectangle of the single tree crown outline, and scaled to the same size as the central block; finally, the improved Transformer classification model is used to perform feature learning on the central block and the circumscribed rectangular block respectively, and the tree species category corresponding to each block is identified, and finally the classification results of single-tree-level tree species along the transmission corridor are obtained. The present application solves the problem of low accuracy of single-tree-level tree species classification in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for classifying tree species along a power transmission corridor provided in an embodiment of the present application is shown;
[0017] Figure 2 A schematic diagram of a process of classifying tree species along a power transmission corridor provided according to an embodiment of the present application is shown;
[0018] Figure 3 An overall flow chart of tree species classification along a power transmission corridor provided according to an embodiment of the present application is shown;
[0019] Figure 4 A flowchart of tree species classification using a dual-channel Transformer provided in an embodiment of the present application is shown;
[0020] Figure 5 An example diagram of a feature combination generation process provided according to an embodiment of the present application is shown;
[0021] Figure 6 A structural block diagram of a device for classifying tree species along a power transmission corridor provided according to an embodiment of the present application is shown.
[0022] The above drawings include the following reference numerals:
[0023] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION
[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0027] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:
[0028] Canopy height model: The canopy height model is a mathematical model used to describe the vertical distribution of the forest canopy. It is usually based on the biological characteristics and crown structure of the trees, combined with factors such as tree height, crown diameter, leaf distribution, etc., to predict the coverage area and density of the crown at different heights.
[0029] Local maximum filter: A digital signal processing technique used to detect local maxima in a signal and highlight them. The local maximum filter works by moving a window in the signal, finding the maximum value in each window, and using it as the output value of the window. This can make the signal smoother and highlight the extreme points in the signal, making them easier to detect and analyze.
[0030] Watershed segmentation algorithm: The watershed segmentation algorithm is an image segmentation algorithm used to segment an image into regions with different semantics. This algorithm is based on the watershed theory and regards the image as a topographic map where the grayscale value represents the height. The goal is to find ridges and valleys on the map and determine which regions belong to the same object. The basic idea of the algorithm is to regard the image as a three-dimensional topographic map where the grayscale value represents the height. First, the grayscale value of the image is converted into a gradient map, and then the gradient map is marked, and the local minimum point is marked as the seed point. Next, it starts from the seed point and expands outward until different seed points meet to form a segmentation line, namely the watershed. Finally, by processing the watershed, different regions are segmented.
[0031] As introduced in the background technology, the prior art only focuses on the overall morphological characteristics, and many important detail features may be ignored, thereby affecting the recognition accuracy. In addition, in the process of tree species identification and classification, there is often a lack of distinction between tree species with significant morphological differences. In order to solve the problem of low accuracy of single-tree-level tree species classification in the prior art, the embodiments of the present application provide a classification method, classification device, computer-readable storage medium and computer program product for tree species along a transmission corridor.
[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0033] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal for a method for classifying tree species along a power transmission corridor according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0034] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0035] In this embodiment, a method for classifying tree species along a transmission corridor is provided, which runs on a mobile terminal, a computer terminal or a similar computing device. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0036] Figure 2 FIG. 1 is a flow chart of a method for classifying tree species along a power transmission corridor according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:
[0037] Step S201, acquiring multi-source remote sensing data, the multi-source remote sensing data including multi-spectral images, thermal infrared images and point cloud data, the point cloud data including airborne laser point cloud data and backpack laser point cloud data.
[0038] Specifically, multispectral images, thermal infrared images, airborne laser point cloud data, and backpack laser point cloud data of drones along the transmission corridor are collected. In addition, related prior data are included, such as the list of dominant tree species in the collection area. Airborne sensors are ground detection sensors installed on drones. Common multispectral sensors, thermal infrared sensors, and lidar point cloud detectors are all conventional airborne sensors. Airborne sensors can be used to collect multi-source remote sensing data, such as multispectral images, thermal infrared images, and airborne lidar point clouds. Backpack lidar point cloud detectors can collect lidar point cloud data based on the ground. Therefore, multispectral remote sensing images are obtained by using multispectral sensors to observe the ground; thermal infrared images are obtained by using thermal infrared sensors to observe the ground; airborne lidar point clouds are obtained by using airborne lidar detectors to observe the ground; and ground lidar point clouds are obtained by using backpack lidar detectors to observe and collect.
[0039] Step S202, feature processing is performed on the multi-source remote sensing data to obtain multiple features, wherein the features at least include texture features, visible light band features, near infrared band features, thermal infrared band features, normalized vegetation index features and photogrammetric point cloud posture features, wherein the photogrammetric point cloud posture features are morphological features of trees in the point cloud data.
[0040] Specifically, multispectral images and thermal infrared images generate images by capturing image data in multiple spectral bands. These bands include visible light bands (red, green, and blue) and invisible light bands (near infrared, thermal infrared, etc.). Since multispectral images and thermal infrared images capture data in multiple bands, these data can complement each other and provide more comprehensive information. By analyzing and fusing data from different bands, different types and states of land objects can be identified and distinguished. At the same time, for the phenomenon of different objects in the same spectrum that may appear in some common processing bands, more band selection also means that land objects can be identified through more dimensions. The point cloud data contains detailed information that can more accurately express the morphological characteristics of trees, such as the height, crown width, branch and leaf density of trees. Therefore, feature processing is performed on the above multispectral images, thermal infrared images, and point cloud data to extract tree-related features in multi-source remote sensing data.
[0041] Step S203, calculate the JM distance between the reference feature and the first feature in turn, and select multiple features according to the JM distance to generate a target feature combination, the reference feature is any one of the features, the first feature is any one of the features except the reference feature, and the target feature combination is a combination of the reference feature and multiple first features.
[0042] Specifically, Figure 3As shown, after the feature extraction is completed, feature combination is performed and the JM distance of each feature combination is calculated, and a series of feature combinations with saturated JM distances are selected as the final feature combination group, that is, the above-mentioned target feature combination. The JM distance is the "distance" calculated between different objects on a single band image. The larger the distance, the better the separability of the two categories of objects in the band. In a feature combination that includes multiple feature categories, the larger the JM distance corresponding to the combination, the lower the correlation between the features in the combination, and the better the separability of the target object among these features. When the JM distance tends to be saturated, the separability will no longer change even if a new independent feature is added. At this time, the classifiability of the feature combination reaches the best. The above-mentioned objects refer to objects on the ground, and the present embodiment also refers to trees.
[0043] Step S204, using a local maximum filter to identify treetops from the canopy height model to obtain a treetop image, and using a watershed segmentation algorithm to delineate the outline of the tree crown from the above canopy height model to obtain a canopy image. The above canopy height model is a digital model that describes the distribution of vegetation canopy height.
[0044] Specifically, since the treetop is often the highest point on a tree and the elevation of points around the treetop is significantly lower than the treetop, a local maximum filter is used to filter the point cloud, and the treetop in the canopy height model is identified by identifying local peak points, so that the treetop image can be obtained. The watershed algorithm is an image segmentation technology that is widely used in image processing and computer vision. Its name comes from the concept of watershed in geography, which refers to treating an image as terrain and segmenting the image into different areas by simulating the process of water flowing in the terrain. In the watershed algorithm, the grayscale value of the image is regarded as the height of the terrain. High grayscale values represent peaks and low grayscale values represent valleys. First, the height threshold is set according to the characteristics of the dominant tree species in the transmission scenario, and the valley areas in the image are marked. These areas are usually the segmentation objects we are interested in, that is, the crowns that are often at a lower level and in contact with other trees. The process of simulating the water flow starts from the marked valley, and the water flow expands from the valley to the peak. When the water flows from different valleys to meet, a watershed is formed. The watershed is where the water flows meet, and these locations are the edges in the image. Through the watershed algorithm, the image is segmented into multiple independent regions, each of which represents an object or background, thereby segmenting the edge of the tree crown and obtaining the tree crown image.
[0045] Step S205, cropping the treetop image and the tree crown image to obtain a plurality of center blocks and a plurality of circumscribed rectangular blocks, respectively. The center block is a block cropped from the treetop image with the treetop as the center, and the circumscribed rectangular block is a block cropped from the tree crown image with the circumscribed rectangle of the tree crown as the edge.
[0046] Specifically, the difference in tree species often leads to huge differences in tree morphology: for example, Pinaceae trees often do not have large branches and have obvious tops; Moraceae trees often have many branches of similar thickness and luxuriance and do not have obvious treetops. In order to meet the classification requirements of morphological differences between different tree species, the acquired image data is cropped from two perspectives. The first is a tile cropped with the treetop as the center, with a tile size of 64*64; the second is a tile cropped with the circumscribed rectangle of the crown as the edge, which is scaled to the same size as the tile in the center of the treetop after cropping.
[0047] Step S206, input the first input data and the second input data into the improved Transformer classification model at the same time to obtain the single-tree-level tree species classification results along the transmission corridor, the above-mentioned first input data is the above-mentioned target feature combination and the above-mentioned central block, the above-mentioned second input data is the above-mentioned target feature combination and the above-mentioned circumscribed rectangular block, and the above-mentioned improved Transformer classification model is a Transformer model that obtains the above-mentioned single-tree-level tree species classification results through dual-channel input and parallel input processing.
[0048] Specifically, Figure 4 As shown in the figure, the single input of the Transformer model is improved into two dual inputs that do not affect each other. After the dual inputs enter the model, they are completely parallel and unrelated in other modules except the attention module, but in the multi-head self-attention module, the two share the same attention. This allows the blocks generated from different angles for the same actual object to refer to each other, strengthen their own classification according to each other's global features, and minimize the correlation between the classifications of the two, so that the last two can generate classification results from different angles. For different dominant tree species in the transmission corridor scene, one of the two channels will always adapt to its tree shape characteristics. According to the list of dominant tree species, the corresponding classification results are extracted from different channels and combined into the final classification results. The problem of poor recognition accuracy caused by the large difference in tree species posture is greatly reduced. Therefore, a dual-channel Transformer classifier is used for tree species recognition. The target feature combination and the two cropped blocks are combined and input into the dual-channel Transformer classifier for tree species classification.
[0049] In this embodiment, multispectral images, thermal infrared images and point cloud data along the transmission corridor are collected; then the multispectral images, thermal infrared images and point cloud data are feature processed to obtain multiple features; the best feature combination, i.e., the target feature combination, is obtained by calculating the JM distance saturation between features; the treetops are identified from the canopy height model using a local maximum filter, and then the outline of the crown is delineated from the canopy height model using a watershed segmentation algorithm; multiple central blocks are cropped from the feature data with the single treetop as the center, and multiple circumscribed rectangular blocks are cropped with the circumscribed rectangle of the single tree crown outline, and scaled to the same size as the central block; finally, the improved Transformer classification model is used to perform feature learning on the central block and the circumscribed rectangular block respectively, and the tree species category corresponding to each block is identified, and finally the single tree-level tree species classification results along the transmission corridor are obtained. This application solves the problem of low accuracy in the classification of single tree-level tree species in the prior art.
[0050] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for classifying tree species along the power transmission corridor of the present application will be described in detail below in combination with specific embodiments.
[0051] In order to accurately reflect the canopy height of the tree and help analyze and study the height and structure of the tree, in an optional implementation, before the above step S202, the method further includes:
[0052] Step S301, filtering the above point cloud data to remove noise, and obtaining valid point cloud data;
[0053] Step S302, registering the airborne laser point cloud data and the backpack laser point cloud data in the valid point cloud data using a registration algorithm to obtain fused point cloud data;
[0054] Step S303, removing the discrete abnormal points in the above fused point cloud data to obtain effective fused point cloud data;
[0055] Step S304, distinguishing ground points and non-ground points in the above-mentioned effective fused point cloud data according to a set elevation threshold, to obtain a ground point set and a non-ground point set;
[0056] Step S305, using the natural neighbor interpolation method to rasterize the ground point set into a digital elevation model, and at the same time rasterize the non-ground point set into a digital surface model;
[0057] Step S306, calculating the difference between the digital elevation model and the digital surface model to obtain the canopy height model.
[0058] In the above embodiment, if Figure 5The upper part (i.e., the point cloud data part) is processed. The airborne laser radar point cloud data and the backpack laser radar point cloud data are noise filtered and denoised, and the airborne laser point cloud and backpack laser point cloud data are registered to generate fused point cloud data, and the fused point cloud data is used to generate the canopy height model (CHM). Specifically, the following steps are included: first, the discrete abnormal points in the fused point cloud data are removed to obtain the effective fused point cloud data; the ground points and non-ground points in the fused point cloud data are separated according to the height characteristics of the dominant tree species in the transmission corridor, that is, a suitable elevation threshold is set to separate the ground point set and the non-ground point set of the effective fused point cloud data, and the points whose elevation is less than the above-set elevation threshold are the above-mentioned ground points, and the points whose elevation is greater than or equal to the above-set elevation threshold are the above-mentioned non-ground points; the ground points and non-ground points are rasterized into a digital elevation model (DEM) and a digital surface model (DSM) respectively by the natural neighbor interpolation method; the difference between the digital elevation model (DEM) and the digital surface model (DSM) is calculated, and the difference is the canopy height model (CHM) to provide tree canopy height information. CHM can accurately reflect the tree canopy height, which is helpful for analyzing and studying the tree height and tree structure. The above-mentioned registration method is mainly a feature-based registration method, which includes the following steps: feature extraction and description: extract key feature points, such as corner points and edge points, from the effective point cloud data, and calculate descriptors for each feature point, such as FPFH (Fast Point Feature Histogram); feature matching: use descriptors to find corresponding feature point pairs in multi-platform point cloud data; coarse registration: according to the matched feature point pairs, the initial transformation matrix is estimated based on RANSAC (Random Sampling Consensus Algorithm); fine registration: use ICP (Iterative Closest Point) algorithm to make fine adjustments based on the preliminary registration.
[0059] In order to maximize the extraction of tree species features and enhance the reliability of identification, in an optional implementation, the above step S202 includes:
[0060] Step S2021, performing a preprocessing operation and a registration and stitching operation on the multispectral image and the thermal infrared image to obtain image features, wherein the image features include the texture features, the visible light band features, the near infrared band features, the thermal infrared band features and the normalized vegetation index features, the registration and stitching operation at least includes orthogonal correction, radiation calibration, stitching and resampling, and the preprocessing operation is an operation of preprocessing the image;
[0061] Step S2022: performing feature extraction processing on the point cloud data using the canopy height model to obtain the photogrammetric point cloud posture features.
[0062] In the above embodiments, the multispectral image, the thermal infrared image and the point cloud data are processed in different ways. Figure 5 As shown, the processing methods for the above multispectral images and the above thermal infrared images include orthogonal correction, radiation calibration, splicing and resampling, extracting the grayscale co-occurrence matrix of multispectral data to generate texture information, extracting visible light R, G, B bands, near infrared bands, thermal infrared bands and specific vegetation indices to calculate and extract spectral features and vegetation indices. The processing method for point cloud data is to perform rasterization processing to generate a canopy height model (CHM), and then use the canopy height model to perform feature extraction processing on the above point cloud data to obtain the above photogrammetric point cloud posture feature (PPC). Combining multispectral data, point cloud data and tree 3D posture features (that is, photogrammetric point cloud posture features PPC) for tree species recognition can significantly improve the accuracy and robustness of recognition. Multispectral data provides rich spectral features, reflecting the spectral differences of different tree species in different bands, while point cloud data and 3D posture data provide detailed geometric and structural features, such as tree height, canopy shape, trunk diameter, and arrangement of branches and leaves. These visual and posture features complement each other and can more comprehensively describe the uniqueness of tree species and enhance the reliability of identification.
[0063] In order to extract the spectral differences of different tree species in different bands and comprehensively describe the uniqueness of the tree species, in an optional implementation manner, the above step S2021 includes:
[0064] Step S20211, performing the above-mentioned preprocessing operation on the multispectral image and the thermal infrared image to enhance the quality of the images, and obtaining a valid multispectral image and a valid thermal infrared image respectively;
[0065] Step S20212, performing the above-mentioned registration and splicing operation on the above-mentioned effective multispectral image and the above-mentioned effective thermal infrared image, and extracting gray-level co-occurrence matrix and spectral band information at the same time;
[0066] Step S20213, extracting the texture features from the gray level co-occurrence matrix;
[0067] Step S20214, extracting a visible light band, a near infrared band and a thermal infrared band from the spectral band information, and obtaining the visible light band feature, the near infrared band feature and the thermal infrared band feature respectively;
[0068] Step S20215, calculating the above-mentioned normalized vegetation index feature according to the first formula, the above-mentioned first formula is NIR represents the above-mentioned near-infrared band characteristics, R represents the reflectance value of red light in the above-mentioned visible light band, and NDVI represents the above-mentioned normalized vegetation index characteristics.
[0069] In the above embodiment, the multispectral image and the thermal infrared image are first processed to enhance the image quality, and effective multispectral images and effective thermal infrared images are obtained respectively. The effective multispectral image and the effective thermal infrared image are subjected to orthogonal correction, radiation calibration, splicing and resampling, and the visible light R, G, B bands, near infrared bands, and thermal infrared bands are extracted, that is, the visible light band features, the near infrared band features, and the thermal infrared band features. The corresponding normalized vegetation index NDVI is calculated according to the extracted band data, and the calculation method is the first formula above. Then the grayscale co-occurrence matrix of the object is extracted and the texture features are extracted from it. The texture features are mainly reflected in eight parameters: angular moment (SM), contrast (con), variance (var), homogeneity (homm), correlation (cor), entropy (ent), dissimilarity and texture average features.
[0070] In order to improve the degree of differentiation between tree species with significant morphological differences, in an optional implementation, the above step S2022 includes:
[0071] Step S20221, determining an extraction standard based on the pixel size of the multispectral image, wherein the extraction standard is to set the maximum elevation of all pixels in each pixel, the mean of all elevation points in each pixel, and the entropy and key percentile of the elevation distribution in each pixel;
[0072] Step S20222: extracting point cloud posture information satisfying the above extraction criteria from the above canopy height model to obtain the above photogrammetric point cloud posture features.
[0073] In the above embodiment, in order to strengthen the feature extraction of point cloud data, the photogrammetric point cloud posture feature (PPC) is added to the classification of tree species identification as feature data describing the posture feature of a single tree. The addition of the photogrammetric point cloud posture feature greatly improves the depiction of the morphological features of the tree species itself by the point cloud feature data, increases the extracted posture features, and effectively improves the accuracy of tree species identification. The photogrammetric point cloud posture feature is extracted from the canopy height model generated by the point cloud by setting a series of thresholds, such as the statistical characteristics of the elevation of the point cloud in the canopy height model, the specific percentile morphological features, etc. These features can well represent the morphological characteristics of the trees themselves. For example, for multi-branch tree species, the point cloud features at 90% of the elevation positions are generally dispersed, and gradually close as the percentage decreases; while for single treetop tree species, the opposite is true. In this way, the morphological differences between the two trees are well reflected, and the deeper morphological features of the tree species themselves are also extracted. Based on the pixel size of the multispectral image, the corresponding point cloud posture feature data are extracted from the fused point cloud according to the established standards. The establishment of standards should be based on the ability to express the morphological characteristics of point clouds. The standards should be: the maximum elevation of all points in each pixel, the mean of all elevation points in each pixel, the entropy of the elevation distribution in each pixel, and the critical percentile of the elevation distribution in each pixel.
[0074] In order to avoid poor recognition accuracy caused by large correlation between features in a multi-feature combination, in an optional implementation, the above step S203 includes:
[0075] Step S2031, a calculation step, sequentially calculating the JM distance between the reference feature and the first feature according to a second formula, the second formula being: Where JM represents the above JM distance, d M represents the Mahalanobis distance between the reference feature and the first feature, μ1 and μ2 are the mean vectors of the reference feature and the first feature, ∑1 and ∑2 are the covariance matrices of the reference feature and the first feature, and T represents the transpose of the difference between the mean vectors of the reference feature and the first feature;
[0076] Step S2032, a combining step, adding the current feature to the reference feature combination, the current feature being the first feature having the largest JM distance calculated with the reference feature, and the reference feature combination being the combination formed by the features having the largest JM distance;
[0077] Step S2033, a screening step, screening the current feature from the first feature set, the first feature set being a set formed by all the first features;
[0078] Step S2034, repeatedly executing the above calculation step, the above combination step and the above screening step at least once in sequence, until the number of the above features in the above reference feature combination reaches the set number or the above JM distance reaches saturation, and the above reference feature combination is determined as the above target feature combination.
[0079] In the above embodiment, in order to solve the problem of poor recognition accuracy caused by large correlation between features in multiple feature combinations in tree species recognition, the JM distance is used to determine the feature combination, such as Figure 5 In the lower right part, one feature is selected from all the features as the reference feature, and it is combined with other features in turn and the change of the JM distance of the combination is calculated. The combination with the largest JM distance is retained, and the remaining features are combined with the combination in turn and the change of the JM distance is calculated. When the number of features in the group reaches the requirement or the JM distance is saturated and no longer increases, a set of suitable feature combinations is obtained. This operation is performed on each extracted feature in turn, and the combinations with too similar components are eliminated from the final result until the number of combinations reaches the requirement, so that a series of feature combinations that meet the classification requirements are obtained, that is, the above-mentioned target feature combination. The JM distance is the "distance" calculated between different objects on a single band image. The larger the distance, the better the separability of the two categories of objects in the band. In a feature combination containing multiple feature categories, the larger the JM distance corresponding to the combination, which means that the correlation between the features in the combination is lower, and the target object can be better classified among these features. When the JM distance tends to saturate, the separability will not change even if new features are added. At this time, the classification performance of the feature combination is optimal.
[0080] In order to avoid the reduction of recognition accuracy due to the large difference in tree species morphology, in an optional implementation, the above step S205 includes:
[0081] Step S205 1, cutting the treetop image according to a set block size with the treetop as the center to obtain the center block;
[0082] Step S2052, cutting the tree crown image with the circumscribed rectangle of the tree crown as the edge to obtain a preliminary circumscribed rectangle image block;
[0083] Step S2053, scaling the preliminary circumscribed rectangular block to be consistent with the set block size, to obtain the circumscribed rectangular block.
[0084] In the above embodiment, if Figure 5The upper right part of the image (i.e., the treetop center block and the crown circumscribed rectangular block part) performs two types of cropping for each identified treetop and crown, namely, the treetop-centered cropping block and the crown circumscribed rectangular cropping block. These two types of blocks take into account the differences in various morphological characteristics of trees. The classification effect of multi-branched tree species is better in the crown circumscribed rectangular block group, and the classification effect of single-treetop tree species is better in the treetop center block group. Different tree species often lead to huge differences in tree morphology: for example, pine trees often have no large branches and have obvious highest treetops; Moraceae trees often have many branches with similar thickness and luxuriance and no obvious treetops. In order to solve the problem of reduced recognition accuracy caused by excessive morphological differences between tree species, two tree shape block cropping methods are used. The treetop-centered cropping block and the crown circumscribed rectangular cropping block correspond to the two common tree shapes in nature: multi-branched shape and single treetop shape. The problem of reduced recognition accuracy caused by large differences in tree species morphology was avoided. Therefore, the acquired image data was cropped from two perspectives. The first was to crop the block with the treetop as the center, with a block size of 64*64; the second was to crop the block with the circumscribed rectangle of the tree crown as the edge, and after cropping, it was scaled to the same size as the center block of the treetop.
[0085] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0086] The embodiment of the present application also provides a device for classifying tree species along a transmission corridor. It should be noted that the device for classifying tree species along a transmission corridor in the embodiment of the present application can be used to execute the method for classifying tree species along a transmission corridor provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been explained will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.
[0087] The following introduces the tree species classification device along the power transmission corridor provided in the embodiment of the present application.
[0088] Figure 6 is a structural block diagram of a device for classifying tree species along a power transmission corridor according to an embodiment of the present application. Figure 6 As shown, the device comprises:
[0089] The acquisition unit 10 is used to acquire multi-source remote sensing data, the multi-source remote sensing data includes multi-spectral images, thermal infrared images and point cloud data, and the point cloud data includes airborne laser point cloud data and backpack laser point cloud data.
[0090] Specifically, multispectral images, thermal infrared images, airborne laser point cloud data, and backpack laser point cloud data of drones along the transmission corridor are collected. In addition, related prior data are included, such as the list of dominant tree species in the collection area. Airborne sensors are ground detection sensors installed on drones. Common multispectral sensors, thermal infrared sensors, and lidar point cloud detectors are all conventional airborne sensors. Airborne sensors can be used to collect multi-source remote sensing data, such as multispectral images, thermal infrared images, and airborne lidar point clouds. Backpack lidar point cloud detectors can collect lidar point cloud data based on the ground. Therefore, multispectral remote sensing images are obtained by using multispectral sensors to observe the ground; thermal infrared images are obtained by using thermal infrared sensors to observe the ground; airborne lidar point clouds are obtained by using airborne lidar detectors to observe the ground; and ground lidar point clouds are obtained by using backpack lidar detectors to observe and collect.
[0091] The feature processing unit 20 is used to perform feature processing on the above-mentioned multi-source remote sensing data to obtain multiple features, and the above-mentioned features at least include texture features, visible light band features, near infrared band features, thermal infrared band features, normalized vegetation index features and photogrammetric point cloud posture features. The above-mentioned photogrammetric point cloud posture features are morphological features of trees in the above-mentioned point cloud data.
[0092] Specifically, multispectral images and thermal infrared images generate images by capturing image data in multiple spectral bands. These bands include visible light bands (red, green, and blue) and invisible light bands (near infrared, thermal infrared, etc.). Since multispectral images and thermal infrared images capture data in multiple bands, these data can complement each other and provide more comprehensive information. By analyzing and fusing data from different bands, different types and states of land objects can be identified and distinguished. At the same time, for the phenomenon of different objects in the same spectrum that may appear in some common processing bands, more band selection also means that land objects can be identified through more dimensions. The point cloud data contains detailed information that can more accurately express the morphological characteristics of trees, such as the height, crown width, branch and leaf density of trees. Therefore, feature processing is performed on the above multispectral images, thermal infrared images, and point cloud data to extract tree-related features in multi-source remote sensing data.
[0093] The feature combination unit 30 is used to calculate the JM distance between the reference feature and the first feature in sequence, so as to select multiple features according to the JM distance to generate a target feature combination, wherein the reference feature is any one of the features, the first feature is any one of the features except the reference feature, and the target feature combination is a combination of the reference feature and multiple first features.
[0094] Specifically, Figure 3 As shown in the figure, after the feature extraction is completed, the feature combination is performed and the JM distance of each feature combination is calculated. A series of feature combinations with saturated JM distance are selected as the final feature combination group, that is, the above-mentioned target feature combination. The JM distance is the "distance" calculated between different objects on a single band image. The larger the distance, the better the separability of the two categories of objects in the band. In a feature combination that contains multiple feature categories, the larger the JM distance corresponding to the combination, the lower the correlation between the features in the combination, and the better the separability of the target object among these features. When the JM distance tends to be saturated, the separability will not change even if a new independent feature is added. At this time, the classification performance of the feature combination reaches the best.
[0095] The identification unit 40 is used to use a local maximum filter to identify treetops from the canopy height model to obtain a treetop image, and use a watershed segmentation algorithm to delineate the outline of the tree crown from the above-mentioned canopy height model to obtain a canopy image. The above-mentioned canopy height model is a digital model that describes the distribution of vegetation canopy height.
[0096] Specifically, since the treetop is often the highest point on a tree and the elevation of points around the treetop is significantly lower than the treetop, a local maximum filter is used to filter the point cloud, and the treetop in the canopy height model is identified by identifying local peak points, so that the treetop image can be obtained. The watershed algorithm is an image segmentation technology that is widely used in image processing and computer vision. Its name comes from the concept of watershed in geography, which refers to treating an image as terrain and segmenting the image into different areas by simulating the process of water flowing in the terrain. In the watershed algorithm, the grayscale value of the image is regarded as the height of the terrain. High grayscale values represent peaks and low grayscale values represent valleys. First, the height threshold is set according to the characteristics of the dominant tree species in the transmission scenario, and the valley areas in the image are marked. These areas are usually the segmentation objects we are interested in, that is, the crowns that are often at a lower level and in contact with other trees. The process of simulating the water flow starts from the marked valley, and the water flow expands from the valley to the peak. When the water flows from different valleys to meet, a watershed is formed. The watershed is where the water flows meet, and these locations are the edges in the image. Through the watershed algorithm, the image is segmented into multiple independent regions, each of which represents an object or background, thereby segmenting the edge of the tree crown and obtaining the tree crown image.
[0097] The cropping unit 50 is used to crop the above-mentioned treetop image and the above-mentioned crown image to obtain multiple center blocks and multiple circumscribed rectangular blocks respectively. The above-mentioned center block is a block cropped from the above-mentioned treetop image with the above-mentioned treetop as the center, and the above-mentioned circumscribed rectangular block is a block cropped from the above-mentioned crown image with the circumscribed rectangle of the above-mentioned crown as the edge.
[0098] Specifically, the difference in tree species often leads to huge differences in tree morphology: for example, Pinaceae trees often do not have large branches and have obvious tops; Moraceae trees often have many branches of similar thickness and luxuriance and do not have obvious treetops. In order to meet the classification requirements of morphological differences between different tree species, the acquired image data is cropped from two perspectives. The first is a tile cropped with the treetop as the center, with a tile size of 64*64; the second is a tile cropped with the circumscribed rectangle of the crown as the edge, which is scaled to the same size as the tile in the center of the treetop after cropping.
[0099] The classification unit 60 is used to input the first input data and the second input data into the improved Transformer classification model at the same time to obtain the single-tree-level tree species classification results along the transmission corridor, the above-mentioned first input data is the above-mentioned target feature combination and the above-mentioned central block, the above-mentioned second input data is the above-mentioned target feature combination and the above-mentioned circumscribed rectangular block, and the above-mentioned improved Transformer classification model is a Transformer model that obtains the above-mentioned single-tree-level tree species classification results through dual-channel input and parallel input processing.
[0100] Specifically, Figure 4 As shown in the figure, the single input of the Transformer model is improved into two dual inputs that do not affect each other. After the dual inputs enter the model, they are completely parallel and unrelated in other modules except the attention module, but in the multi-head self-attention module, the two share the same attention. This allows the blocks generated from different angles for the same actual object to refer to each other, strengthen their own classification according to each other's global features, and minimize the correlation between the classifications of the two, so that the last two can generate classification results from different angles. For different dominant tree species in the transmission corridor scene, one of the two channels will always adapt to its tree shape characteristics. According to the list of dominant tree species, the corresponding classification results are extracted from different channels and combined into the final classification results. The problem of poor recognition accuracy caused by the large difference in tree species posture is greatly reduced. Therefore, a dual-channel Transformer classifier is used for tree species recognition. The target feature combination and the two cropped blocks are combined and input into the dual-channel Transformer classifier for tree species classification.
[0101] In this embodiment, multispectral images, thermal infrared images and point cloud data along the transmission corridor are collected; then the multispectral images, thermal infrared images and point cloud data are feature processed to obtain multiple features; the best feature combination, i.e., the target feature combination, is obtained by calculating the JM distance saturation between features; the treetops are identified from the canopy height model using a local maximum filter, and then the outline of the crown is delineated from the canopy height model using a watershed segmentation algorithm; multiple central blocks are cropped from the feature data with the single treetop as the center, and multiple circumscribed rectangular blocks are cropped with the circumscribed rectangle of the single tree crown outline, and scaled to the same size as the central block; finally, the improved Transformer classification model is used to perform feature learning on the central block and the circumscribed rectangular block respectively, and the tree species category corresponding to each block is identified, and finally the single tree-level tree species classification results along the transmission corridor are obtained. This application solves the problem of low accuracy in the classification of single tree-level tree species in the prior art.
[0102] In order to accurately reflect the canopy height of the tree and help analyze and study the height and structure of the tree, in an optional embodiment, the device also includes:
[0103] A filtering unit, used for performing noise filtering on the point cloud data to obtain valid point cloud data before performing feature processing on the multi-source remote sensing data to obtain multiple features;
[0104] A registration unit, used for registering the airborne laser point cloud data and the backpack laser point cloud data in the valid point cloud data by using a registration algorithm to obtain fused point cloud data;
[0105] A removal unit is used to remove discrete abnormal points in the above fused point cloud data to obtain effective fused point cloud data;
[0106] A distinguishing unit, used for distinguishing ground points and non-ground points in the above-mentioned effective fused point cloud data according to a set elevation threshold, to obtain a ground point set and a non-ground point set;
[0107] A rasterization unit, used to rasterize the ground point set into a digital elevation model using a natural neighbor interpolation method, and to rasterize the non-ground point set into a digital surface model;
[0108] The calculation unit is used to calculate the difference between the digital elevation model and the digital surface model to obtain the canopy height model.
[0109] In the above embodiment, if Figure 5The upper part (i.e., the point cloud data part) is processed. The airborne laser radar point cloud data and the backpack laser radar point cloud data are noise filtered and denoised, and the airborne laser point cloud and backpack laser point cloud data are registered to generate fused point cloud data, and the fused point cloud data is used to generate the canopy height model (CHM). Specifically, the following steps are included: first, the discrete abnormal points in the fused point cloud data are removed to obtain the effective fused point cloud data; the ground points and non-ground points in the fused point cloud data are separated according to the height characteristics of the dominant tree species in the transmission corridor, that is, a suitable elevation threshold is set to separate the ground point set and the non-ground point set of the effective fused point cloud data, and the points whose elevation is less than the above-set elevation threshold are the above-mentioned ground points, and the points whose elevation is greater than or equal to the above-set elevation threshold are the above-mentioned non-ground points; the ground points and non-ground points are rasterized into a digital elevation model (DEM) and a digital surface model (DSM) respectively by the natural neighbor interpolation method; the difference between the digital elevation model (DEM) and the digital surface model (DSM) is calculated, and the difference is the canopy height model (CHM) to provide tree canopy height information. CHM can accurately reflect the tree canopy height, which is helpful for analyzing and studying the tree height and tree structure. The above-mentioned registration method is mainly a feature-based registration method, which includes the following steps: feature extraction and description: extract key feature points, such as corner points and edge points, from the effective point cloud data, and calculate descriptors for each feature point, such as FPFH (Fast Point Feature Histogram); feature matching: use descriptors to find corresponding feature point pairs in multi-platform point cloud data; coarse registration: according to the matched feature point pairs, the initial transformation matrix is estimated based on RANSAC (Random Sampling Consensus Algorithm); fine registration: use ICP (Iterative Closest Point) algorithm to make fine adjustments based on the preliminary registration.
[0110] In order to maximize the extraction of tree species features and enhance the reliability of identification, in an optional implementation, the feature processing unit includes:
[0111] A first feature processing module performs a preprocessing operation and a registration and stitching operation on the multispectral image and the thermal infrared image to obtain image features, wherein the image features include the texture features, the visible light band features, the near infrared band features, the thermal infrared band features and the normalized vegetation index features, the registration and stitching operation at least includes orthogonal correction, radiation calibration, stitching and resampling, and the preprocessing operation is an operation of preprocessing the image;
[0112] The second feature processing module uses the canopy height model to perform feature extraction processing on the point cloud data to obtain the photogrammetry point cloud posture features.
[0113] In the above embodiments, the multispectral image, the thermal infrared image and the point cloud data are processed in different ways. Figure 5 As shown, the processing methods for the above multispectral images and the above thermal infrared images include orthogonal correction, radiation calibration, splicing and resampling, extracting the grayscale co-occurrence matrix of multispectral data to generate texture information, extracting visible light R, G, B bands, near infrared bands, thermal infrared bands and specific vegetation indices to calculate and extract spectral features and vegetation indices. The processing method for point cloud data is to perform rasterization processing to generate a canopy height model (CHM), and then use the canopy height model to perform feature extraction processing on the above point cloud data to obtain the above photogrammetric point cloud posture feature (PPC). Combining multispectral data, point cloud data and tree 3D posture features (that is, photogrammetric point cloud posture features PPC) for tree species recognition can significantly improve the accuracy and robustness of recognition. Multispectral data provides rich spectral features, reflecting the spectral differences of different tree species in different bands, while point cloud data and 3D posture data provide detailed geometric and structural features, such as tree height, canopy shape, trunk diameter, and arrangement of branches and leaves. These visual and posture features complement each other and can more comprehensively describe the uniqueness of tree species and enhance the reliability of identification.
[0114] In order to extract the spectral differences of different tree species in different bands and comprehensively describe the uniqueness of the tree species, in an optional implementation manner, the first feature processing module includes:
[0115] A preprocessing submodule performs the preprocessing operation on the multispectral image and the thermal infrared image to enhance the quality of the images, and obtains a valid multispectral image and a valid thermal infrared image respectively;
[0116] A first extraction submodule performs the above-mentioned registration and splicing operation on the above-mentioned effective multispectral image and the above-mentioned effective thermal infrared image, and simultaneously extracts the gray-level co-occurrence matrix and spectral band information;
[0117] A second extraction submodule extracts the texture features from the gray-level co-occurrence matrix;
[0118] The third extraction submodule extracts the visible light band, the near infrared band and the thermal infrared band from the spectral band information to obtain the visible light band feature, the near infrared band feature and the thermal infrared band feature respectively;
[0119] The calculation submodule calculates the above-mentioned normalized vegetation index characteristics according to the first formula. The above-mentioned first formula is: NIR represents the above-mentioned near-infrared band characteristics, R represents the reflectance value of red light in the above-mentioned visible light band, and NDVI represents the above-mentioned normalized vegetation index characteristics.
[0120] In the above embodiment, the multispectral image and the thermal infrared image are first processed to enhance the image quality, and effective multispectral images and effective thermal infrared images are obtained respectively. The effective multispectral image and the effective thermal infrared image are subjected to orthogonal correction, radiation calibration, splicing and resampling, and the visible light R, G, B bands, near infrared bands, and thermal infrared bands are extracted, that is, the visible light band features, the near infrared band features, and the thermal infrared band features. The corresponding normalized vegetation index NDVI is calculated according to the extracted band data, and the calculation method is the first formula above. Then the grayscale co-occurrence matrix of the object is extracted and the texture features are extracted from it. The texture features are mainly reflected in eight parameters: angular moment (SM), contrast (con), variance (var), homogeneity (homm), correlation (cor), entropy (ent), dissimilarity and texture average features.
[0121] In order to improve the discrimination between tree species with significant morphological differences, in an optional implementation, the second feature processing module includes:
[0122] Determine a submodule, and determine an extraction standard based on the pixel size of the multispectral image, wherein the extraction standard is to set the maximum elevation of all pixel points in each pixel, the mean of all elevation points in each pixel, and the entropy and key percentile of the elevation distribution in each pixel;
[0123] The fourth extraction submodule extracts point cloud posture information that meets the above extraction criteria from the above canopy height model to obtain the above photogrammetry point cloud posture features.
[0124] In the above embodiment, in order to strengthen the feature extraction of point cloud data, the photogrammetric point cloud posture feature (PPC) is added to the classification of tree species identification as feature data describing the posture feature of a single tree. The addition of the photogrammetric point cloud posture feature greatly improves the depiction of the morphological features of the tree species itself by the point cloud feature data, increases the extracted posture features, and effectively improves the accuracy of tree species identification. The photogrammetric point cloud posture feature is extracted from the canopy height model generated by the point cloud by setting a series of thresholds, such as the statistical characteristics of the elevation of the point cloud in the canopy height model, the specific percentile morphological features, etc. These features can well represent the morphological characteristics of the trees themselves. For example, for multi-branch tree species, the point cloud features at 90% of the elevation positions are generally dispersed, and gradually close as the percentage decreases; while for single treetop tree species, the opposite is true. In this way, the morphological differences between the two trees are well reflected, and the deeper morphological features of the tree species themselves are also extracted. Based on the pixel size of the multispectral image, the corresponding point cloud posture feature data are extracted from the fused point cloud according to the established standards. The establishment of standards should be based on the ability to express the morphological characteristics of point clouds. The standards should be: the maximum elevation of all points in each pixel, the mean of all elevation points in each pixel, the entropy of the elevation distribution in each pixel, and the critical percentile of the elevation distribution in each pixel.
[0125] In order to avoid poor recognition accuracy caused by large correlation between features in a multi-feature combination, in an optional implementation, the feature combination unit includes:
[0126] A calculation module is used to perform a calculation step to sequentially calculate the JM distance between the reference feature and the first feature according to a second formula, wherein the second formula is: Where JM represents the above JM distance, d M represents the Mahalanobis distance between the reference feature and the first feature, μ1 and μ2 are the mean vectors of the reference feature and the first feature, ∑1 and ∑2 are the covariance matrices of the reference feature and the first feature, and T represents the transpose of the difference between the mean vectors of the reference feature and the first feature;
[0127] A combination module, used for executing the combination step, adding the current feature to the reference feature combination, the current feature being the first feature having the largest JM distance calculated with the reference feature, and the reference feature combination being the combination formed by the features having the largest JM distance;
[0128] A screening module, used to perform a screening step to screen out the current feature from a first feature set, where the first feature set is a set formed by all the first features;
[0129] A repeating module is used to repeatedly execute the calculation step, the combination step and the screening step at least once in sequence, until the number of the features in the reference feature combination reaches a set number or the JM distance reaches saturation, and the reference feature combination is determined as the target feature combination.
[0130] In the above embodiment, in order to solve the problem of poor recognition accuracy caused by large correlation between features in multiple feature combinations in tree species recognition, the JM distance is used to determine the feature combination, such as Figure 5 In the lower right part, one feature is selected from all the features as the reference feature, and it is combined with other features in turn and the change of the JM distance of the combination is calculated. The combination with the largest JM distance is retained, and the remaining features are combined with the combination in turn and the change of the JM distance is calculated. When the number of features in the group reaches the requirement or the JM distance is saturated and no longer increases, a set of suitable feature combinations is obtained. This operation is performed on each extracted feature in turn, and the combinations with too similar components are eliminated from the final result until the number of combinations reaches the requirement, so that a series of feature combinations that meet the classification requirements are obtained, that is, the above-mentioned target feature combination. The JM distance is the "distance" calculated between different objects on a single band image. The larger the distance, the better the separability of the two categories of objects in the band. In a feature combination containing multiple feature categories, the larger the JM distance corresponding to the combination, which means that the correlation between the features in the combination is lower, and the target object can be better classified among these features. When the JM distance tends to saturate, the separability will not change even if new features are added. At this time, the classification performance of the feature combination is optimal.
[0131] In order to avoid the reduction of recognition accuracy due to the large difference in tree species morphology, in an optional implementation, the cutting unit includes:
[0132] A first cropping module crops the treetop image according to a set block size with the treetop as the center to obtain the center block;
[0133] The second cropping module crops the tree crown image using the circumscribed rectangle of the tree crown as an edge to obtain a preliminary circumscribed rectangular image block;
[0134] The scaling module scales the preliminary circumscribed rectangular block to be consistent with the set block size to obtain the circumscribed rectangular block.
[0135] In the above embodiment, if Figure 5The upper right part of the image (i.e., the treetop center block and the crown circumscribed rectangular block part) performs two types of cropping for each identified treetop and crown, namely, the treetop-centered cropping block and the crown circumscribed rectangular cropping block. These two types of blocks take into account the differences in various morphological characteristics of trees. The classification effect of multi-branched tree species is better in the crown circumscribed rectangular block group, and the classification effect of single-treetop tree species is better in the treetop center block group. Different tree species often lead to huge differences in tree morphology: for example, pine trees often have no large branches and have obvious highest treetops; Moraceae trees often have many branches with similar thickness and luxuriance and no obvious treetops. In order to solve the problem of reduced recognition accuracy caused by excessive morphological differences between tree species, two tree shape block cropping methods are used. The treetop-centered cropping block and the crown circumscribed rectangular cropping block correspond to the two common tree shapes in nature: multi-branched shape and single treetop shape. The problem of reduced recognition accuracy caused by large differences in tree species morphology was avoided. Therefore, the acquired image data was cropped from two perspectives. The first was to crop the block with the treetop as the center, with a block size of 64*64; the second was to crop the block with the circumscribed rectangle of the tree crown as the edge, and after cropping, it was scaled to the same size as the center block of the treetop.
[0136] The above-mentioned tree species classification device along the power transmission corridor includes a processor and a memory. The above-mentioned acquisition unit, feature processing unit and feature combination unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in the form of any combination.
[0137] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of low accuracy of single-wood-level tree species classification in the prior art can be solved by adjusting kernel parameters.
[0138] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0139] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method for classifying tree species along the transmission corridor.
[0140] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the program executes the above-mentioned method for classifying tree species along the power transmission corridor when running.
[0141] An embodiment of the present invention provides a tree species classification system along a power transmission corridor. The tree species classification system along a power transmission corridor includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented:
[0142] Step S201, acquiring multi-source remote sensing data, the multi-source remote sensing data including multi-spectral images, thermal infrared images and point cloud data, the point cloud data including airborne laser point cloud data and backpack laser point cloud data;
[0143] Step S202, feature processing is performed on the multi-source remote sensing data to obtain multiple features, wherein the features at least include texture features, visible light band features, near infrared band features, thermal infrared band features, normalized vegetation index features, and photogrammetric point cloud posture features, wherein the photogrammetric point cloud posture features are morphological features of trees in the point cloud data;
[0144] Step S203, sequentially calculating the JM distance between the reference feature and the first feature, so as to select a plurality of the features according to the JM distance to generate a target feature combination, wherein the reference feature is any one of the features, the first feature is any one of the features other than the reference feature, and the target feature combination is a combination of the reference feature and a plurality of the first features;
[0145] Step S204, using a local maximum filter to identify treetops from the canopy height model to obtain a treetop image, and using a watershed segmentation algorithm to delineate the outline of the treetops from the canopy height model to obtain a canopy image, wherein the canopy height model is a digital model that describes the distribution of vegetation canopy height;
[0146] Step S205, cutting the treetop image and the tree crown image to obtain a plurality of center blocks and a plurality of circumscribed rectangular blocks, respectively. The center block is a block cut from the treetop image with the treetop as the center, and the circumscribed rectangular block is a block cut from the tree crown image with the circumscribed rectangle of the tree crown as the edge.
[0147] Step S206, input the first input data and the second input data into the improved Transformer classification model at the same time to obtain the single-tree-level tree species classification results along the transmission corridor, the above-mentioned first input data is the above-mentioned target feature combination and the above-mentioned central block, the above-mentioned second input data is the above-mentioned target feature combination and the above-mentioned circumscribed rectangular block, and the above-mentioned improved Transformer classification model is a Transformer model that obtains the above-mentioned single-tree-level tree species classification results through dual-channel input and parallel input processing.
[0148] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:
[0149] Step S201, acquiring multi-source remote sensing data, the multi-source remote sensing data including multi-spectral images, thermal infrared images and point cloud data, the point cloud data including airborne laser point cloud data and backpack laser point cloud data;
[0150] Step S202, feature processing is performed on the multi-source remote sensing data to obtain multiple features, wherein the features at least include texture features, visible light band features, near infrared band features, thermal infrared band features, normalized vegetation index features, and photogrammetric point cloud posture features, wherein the photogrammetric point cloud posture features are morphological features of trees in the point cloud data;
[0151] Step S203, sequentially calculating the JM distance between the reference feature and the first feature, so as to select a plurality of the features according to the JM distance to generate a target feature combination, wherein the reference feature is any one of the features, the first feature is any one of the features other than the reference feature, and the target feature combination is a combination of the reference feature and a plurality of the first features;
[0152] Step S204, using a local maximum filter to identify treetops from the canopy height model to obtain a treetop image, and using a watershed segmentation algorithm to delineate the outline of the treetops from the canopy height model to obtain a canopy image, wherein the canopy height model is a digital model that describes the distribution of vegetation canopy height;
[0153] Step S205, cutting the treetop image and the tree crown image to obtain a plurality of center blocks and a plurality of circumscribed rectangular blocks, respectively. The center block is a block cut from the treetop image with the treetop as the center, and the circumscribed rectangular block is a block cut from the tree crown image with the circumscribed rectangle of the tree crown as the edge.
[0154] Step S206, input the first input data and the second input data into the improved Transformer classification model at the same time to obtain the single-tree-level tree species classification results along the transmission corridor, the above-mentioned first input data is the above-mentioned target feature combination and the above-mentioned central block, the above-mentioned second input data is the above-mentioned target feature combination and the above-mentioned circumscribed rectangular block, and the above-mentioned improved Transformer classification model is a Transformer model that obtains the above-mentioned single-tree-level tree species classification results through dual-channel input and parallel input processing.
[0155] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0157] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0160] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0161] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0162] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0163] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0164] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:
[0165] 1) The classification method of tree species along the power transmission corridor of the present application is to collect multispectral images, thermal infrared images and point cloud data along the power transmission corridor; then perform feature processing on the multispectral images, thermal infrared images and point cloud data to obtain multiple features; obtain the best feature combination, i.e., the target feature combination, by calculating the JM distance saturation between features; use the local maximum filter to identify the treetops from the canopy height model, and then use the watershed segmentation algorithm to circle the outline of the crown from the canopy height model; with the single treetop as the center, cut out multiple center blocks from the feature data, and then cut out multiple circumscribed rectangular blocks with the circumscribed rectangle of the single tree crown outline, and scale them to the same size as the center block; finally, use the improved Transformer classification model to perform feature learning on the center block and the circumscribed rectangular block respectively, identify the tree species category corresponding to each block, and finally obtain the classification results of single tree-level tree species along the power transmission corridor. The present application solves the problem of low accuracy of single tree-level tree species classification in the prior art.
[0166] 2) The classification device of tree species along the power transmission corridor of the present application collects multispectral images, thermal infrared images and point cloud data along the power transmission corridor; then the multispectral images, thermal infrared images and point cloud data are processed to obtain multiple features; the best feature combination, i.e., the target feature combination, is obtained by calculating the JM distance saturation between features; the treetops are identified from the canopy height model using a local maximum filter, and then the outline of the crown is delineated from the canopy height model using a watershed segmentation algorithm; multiple central blocks are cropped from the feature data with the single treetop as the center, and multiple circumscribed rectangular blocks are cropped with the circumscribed rectangle of the single tree crown outline, and scaled to the same size as the central block; finally, the improved Transformer classification model is used to perform feature learning on the central block and the circumscribed rectangular block respectively, and the tree species category corresponding to each block is identified, and finally the classification results of single-tree-level tree species along the power transmission corridor are obtained. The present application solves the problem of low accuracy of single-tree-level tree species classification in the prior art.
[0167] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A method for classifying tree species along a power transmission corridor, characterized in that: include: Acquiring multi-source remote sensing data, wherein the multi-source remote sensing data includes multispectral images, thermal infrared images, and point cloud data, wherein the point cloud data includes airborne laser point cloud data and backpack laser point cloud data; Performing feature processing on the multi-source remote sensing data to obtain a plurality of features, wherein the features at least include texture features, visible light band features, near infrared band features, thermal infrared band features, normalized vegetation index features, and photogrammetric point cloud posture features, wherein the photogrammetric point cloud posture features are morphological features of trees in the point cloud data; sequentially calculating the JM distance between the reference feature and the first feature, so as to select a plurality of the features according to the JM distance to generate a target feature combination, wherein the reference feature is any one of the features, the first feature is any one of the features other than the reference feature, and the target feature combination is a combination of the reference feature and the plurality of the first features; Using a local maximum filter to identify treetops from a canopy height model to obtain a treetop image, and using a watershed segmentation algorithm to delineate the outline of the treetops from the canopy height model to obtain a canopy image, wherein the canopy height model is a digital model that describes the distribution of vegetation canopy height; The treetop image and the tree crown image are cropped to obtain a plurality of central blocks and a plurality of circumscribed rectangular blocks, respectively, wherein the central block is a block cropped from the treetop image with the treetop as the center, and the circumscribed rectangular block is a block cropped from the tree crown image with the circumscribed rectangle of the tree crown as the edge; The first input data and the second input data are simultaneously input into the improved Transformer classification model to obtain the single-tree-level tree species classification results along the transmission corridor, wherein the first input data is the target feature combination and the central block, and the second input data is the target feature combination and the circumscribed rectangular block. The improved Transformer classification model is a Transformer model that obtains the single-tree-level tree species classification results through dual-channel input and parallel input processing.
2. The method according to claim 1, characterized in that Before performing feature processing on the multi-source remote sensing data to obtain a plurality of features, the method further includes: Performing noise filtering on the point cloud data to obtain valid point cloud data; Using a registration algorithm to register the airborne laser point cloud data and the backpack laser point cloud data in the valid point cloud data to obtain fused point cloud data; Eliminating discrete abnormal points in the fused point cloud data to obtain effective fused point cloud data; Distinguishing ground points and non-ground points in the effective fused point cloud data according to a set elevation threshold to obtain a ground point set and a non-ground point set; The ground point set is rasterized into a digital elevation model using a natural neighbor interpolation method, and the non-ground point set is rasterized into a digital surface model; The difference between the digital elevation model and the digital surface model is calculated to obtain the canopy height model.
3. The method according to claim 1, characterized in that The multi-source remote sensing data is subjected to feature processing to obtain multiple features, including: Performing a preprocessing operation and a registration and stitching operation on the multispectral image and the thermal infrared image to obtain image features, wherein the image features include the texture features, the visible light band features, the near infrared band features, the thermal infrared band features and the normalized vegetation index features, the registration and stitching operation at least includes orthogonal correction, radiation calibration, stitching and resampling, and the preprocessing operation is an operation of preprocessing the image; The canopy height model is used to perform feature extraction processing on the point cloud data to obtain the photogrammetric point cloud posture features.
4. The method according to claim 3, characterized in that The multispectral image and the thermal infrared image are preprocessed and registered and stitched to obtain image features, including: Performing the preprocessing operation on the multispectral image and the thermal infrared image to enhance the quality of the images, and obtaining a valid multispectral image and a valid thermal infrared image respectively; Performing the registration and splicing operation on the effective multispectral image and the effective thermal infrared image, and extracting gray-level co-occurrence matrix and spectral band information; Extracting the texture feature from the gray level co-occurrence matrix; Extracting a visible light band, a near infrared band and a thermal infrared band from the spectral band information, and obtaining the visible light band feature, the near infrared band feature and the thermal infrared band feature respectively; The normalized vegetation index feature is calculated according to a first formula, wherein the first formula is: NIR represents the near infrared band characteristics, R represents the reflectance value of red light in the visible light band, and NDVI represents the normalized difference vegetation index characteristics.
5. The method according to claim 3, characterized in that: The point cloud data is subjected to feature extraction processing using the canopy height model to obtain the photogrammetric point cloud posture features, including: Taking the pixel size of the multispectral image as a benchmark, determining an extraction standard, wherein the extraction standard is to set the maximum elevation of all pixel points in each pixel, the mean of all elevation points in each pixel, and the entropy and key percentile of the elevation distribution in each pixel; Point cloud posture information that meets the extraction criteria is extracted from the canopy height model to obtain the photogrammetric point cloud posture features.
6. The method according to claim 1, characterized in that The JM distance between the first feature and the second feature is calculated in sequence, and a plurality of the features are selected according to the JM distance to generate a target feature combination, including: A calculation step is to calculate the JM distance between the reference feature and the first feature in sequence according to a second formula, wherein the second formula is: Wherein, JM represents the JM distance, d M represents the Mahalanobis distance between the reference feature and the first feature, μ1 and μ2 are respectively the mean vectors of the reference feature and the first feature, ∑1 and ∑2 are respectively the covariance matrices of the reference feature and the first feature, and T represents the transpose of the difference between the mean vectors of the reference feature and the first feature; A combining step, adding the current feature to a reference feature combination, wherein the current feature is the first feature having the largest JM distance calculated with the reference feature, and the reference feature combination is a combination formed by the features having the largest JM distance; a screening step of screening the current feature from a first feature set, where the first feature set is a set formed by all the first features; The calculating step, the combining step and the screening step are sequentially and repeatedly performed at least once until the number of the features in the reference feature combination reaches a set number or the JM distance reaches saturation, and the reference feature combination is determined as the target feature combination.
7. The method according to claim 1, characterized in that The treetop image and the tree crown image are cropped to obtain a plurality of central blocks and a plurality of circumscribed rectangular blocks, respectively, including: Cutting the treetop image according to a set block size with the treetop as the center to obtain the center block; The tree crown image is cropped with the circumscribed rectangle of the tree crown as the edge to obtain a preliminary circumscribed rectangular image block; The preliminary circumscribed rectangular block is scaled to be consistent with the set block size to obtain the circumscribed rectangular block.
8. A device for classifying tree species along a power transmission corridor, characterized in that: The device comprises: An acquisition unit, used for acquiring multi-source remote sensing data, wherein the multi-source remote sensing data includes multi-spectral images, thermal infrared images and point cloud data, wherein the point cloud data includes airborne laser point cloud data and backpack laser point cloud data; A feature processing unit, used for performing feature processing on the multi-source remote sensing data to obtain a plurality of features, wherein the features at least include texture features, visible light band features, near infrared band features, thermal infrared band features, normalized vegetation index features and photogrammetric point cloud posture features, wherein the photogrammetric point cloud posture features are morphological features of trees in the point cloud data; A feature combination unit, used for sequentially calculating the JM distance between the reference feature and the first feature, so as to select a plurality of the features according to the JM distance to generate a target feature combination, wherein the reference feature is any one of the features, the first feature is any one of the features other than the reference feature, and the target feature combination is a combination of the reference feature and a plurality of the first features; An identification unit is used to identify treetops from a canopy height model using a local maximum filter to obtain a treetop image, and to delineate the outline of the treetops from the canopy height model using a watershed segmentation algorithm to obtain a canopy image. The canopy height model is a digital model that describes the distribution of vegetation canopy heights. a cropping unit, configured to crop the treetop image and the tree crown image to obtain a plurality of central image blocks and a plurality of circumscribed rectangular image blocks, respectively, wherein the central image block is an image block cropped from the treetop image with the treetop as the center, and the circumscribed rectangular image block is an image block cropped from the tree crown image with the circumscribed rectangle of the tree crown as the edge; A classification unit is used to input the first input data and the second input data into the improved Transformer classification model at the same time to obtain the single-tree-level tree species classification results along the transmission corridor, wherein the first input data is the target feature combination and the central block, and the second input data is the target feature combination and the circumscribed rectangular block. The improved Transformer classification model is a Transformer model that obtains the single-tree-level tree species classification results through dual-channel input parallel processing.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.
10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
