Tree fall risk assessment system and method based on tof depth camera

Abstract

The application provides a tree toppling risk assessment system and method based on a TOF depth camera, and the system comprises a TOF depth acquisition system, an edge detection module, a three-dimensional modeling module, an automatic identification module, a centroid estimation module and a risk assessment module. The application combines three-dimensional model reconstruction technology and mechanical principles based on a TOF camera, and adopts a specific formula algorithm to provide a new tree toppling assessment system and method, so that the tree toppling risk can be highly automatically assessed, the assessment efficiency and accuracy are improved, and the problem of a shortage of urban tree management professionals is relieved to a certain extent, and scientific basis is provided for prevention and decision-making.

Description

[0001] The present application relates to the field of tree collapse risk assessment, and particularly relates to a system and method for tree form three-dimensional reconstruction based on a TOF depth camera and tree collapse risk assessment through centroid estimation.

[0002] In recent years, the increase in extreme weather has made tree collapse accidents more frequent. To prevent tree collapse, support frames and other methods are generally used, for example, Chinese patents CN211881409U and CN207885322U have proposed support frames of different structures to prevent collapse, but these devices are only used when a collapse risk is found, and cannot directly find the risk. Currently, the collapse risk management of urban trees mainly relies on manual visual inspection by professionals, which is extremely costly and not very effective.

[0003] Chinese patent CN116022367A proposes a device for monitoring the collapse of utility poles and trees after a typhoon disaster using a drone. The structure of the drone is improved to make the overall drone stable and fixed to the ground. The utility poles and trees are observed for collapse risk through video recording. Compared to manual on-site visual inspection, this device has improved to some extent, but it only records videos and takes pictures and does not perform in-depth analysis afterwards. In simple terms, it focuses on image collection but lacks post-analysis, and still mainly relies on manual judgment. This is time-consuming and labor-intensive, and highly dependent on experience without scientific basis, resulting in a high rate of misjudgment.

[0004] The inventor has found that trees prone to collapse usually have unstable mechanical structures, such as excessively inclined trunks, insufficient root grip, excessively low height at the main trunk bifurcation, and damaged tree bodies. Based on these characteristics, the centroid position and inclination angle of the tree are estimated through a three-dimensional tree model to assess the risk of tree collapse.

[0005] ​​In addition, with the development of computer vision technology, the three-dimensional reconstruction of trees in real environment has attracted extensive attention, which provides a new idea for the automatic management of urban trees. In the past, the reconstruction of three-dimensional images was mostly based on two-dimensional RGB images, and the software development was complex, and the modeling accuracy was low under the condition of small number of collected images (such as ORB algorithm using two-dimensional image feature points for reconstruction). With the rapid development of depth camera technology, hardware can directly collect object depth information, which makes the three-dimensional reconstruction, target detection, environment perception and other applications have higher accuracy and practicality. Among various types of depth cameras, TOF camera has the characteristics of long effective distance (up to dozens of meters), active measurement less affected by environmental light, high frame rate (up to hundreds of fps), acceptable accuracy (centimeter level), reasonable price and the like, and is more suitable for tree measurement and three-dimensional modeling application. At the same time, the TOF depth camera is carried on the unmanned aerial vehicle platform as described in CN116022367A, which can realize highly automated data acquisition, risk assessment software and hardware system, and has high practicality. SUMMARY

[0006] The purpose of the present application is to solve the problems in the prior art, and to provide a tree collapse risk assessment system and method based on a TOF depth camera, which can accurately assess the tree collapse risk and improve the assessment efficiency and accuracy, and provide a scientific basis for prevention and decision-making.

[0007] To achieve the above-mentioned purpose, the present application provides a tree collapse risk assessment system based on a TOF depth camera, comprising:

[0008] TOF depth acquisition system: the unmanned aerial vehicle carries a TOF depth camera to collect multi-angle tree panoramic RGBD images;

[0009] Edge detection module: Canny operator or Laplace filter is used to analyze the position of the tree in the image according to the image, and an image composed only of trees is obtained, and information irrelevant to the three-dimensional reconstruction module is removed;

[0010] Three-dimensional modeling module: selecting a model coordinate system, edge detection, and establishing a tree point cloud model;

[0011] Automatic identification module: classifying each component of the tree into tree roots, tree trunks, branches and leaves, and detecting whether the depth information of the tree roots and tree trunks shows damage;

[0012] Centroid estimation module: according to different types of wood and parts, input the density multiplier of each component to calculate the centroid position of the tree;

[0013] Risk assessment module: according to the tree shape identified by the automatic identification module and the centroid position of the centroid estimation module, the tree collapse risk is evaluated through the principle of mechanics.

[0014] As preferred, the multi-angle in the TOF depth acquisition system is m angles, each angle has n frames of images, and the number of panoramic RGBD images is m*n.

[0015] As preferred, the coordinate system in the three-dimensional modeling module takes the centroid point of the tree root system plane as the origin, a certain angle RGB image plane with the largest trunk diameter as the x and z axes, and the direction perpendicular to the plane as the y axis.

[0016] As preferred, the modeling method is:

[0017] a) mark the input point information of the i-th frame as s i =(x, yd, c) i , where x and y are the horizontal / vertical pixel coordinates, d is the depth, and c is the RGB color (RGB color vector), and the number of frames should be m*n;

[0018] b) coordinate transformation is performed on all frames to obtain p i =(x, y, z, c) = T(s i )=R(s i )+t, where T(s i ) represents the coordinate transformation of the i-th frame information point, and R(s i ) and t represent the rotation transformation and displacement transformation obtained by the SIFT algorithm. (R, t) = SIFT(s i ), where SIFT uses a set of consecutive frames;

[0019] c) take multiple frames taken at the same angle as a group, and for images taken at different groups, introduce group transformation Calculate the projection error E i，j =∑||T i，j (s i )-s j ||2 for different groups of frames, and filter out the corresponding matches if there is a projection with too large error;

[0020] d) calculate the overall matching error of the system E = w sparse E sparse +w photo E photo +w geo E geo , and use the preconditioned conjugate gradient optimization method to minimize the projection error. The overall system error is composed of sparse matching error and dense matching error. The sparse matching error , the camera error in the dense matching error , grad is the calculation gradient function, π is the projection function of the camera, and the geometric error in the dense matching is , is to s i，k is the operator that regularizes the transpose into a projection matrix, D is the depth map, and w x is the weight coefficient to adjust the proportion of different matching errors;

[0021] e) Dynamically reconstruct the three-dimensional point cloud database, use the point cloud information obtained by minimizing the projection error for TSDF reconstruction, and update the weight value of the point cloud at the same time, and the update method is to input more optimized point clouds into the integration formula , W' = W + w, where D, W are the original voxel position and weight, d, w are the new voxel information, and D', W' are the updated voxel information.

[0022] As a preferred, the damage in the automatic identification module is divided into severe damage and general damage, the severe damage is included in the high-risk classification, and the general damage reduces the density multiplier in the subsequent centroid estimation module.

[0023] As a preferred, the identification method in the automatic identification module is:

[0024] a) Differentiate the tree root, trunk, branch, and leaf area in the point cloud database by color features, Mask x，y，z = argmax type (Classifier(pixel)), where type is a subscript indicating the tree root, trunk, branch, and leaf area;

[0025] b) Identify the trunk damage area by image segmentation, based on the trunk point cloud obtained by classification, use the Canny operator as the edge recognition algorithm to find the part with more color change points and identify the trunk damage area, Edge = (Canny(pixel)), defect = if density(Edge) > th, where Edge is the marked trunk image edge information, defect is the evaluation of the similarity between the trunk edge and the damage area by the density function, density is a function for analyzing defects according to the edge information given by the Canny operator, and th is a threshold value, which is higher than the threshold value will be marked as a damage area;

[0026] c) Remove noise in the three-dimensional point cloud with target identification information to improve system robustness;

[0027] d) Use Mask R-CNN neural network algorithm for overall image segmentation and identification, Mask x，y，z = arg max type (RCNN(pixel)), type includes trunk, branch, leaf, root, and part of tree damage.

[0028] As preferred, the technical method of the center-of-mass estimation module is to first calculate the center-of-mass coordinates of each object, and then calculate the overall center-of-mass coordinates by using the multiplication coefficient method.

[0029] As preferred, the technical method is: (x mass , y mass , z mass ) = ∑(π type × ρ type × (x i , y i , z i )) / m all , wherein (x mass , y mass , z mass ) is the center-of-mass position coordinate of the tree to be obtained, (x i , y i , z i ) is the center-of-mass position coordinate of each component; π type is the prior information introduced according to different parts; ρ type is the density function of different parts of the tree; m all is the total mass, which can be obtained by integrating ρ type , and the weighted average coordinate of the above factors is the center-of-mass position coordinate of the tree.

[0030] As preferred, the criteria for assessing the existence of risks in the risk assessment module include, but are not limited to, one or a combination of the following criteria:

[0031] a) The center-of-mass is too high: whether the ratio of the center-of-mass height H to the total height H0 of the tree H / H0 exceeds a set threshold value;

[0032] b) The inclination angle is too large: whether the angle θ formed between the line connecting the center-of-mass and the fulcrum and the vertical direction exceeds a set threshold value;

[0033] c) The inclination moment is too large: whether the product of the gravity G and the distance D between the center-of-mass and the fulcrum exceeds a set threshold value.

[0034] To achieve the above-mentioned purposes, the present application further provides a tree toppling risk assessment method based on a TOF depth camera, which comprises the following steps in sequence:

[0035] a) Obtaining a set of multi-angle RGBD depth images of the tree;

[0036] b) Performing edge detection on the obtained RGBD images to extract tree-related information points;

[0037] c) Judging whether the tree has been collected, if not, returning to step a), and if yes, entering step d);

[0038] d) according to the image set representing the tree information, selecting a coordinate system, and establishing a three-dimensional point cloud data set of the tree;

[0039] e) evaluating whether the modeling effect meets the standard, if not, introducing more collected data to improve the accuracy and then returning to step d), if yes, entering step f);

[0040] f) in the three-dimensional point cloud data set, identifying each component such as the root, trunk, branch and leaf of the tree, establishing a three-dimensional point cloud sub-data set, and identifying the damage of the root and trunk area;

[0041] g) according to the three-dimensional point cloud sub-data set, estimating the centroid of each component, and then estimating the overall centroid of the tree;

[0042] h) according to the estimated centroid position, performing mechanical analysis and evaluating the tree toppling risk.

[0043] The beneficial effects of the present application are: the present application is based on a TOF camera, combines three-dimensional model reconstruction technology and mechanical principles, adopts a unique formula algorithm to propose a new tree toppling evaluation system and method, can highly automatically evaluate the tree toppling risk, improves the evaluation efficiency and accuracy, and to some extent alleviates the problem of shortage of urban tree management professionals, provides a scientific basis for prevention and decision-making.

[0044] The features and advantages of the present application will be described in detail in conjunction with the embodiments and the accompanying drawings.

DRAWINGS

[0045] Figure 1 is a structural schematic diagram of the tree toppling risk evaluation system of the present application based on a TOF depth camera;

[0046] Figure 2 is a schematic diagram of the centroid estimation mechanical model in the present application;

[0047] Figure 3 is an algorithm flowchart of the tree toppling risk evaluation method of the present application based on a TOF depth camera.

DETAILED DESCRIPTION

[0048] Example 1:

[0049] Referring to Figure 1 , the tree toppling risk evaluation system based on a TOF depth camera comprises:

[0050] TOF depth acquisition system: a UAV carries a TOF depth camera to acquire multi-angle panoramic RGBD images of the tree; in the TOF depth acquisition system, the multi-angle is m angles, there are n frames of images at each angle, and the number of panoramic RGBD images is m*n.

[0051] Edge detection module: Canny operator or Laplace filter is used to analyze the position of trees in the image and obtain an image composed of trees only, removing information irrelevant to the three-dimensional reconstruction module.

[0052] Three-dimensional modeling module: Select the model coordinate system, for example, the centroid point of the tree root plane as the origin, the largest diameter of the trunk at a certain angle of the RGB image plane as the x and z axes, and the direction perpendicular to the plane as the y axis. Edge detection is performed to establish a tree point cloud model. The following is an embodiment of the modeling steps:

[0053] a) Label the input point information of the i-th frame as s i = (x, y, d, c) i , where x and y are the horizontal / vertical coordinates of the pixels, d is the depth, and c is the RGB color (RGB color vector). The number of frames should be m*n;

[0054] b) Perform coordinate transformation on all frames to obtain p i = (x, y, z, c) = T(s i ) = R(s i ) + t, where T(s i ) represents the coordinate transformation of the i-th frame information point, and R(s i ) and t represent the rotation transformation and displacement transformation obtained by the SIFT algorithm. (R, t) = SIFT(s i ), where SIFT uses a set of consecutive frames;

[0055] c) Take multiple frames taken at the same angle as a group. For images taken at different angles, introduce inter-group transformation Calculate the projection error E i，j = ∑||T i，j (s i ) - s j ||2 for different groups of frames. If there is a projection with too large an error, filter out the corresponding match;

[0056] d) Calculate the overall system matching error E = w sparse E sparse + w phoo E photo + w geo E geo , and minimize the projection error using the preconditioned conjugate gradient optimization method. The overall system error is composed of sparse matching error and dense matching error. The sparse matching error , the camera error in the dense matching error , grad is the gradient calculation function, π is the camera projection function, the geometric error in the dense matching , is to s i，k is the operator that regularizes the transpose into a projection matrix, D is the depth map, and w x is the weight coefficient to adjust the proportion of different matching errors.

[0057] e) Dynamically reconstruct the three-dimensional point cloud database, use the point cloud information obtained by minimizing the projection error for TSDF reconstruction, and update the weight value of the point cloud at the same time. The update method is to input more optimized point clouds into the integration formula , W' = W + w, where D, W are the original voxel position and weight, d, w are the new voxel information, and D', W' are the updated voxel information.

[0058] Automatic recognition module: classify the parts of the tree into tree roots, tree trunks, branches, and leaves, and detect whether the tree root and trunk depth information shows damage; damage is divided into serious loss and general damage, serious damage is included in the high-risk classification, and general damage reduces the density multiplier in the subsequent centroid estimation module. The following is an embodiment of the recognition steps:

[0059] a) Differentiate the tree root, trunk, branch, and leaf areas in the point cloud database by color features, Mask x，y，z = argmax type (Classifier(pixel)), where the type index represents the tree root, trunk, branch, and leaf area;

[0060] b) Identify the trunk damage area through image segmentation. Based on the classified trunk point cloud, use the Canny operator as the edge recognition algorithm to find the part with more color change points and identify the trunk damage area, Edge = (Canny(pixel)), defect = if density(Edge) > th, where Edge is the labeled trunk image edge information, defect is the evaluation of the similarity between the trunk edge and the damage area through the density function, density is a function that analyzes defects based on the edge information given by the Canny operator, and th is the threshold value. Higher than the threshold value will be marked as a damage area;

[0061] c) Remove noise from the three-dimensional point cloud with target recognition information to improve system robustness;

[0062] d) Use the Mask R-CNN neural network algorithm for overall image segmentation and recognition, Mask x，y，z = arg max type (RCNN(pixel)), type includes tree trunk, branch, leaf, root, and tree damage.

[0063] centroid estimation module: according to different wood species, parts, input the density multiplier of each component, calculate the centroid position of the tree. See Figure 2 , first calculate the centroid coordinates of each object (each object is approximately uniform density, calculate the geometric center), and then calculate the overall centroid coordinates by multiplying the coefficient. The detailed calculation method is as follows: (x mass , y mass , z mass ) = ∑(π type × ρ type × (x i , y i , z i )) / / m all , where (x mass , y mass , z mass ) is the centroid position coordinate of the tree to be solved, (x i , y i , z i ) is the centroid position coordinate of each component; π type is the prior information introduced according to different parts; ρ type is the density function of different parts of the tree; m all is the total mass, which can be obtained by integrating ρ type , and the weighted average coordinate of the above factors is the centroid position coordinate of the tree.

[0064] risk assessment module: according to the tree shape identified by the automatic recognition module and the centroid position estimated by the centroid estimation module, the tree falling risk is evaluated through the principle of mechanics. The standards for evaluating the risk include but are not limited to one or a combination of the following standards:

[0065] a) centroid is too high: see Figure 2 , the ratio of centroid height H to total tree height H0: whether H / H0 exceeds the set threshold;

[0066] b) inclination angle is too large: whether the angle θ formed by the line connecting the centroid and the fulcrum (approximately the geometric center position of the tree root) and the vertical direction exceeds the set threshold;

[0067] c) inclination moment is too large (root system stress is too large): whether the product of gravity G and the distance D between the centroid and the fulcrum exceeds the set threshold.

[0068] see Figure 3 , the tree falling risk assessment method based on TOF depth camera, which comprises the following steps in turn:

[0069] a) obtain a set of multi-angle RGBD depth images of the tree;

[0070] b) edge detection is performed on the acquired RGBD image to extract tree-related information points;

[0071] c) it is determined whether the tree is collected, if not, go back to step a), if yes, go to step d);

[0072] d) according to the image set representing the tree information, a coordinate system is selected, and a three-dimensional point cloud data set of the tree is established;

[0073] e) it is determined whether the modeling effect meets the standard, if not, more data is introduced to improve the accuracy and then go back to step d), if yes, go to step f);

[0074] f) in the three-dimensional point cloud data set, each component such as root, trunk, branch and leaf is identified, a three-dimensional point cloud sub-data set is established, and damage identification is performed on the root and trunk area;

[0075] g) according to the three-dimensional point cloud sub-data set, the centroid of each component is estimated, and the overall centroid of the tree is estimated;

[0076] h) according to the estimated centroid position, mechanical analysis is performed to evaluate the risk of tree falling.

[0077] The above examples are illustrative of the present application, but not limiting of the present application. Any simple transformation of the present application belongs to the protection scope of the present application.

Claims

1. A tree-falling risk assessment system based on a TOF depth camera, characterized in that: include: TOF Depth Acquisition System: A drone equipped with a TOF depth camera acquires panoramic RGBD images of trees from multiple angles; Edge detection module: Using the Canny operator or Laplacian filter, the position of trees in the image is analyzed to obtain an image consisting only of trees, removing information irrelevant to the 3D reconstruction module; 3D modeling module: Select the model coordinate system, perform edge detection, and build a tree point cloud model; Automatic identification module: Classifies tree parts into roots, trunk, branches and leaves, and detects whether the depth information of roots and trunk shows damage; damage is classified into severe damage and general damage. Severe damage is included in the high-risk category, while general damage is reduced in the density multiplier in the subsequent centroid estimation module. Centroid Estimation Module: Based on different wood species and parts, input the density multiplier of each component to calculate the centroid location of the tree; Risk assessment module: Based on the tree shape identified by the automatic identification module and the centroid position of the centroid estimation module, the risk of tree falling is assessed through mechanical principles.

2. The tree-falling risk assessment system based on a TOF depth camera as described in claim 1, characterized in that: The TOF depth acquisition system has m angles, each angle has n frames, and the number of panoramic RGBD images is m*n.

3. The tree-falling risk assessment system based on a TOF depth camera as described in claim 1, characterized in that: In the 3D modeling module, the coordinate system takes the centroid of the tree root plane as the origin, the RGB image plane at a certain angle with the largest trunk diameter as the x and z axes, and the direction perpendicular to this plane as the y axis.

4. The tree-falling risk assessment system based on a TOF depth camera as described in claim 3, characterized in that: The modeling method is as follows: a) Mark the input point information of the i-th frame as , where x and y are the horizontal / vertical coordinates of the pixel, d is the depth, c is the RGB color (RGB color vector), and the number of frames should be m*n; b) Perform coordinate transformation on all frames to obtain , in This represents the coordinate transformation of the information point in the i-th frame, while and This represents the rotation and translation transformations obtained through the SIFT algorithm; Here, SIFT uses a set of consecutive frames. c) Take multiple frames taken from the same angle as a group, and then introduce inter-group transformation for images taken from different groups. Calculate the projection error for different groups of frames. If there is a projection with excessively large error, the corresponding match will be filtered out. d) Calculate the overall matching error of the system The projection error is minimized using a preprocessed conjugate gradient optimization method; the overall system error consists of sparse matching error and dense matching error. Sparse matching error Camera error in dense matching error grad is the function for calculating the gradient. It is the camera's projection function, and the geometric error in dense matching. , It is Operators that regularize the transpose to the projection matrix It is a depth map, and It is a weighting coefficient that adjusts different matching error ratios; e) Dynamically reconstruct the 3D point cloud database, using the point cloud information obtained by minimizing the projection error for TSDF reconstruction, and simultaneously update the weight values ​​of the point clouds. The method for updating the weights is to input more optimized point clouds into the integration formula. , Where D and W are the original voxel positions and weights, d and w are the newly added voxel information, and D' and W' are the updated voxel information.

5. The tree-falling risk assessment system based on a TOF depth camera as described in claim 1, characterized in that: The identification method in the automatic identification module is as follows: a) Differentiate tree root, trunk, branch, and leaf regions in the point cloud database using color features. The type subscript indicates the root, trunk, branch, and leaf regions. b) Identify tree trunk damage areas through image segmentation. Based on the tree trunk point cloud obtained from classification, use the Canny operator as an edge detection algorithm to find the parts with more color change points and identify the tree trunk damage areas. In this context, Edge represents the marked edge information of the tree trunk image, defect is the evaluation of the similarity between the tree trunk edge and the damaged area through the density function, density is the function that analyzes the defect based on the edge information given by the Canny operator, and th is the threshold; areas higher than the threshold will be marked as damaged areas. c) Combining steps a and b, noise in the 3D point cloud containing target recognition information is removed to improve system robustness; d) Use the Mask R-CNN neural network algorithm for overall image segmentation and recognition. The type includes the trunk, branches, leaves, roots, and damaged parts of the tree.

6. The tree-falling risk assessment system based on a TOF depth camera as described in claim 1, characterized in that: The technical method of the centroid estimation module is as follows: first calculate the centroid coordinates of each object, and then calculate the overall centroid coordinates using the multiplication factor method.

7. The tree-falling risk assessment system based on a TOF depth camera as described in claim 6, characterized in that: The technical method is as follows: ,in These are the coordinates of the tree's centroid to be determined. These are the coordinates of the centroid positions of each component; It is based on prior information introduced from different parts; It is a density function for different parts of the tree; That is the total mass, which can be determined by... Points are earned.

8. The tree-falling risk assessment system based on a TOF depth camera as described in any one of claims 1 to 7, characterized in that: The criteria used in the risk assessment module to assess the existence of risk include, but are not limited to, a combination of one or more of the following criteria: a) Center of mass too high: Center of mass height Total height of trees Ratio: Exceeding the set threshold; b) Excessive tilt angle: The angle formed by the line connecting the center of mass and the fulcrum with the vertical direction. Exceeding the set threshold; c) Excessive tilting moment: The product of gravity G and the distance D between the center of mass and the fulcrum exceeds the set threshold.

9. A tree-falling risk assessment method based on a TOF depth camera, characterized by: The steps are as follows: a) Obtain a multi-angle RGBD depth image set of trees; b) Perform edge detection on the acquired RGBD image and extract tree-related information points; c) Determine whether a tree has been collected. If not, return to step a); if yes, proceed to step d). d) Based on the image set representing tree information, select a coordinate system and establish a 3D point cloud dataset of trees; e) Evaluate whether the modeling effect meets the standard. If not, introduce more collected data to improve accuracy before returning to step d). If yes, proceed to step f. f) In the three-dimensional point cloud dataset, identify the components of the tree root, trunk, branches and leaves, establish a three-dimensional point cloud sub-dataset, and identify damage in the tree root and trunk regions; damage is classified into severe damage and general damage, severe damage is included in the high-risk classification, and general damage is reduced in the density multiplier in the subsequent centroid estimation module. g) Estimate the centroid of each component based on the aforementioned 3D point cloud sub-dataset, and then estimate the overall centroid of the tree; h) Based on the estimated centroid location, perform mechanical analysis to assess the risk of tree tipping over.

Images