Tree modeling method based on image acquisition and image recognition of unmanned aerial vehicle

Abstract

The invention relates to a tree modeling method based on image acquisition and image recognition of an unmanned aerial vehicle, and the method comprises the steps: planning a flight path of the unmanned aerial vehicle according to the scale of a target region and tree distribution, and collecting a multi-angle image of a tree; then, denoising, cutting and enhancing the image through preprocessing, and performing semantic segmentation on the image by using a pre-trained neural network to obtain a semantic tag of each pixel; image feature points are extracted through an SIFT algorithm and are matched, and a preliminary three-dimensional point cloud model is constructed by using a COLMAP incremental reconstruction method; according to the semantic segmentation result of the image, the semantic label of each pixel is transmitted to the point cloud data, and the semantic annotation of the point cloud is completed; traversing the point cloud data, and segmenting the point cloud according to the label to obtain a tree three-dimensional point cloud model; the method can be widely applied to the fields of vegetation monitoring, forest resource investigation, ecological environment protection and the like, and has the advantages of high precision, automation and high efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional image reconstruction, and in particular relates to a tree modeling method based on unmanned aerial vehicle (UAV) image acquisition and image recognition. Background Art

[0002] At present, the traditional tree resource survey method mainly relies on manual ground measurement and manual recording, which has many limitations. Manual ground measurement is inefficient, especially when conducting surveys in large forests or green areas, which requires a lot of time and human resources. The measurement of each tree requires precise positioning and multiple data records. This process is cumbersome and repetitive, which seriously affects the efficiency of the survey. In addition, manual measurement is labor-intensive and its accuracy is easily affected by human factors. Improper operation, improper use of measuring equipment, and differences in the experience of surveyors may lead to inaccurate data, thus affecting the final survey results.

[0003] Traditional manual measurement methods are also difficult to effectively obtain spatial data of trees. For example, traditional methods cannot quickly and accurately obtain three-dimensional information such as the spatial distribution and branch morphology of trees, which puts higher demands on modern ecological monitoring, forest resource management and related applications. This shows that traditional tree resource survey methods have certain limitations in terms of efficiency, accuracy and spatial data acquisition capabilities.

[0004] With the rapid development of drone technology, drones equipped with high-precision cameras have provided a new solution for tree resource surveys. Drones can cover a large area in a short period of time, and the collected data has high accuracy, which can efficiently collect tree information in complex terrain and environments. Drones can quickly capture images of trees from various angles through different flight plans, which provides more perspectives and data support for the three-dimensional modeling of trees. Since drones can easily fly over densely treed areas and avoid areas that are difficult to reach during manual measurements, they can greatly improve work efficiency.

[0005] Although drone technology can achieve large-scale and efficient tree image acquisition, how to extract effective tree information from a large amount of image data and convert it into an accurate three-dimensional model is still a major problem in current technology. Traditional image processing methods usually rely on manual segmentation of image areas or use rough visual algorithms for point cloud reconstruction. These methods have defects such as low accuracy, poor algorithm adaptability, and high processing complexity. For example, manual segmentation of image areas is not only inefficient, but also easily affected by human factors, resulting in insufficient accuracy in tree information extraction. Although the automatic point cloud reconstruction method based on visual algorithms can improve the degree of automation to a certain extent, it also faces the problem of insufficient recognition accuracy and robustness. In particular, the complex morphology, occlusion phenomenon, and similarity of trees to the background make it difficult for traditional visual algorithms to accurately extract tree information, resulting in poor quality of generated point clouds and unsatisfactory reconstruction effects.

[0006] Although the existing image-based tree modeling methods have improved the efficiency of tree information collection to a certain extent, they still have problems such as low recognition accuracy, poor point cloud quality, and limited reconstruction effect. These problems limit the effectiveness of existing technologies in practical applications, especially in scenarios that require high-precision tree modeling, traditional methods often fail to meet the requirements. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a tree modeling method based on drone image collection and image recognition.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] The present invention provides a tree modeling method based on unmanned aerial vehicle image collection and image recognition, comprising the following steps:

[0010] Step S1: planning the flight mode and flight path of the UAV according to the size, shape and distribution of trees in the target area;

[0011] Step S2: using a drone to fly in a target area according to a planned flight mode and flight path to collect multi-angle image data of trees, wherein the multi-angle image data includes a set of tree images taken at multiple different viewing angles and heights;

[0012] Step S3: preprocessing the multi-angle image data, including denoising, cropping, image enhancement, and standardization;

[0013] Step S4: input the preprocessed multi-angle image data into a pre-trained neural network model for tree recognition and semantic segmentation, and output a semantic segmentation map of each tree image, in which each pixel in the semantic segmentation map is assigned a semantic label;

[0014] Step S5: extract feature points and their descriptors from each tree image using the SIFT algorithm, match feature points between multiple tree images based on the feature points and their descriptors using the nearest neighbor matching algorithm, and remove mismatched feature points using the RANSAC algorithm;

[0015] Step S6: Based on the matched feature points, a preliminary three-dimensional point cloud model is constructed by using COLMAP through an incremental reconstruction method;

[0016] Step S7: Calculate the three-dimensional coordinates of each pixel in the semantic segmentation map obtained in step S4 in the preliminary three-dimensional point cloud model, determine the point cloud data corresponding to the pixel according to the three-dimensional coordinates of each pixel, and transfer the semantic label of each pixel to the corresponding point cloud data;

[0017] Step S8: traverse each point cloud data in the preliminary three-dimensional point cloud model, segment the preliminary three-dimensional point cloud model according to the semantic labels of the point cloud data, and obtain a three-dimensional point cloud model of the tree;

[0018] Step S9: Use the SOR filtering algorithm to perform noise reduction and optimization on the tree 3D point cloud model, use the Poisson reconstruction method to obtain the tree 3D mesh model of the tree after noise reduction and optimization, and construct the tree geometric model based on the tree 3D mesh model.

[0019] Furthermore, the semantic labels include tree trunks, leaves and background.

[0020] Furthermore, the neural network model is a Mask R-CNN model, and the training process of the Mask R-CNN model includes the following steps:

[0021] Obtain a collection of tree images taken from multiple different perspectives and heights, and preprocess the tree image collection, including denoising, cropping, image enhancement, and standardization;

[0022] Use the image annotation tool LabelMe to manually annotate each area in the preprocessed tree image set, annotate the parts of the tree trunk blocked by leaves as leaves, assign labels to each pixel in the tree image and generate corresponding segmentation masks, and convert the annotated tree image into a binary mask map;

[0023] The preprocessed tree image set and the binary mask map are input into the Mask R-CNN model for training, and the binary mask map of the tree image is output. According to the output binary mask map and the annotated binary mask map, the loss of each pixel is calculated through the loss function. The calculated total loss value is used to update the model parameters through the back propagation algorithm.

[0024] Furthermore, the loss function of the Mask R-CNN model is:

[0025]

[0026] Where L is the loss function of the Mask R-CNN model, W is the loss weight matrix of the Mask R-CNN model, which represents the loss weight between each predicted semantic label and the true semantic label, p is the p-th pixel of the tree image, y(p) is the true semantic label of the p-th pixel, is the predicted semantic label of the p-th pixel output by the Mask R-CNN model, is the probability that the p-th pixel belongs to the predicted semantic label of the Mask R-CNN model, N is the total number of pixels in the tree image, the background label is 0, the trunk label is 1, and the leaf label is 2.

[0027] Furthermore, the loss weight matrix W of the Mask R-CNN model is:

[0028]

[0029] Among them, the rows represent the true semantic labels, the columns represent the predicted semantic labels, and the values ​​represent the loss weights between the semantic labels.

[0030] Furthermore, the method of using the nearest neighbor matching algorithm to match feature points between multiple tree images according to the local feature points of each tree image includes the following steps:

[0031] Extracting feature points and descriptors from each image, wherein the feature points are identified by the scale-invariant feature transform (SIFT) algorithm, and the descriptors are 128-dimensional vectors for each feature point;

[0032] The descriptors in every two tree images are matched, the Euclidean distance between each pair of descriptors is calculated, and each feature point is matched according to the nearest Euclidean distance to obtain matching point pairs.

[0033] Furthermore, the method of constructing a preliminary three-dimensional point cloud model based on the matched feature points by using COLMAP through an incremental reconstruction method includes the following steps:

[0034] Step A1: Calculate the relative pose between images based on the matching feature point pairs;

[0035] Step A2: Based on the relative position and matching feature point pairs, a preliminary three-dimensional point cloud is calculated by triangulation;

[0036] Step A3: adding at least one image to the reconstruction process, and calculating the relative pose of the added image using matching feature point pairs between the added image and the existing 3D point cloud;

[0037] Step A4: According to the pose and matching feature points of the newly added image, the 3D point cloud is updated by triangulation, and bundle adjustment is performed to optimize the 3D point cloud and camera pose;

[0038] Step A5: Repeat steps A3 and A4 to gradually update and optimize the 3D point cloud and generate a preliminary 3D point cloud model.

[0039] Further, the step S7 includes the following steps:

[0040] According to the two-dimensional coordinates (u, v) of each pixel in the semantic segmentation map and the internal and external parameters of the camera, the three-dimensional coordinates of the pixel in the preliminary three-dimensional point cloud model are calculated by back-projection through the projection matrix.

[0041]

[0042] Among them, (X, Y, Z) is the three-dimensional coordinate of the pixel point in the preliminary three-dimensional point cloud model, Z (u, v) is the depth information of the pixel point, and K is the intrinsic parameter matrix of the camera. The intrinsic parameter matrix K of the camera is:

[0043]

[0044] Among them, f x 、f y is the focal length of the camera, the scaling factor along the x and y axes, c x 、c y is the principal point coordinate of the camera;

[0045] Find the point cloud data point closest to the three-dimensional coordinates of the pixel point in the preliminary three-dimensional point cloud model;

[0046] Transfer the semantic label of a pixel to the closest point cloud data point.

[0047] Furthermore, the traversing of each point cloud data in the preliminary three-dimensional point cloud model, segmenting the preliminary three-dimensional point cloud model according to the semantic labels of the point cloud data, and obtaining the three-dimensional point cloud model of the tree includes the following steps:

[0048] Traversing the point cloud data without semantic labels in the preliminary three-dimensional point cloud model, calculating the point cloud data closest to it, and setting the semantic label of the point cloud data closest to it as the semantic label of the currently traversed point cloud data without semantic labels;

[0049] Each point cloud data in the preliminary 3D point cloud model is traversed, and the point cloud data with the semantic label as background is deleted to obtain the 3D point cloud model of the tree.

[0050] Furthermore, the use of the SOR filtering algorithm to perform noise reduction optimization on the tree three-dimensional point cloud model includes the following steps:

[0051] For each point cloud data p in the tree 3D point cloud model i , define its neighborhood N(p i ), the neighborhood N(p i ) represents the point cloud data p i Other point cloud data sets within a certain radius;

[0052] Calculate each point cloud data p i All points p in the neighborhood of j With point cloud data i The Euclidean distance d(p i ,p j ), the formula is:

[0053]

[0054] Among them, (X i ,Y i ,Z i ) and (X j ,Y j ,Z j ) are point cloud data p i With point cloud data j 3D coordinates in the 3D point cloud model of the tree;

[0055] Calculate each point p i The mean distance μ(p i ) and standard deviation σ(p i ), the formula is:

[0056]

[0057] Among them, |N(p i )| is the point cloud data p i The number of points in the neighborhood;

[0058] According to each point cloud data p i The mean distance μ(p i ) and standard deviation σ(p i ), determine the point cloud data p i Is it an outlier? If the following conditions are met, the point cloud data p is considered i For outliers:

[0059] d(p i ,p j )>μ(p i )+θ·σ(p i )

[0060] Among them, θ is the preset threshold factor;

[0061] The point cloud data judged as outliers are removed from the three-dimensional point cloud model of the tree to obtain a denoised and optimized three-dimensional point cloud model of the tree.

[0062] Compared with the prior art, the present invention has the following advantages:

[0063] (1) The present invention collects multi-angle images of trees by flying a drone in the target area, and uses the pre-trained Mask R-CNN model to perform semantic segmentation of the trees, so that each pixel is accurately assigned a semantic label (such as trunk, leaves, and background). This method can effectively handle complex scenes, including situations where leaves cover the trunk, and improves the accuracy of tree segmentation.

[0064] (2) The Mask R-CNN model loss function of the present invention effectively solves the problem of class imbalance in the image through the weighted cross entropy loss function. The loss weight matrix W in the loss function is set by experience (such as 0 for the background class, 5 for the trunk class, and 10 for the leaf class). By assigning different loss weights to different categories (such as trunks, leaves, and background), especially assigning larger weights to the trunk and leaf categories, the model's attention to these fewer categories can be effectively enhanced, avoiding the impact of class imbalance on model performance and improving the segmentation accuracy of small categories. In addition, the weight matrix in the loss function is set by experience, which enables the model to adaptively adjust the contribution of each category, optimize the gradient propagation during the training process, and enhance the model's learning ability in difficult areas (such as the overlapping area or occluded area between leaves and trunks), thereby enhancing the model's robustness to complex scenes and improving the final semantic segmentation accuracy.

[0065] (3) The present invention uses the SIFT algorithm to extract feature points, and removes mismatched points through the nearest neighbor matching algorithm and the RANSAC algorithm, thereby ensuring the accuracy of point cloud data during 3D reconstruction. Combined with the COLMAP incremental reconstruction method, a preliminary 3D point cloud model is gradually generated, and the camera pose is optimized, thereby improving the accuracy and stability of the modeling process.

[0066] (4) The present invention accurately transfers semantic labels to point cloud data by connecting the three-dimensional coordinates of each pixel in the semantic segmentation map with the preliminary three-dimensional point cloud model. Subsequently, the point cloud is segmented according to the semantic labels using a traversal method, and finally a three-dimensional point cloud model of the tree is obtained. This process ensures that the semantic information of each point cloud data in the final model is complete, and enhances the usability and accuracy of the model.

[0067] (5) The present invention uses the Statistical Outlier Removal (SOR) filtering algorithm to perform noise reduction on the three-dimensional point cloud, which can effectively remove noise points and optimize the quality of the point cloud data. By calculating the Euclidean distance of points in the neighborhood of each point cloud data, outliers are determined and removed, which significantly improves the density and quality of the point cloud data and ensures that the generated three-dimensional point cloud model is smoother and more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flow chart of the method of the present invention;

[0069] Figure 2 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0071] Embodiment 1:

[0072] This embodiment provides a tree modeling method based on drone-collected images and image recognition. Figure 1 As shown, the following steps are included:

[0073] Step S1: planning the flight mode and flight path of the UAV according to the size, shape and distribution of trees in the target area;

[0074] Step S2: Using the planned flight mode and flight path to fly the drone in the target area to collect multi-angle image data of the trees, where the multi-angle image data includes a set of tree images taken from multiple different viewing angles and heights;

[0075] Step S3: preprocessing the multi-angle image data, including denoising, cropping, image enhancement, and standardization;

[0076] Step S4: input the preprocessed multi-angle image data into a pre-trained neural network model for tree recognition and semantic segmentation, and output a semantic segmentation map of each tree image, in which each pixel in the semantic segmentation map is assigned a semantic label;

[0077] Step S5: extract feature points and their descriptors from each tree image using the SIFT algorithm, match feature points between multiple tree images based on the feature points and their descriptors using the nearest neighbor matching algorithm, and remove mismatched feature points using the RANSAC algorithm;

[0078] Step S6: Based on the matched feature points, a preliminary three-dimensional point cloud model is constructed by using COLMAP through an incremental reconstruction method;

[0079] Step S7: Calculate the three-dimensional coordinates of each pixel in the semantic segmentation map obtained in step S4 in the preliminary three-dimensional point cloud model, determine the point cloud data corresponding to the pixel according to the three-dimensional coordinates of each pixel, and transfer the semantic label of each pixel to the corresponding point cloud data;

[0080] Step S8: traverse each point cloud data in the preliminary three-dimensional point cloud model, segment the preliminary three-dimensional point cloud model according to the semantic labels of the point cloud data, and obtain a three-dimensional point cloud model of the tree;

[0081] Step S9: Use the SOR filtering algorithm to perform noise reduction and optimization on the tree 3D point cloud model, use the Poisson reconstruction method to obtain the tree 3D mesh model of the tree after noise reduction and optimization, and construct the tree geometric model based on the tree 3D mesh model.

[0082] Furthermore, semantic labels include trunk, leaves and background.

[0083] Furthermore, the neural network model is a Mask R-CNN model, and the training process of the Mask R-CNN model includes the following steps:

[0084] Obtain a collection of tree images taken from multiple different perspectives and heights, and preprocess the tree image collection, including denoising, cropping, image enhancement, and standardization;

[0085] Use the image annotation tool LabelMe to manually annotate each area in the preprocessed tree image set, annotate the parts of the tree trunk blocked by leaves as leaves, assign labels to each pixel in the tree image and generate corresponding segmentation masks, and convert the annotated tree image into a binary mask map;

[0086] The preprocessed tree image set and the binary mask map are input into the Mask R-CNN model for training, and the binary mask map of the tree image is output. According to the output binary mask map and the annotated binary mask map, the loss of each pixel is calculated through the loss function. The calculated total loss value is used to update the model parameters through the back propagation algorithm.

[0087] Furthermore, the loss function of the Mask R-CNN model is:

[0088]

[0089] Where L is the loss function of the Mask R-CNN model, W is the loss weight matrix of the Mask R-CNN model, which represents the loss weight between each predicted semantic label and the true semantic label, p is the p-th pixel of the tree image, y(p) is the true semantic label of the p-th pixel, is the predicted semantic label of the p-th pixel output by the Mask R-CNN model, is the probability that the p-th pixel belongs to the predicted semantic label of the Mask R-CNN model, N is the total number of pixels in the tree image, the background label is 0, the trunk label is 1, and the leaf label is 2.

[0090] Furthermore, weights are defined for errors between different categories:

[0091] The predicted semantic label is background - the true semantic label is trunk: maximum loss, set to a larger weight (e.g., 10);

[0092] The predicted semantic label is background - the true semantic label is leaf: small loss, set to medium weight (e.g., 5);

[0093] The predicted semantic label is the trunk - the true semantic label is the background: the maximum loss is set to a larger weight (e.g., 10);

[0094] The predicted semantic label is the trunk - the true semantic label is the leaf: small loss, set to a medium weight (e.g., 5);

[0095] Predicted semantic label is leaf - true semantic label is background: small loss, set to medium weight (e.g., 5);

[0096] The predicted semantic label is a leaf and the true semantic label is a trunk: small loss, set to a medium weight (e.g., 5);

[0097] A weight matrix W is defined for the prediction error between each category:

[0098]

[0099] Among them, the rows represent the true semantic labels, the columns represent the predicted semantic labels, and the values ​​represent the loss weights between the semantic labels.

[0100] Furthermore, the nearest neighbor matching algorithm is used to match feature points between multiple tree images according to the local feature points of each tree image, including the following steps:

[0101] Extract feature points and descriptors from each image. Feature points are identified by the scale-invariant feature transform (SIFT) algorithm, and descriptors are 128-dimensional vectors for each feature point.

[0102] The descriptors in every two tree images are matched, the Euclidean distance between each pair of descriptors is calculated, and each feature point is matched according to the nearest Euclidean distance to obtain matching point pairs.

[0103] Furthermore, based on the matched feature points, a preliminary 3D point cloud model is constructed by using COLMAP through an incremental reconstruction method, including the following steps:

[0104] Step A1: Calculate the relative pose between images based on the matching feature point pairs;

[0105] Step A2: Calculate the preliminary 3D point cloud by triangulation based on the relative pose and matching feature point pairs;

[0106] Step A3: adding at least one image to the reconstruction process, and calculating the relative pose of the added image using matching feature point pairs between the added image and the existing 3D point cloud;

[0107] Step A4: According to the pose and matching feature points of the newly added image, the 3D point cloud is updated by triangulation, and bundle adjustment is performed to optimize the 3D point cloud and camera pose;

[0108] Step A5: Repeat steps A3 and A4 to gradually update and optimize the 3D point cloud and generate a preliminary 3D point cloud model.

[0109] Further, step S7 includes the following steps:

[0110] According to the two-dimensional coordinates (u, v) of each pixel in the semantic segmentation map and the internal and external parameters of the camera, the three-dimensional coordinates of the pixel in the preliminary three-dimensional point cloud model are calculated by back-projection through the projection matrix.

[0111]

[0112] Among them, (X, Y, Z) is the three-dimensional coordinate of the pixel in the preliminary three-dimensional point cloud model, Z (u, v) is the depth information of the pixel, K is the intrinsic parameter matrix of the camera, and the intrinsic parameter matrix K of the camera is:

[0113]

[0114] Among them, f x 、f y is the focal length of the camera, the scaling factor along the x and y axes, c x 、c y is the principal point coordinate of the camera;

[0115] Find the point cloud data point closest to the three-dimensional coordinates of the pixel point in the preliminary three-dimensional point cloud model;

[0116] Transfer the semantic label of a pixel to the closest point cloud data point.

[0117] Further, traversing each point cloud data in the preliminary three-dimensional point cloud model, segmenting the preliminary three-dimensional point cloud model according to the semantic labels of the point cloud data, and obtaining a three-dimensional point cloud model of the tree, including the following steps:

[0118] Traversing the point cloud data without semantic labels in the preliminary three-dimensional point cloud model, calculating the point cloud data closest to it, and setting the semantic label of the point cloud data closest to it as the semantic label of the currently traversed point cloud data without semantic labels;

[0119] Each point cloud data in the preliminary 3D point cloud model is traversed, and the point cloud data with the semantic label as background is deleted to obtain the 3D point cloud model of the tree.

[0120] Furthermore, the SOR filtering algorithm is used to perform noise reduction optimization on the tree 3D point cloud model, including the following steps:

[0121] For each point cloud data p in the tree 3D point cloud model i , define its neighborhood N(p i ), neighborhood N(p i ) represents the point cloud data p i Other point cloud data sets within a certain radius;

[0122] Calculate each point cloud data p i All points p in the neighborhood of j With point cloud data i The Euclidean distance d(p i ,p j ), the formula is:

[0123]

[0124] Among them, (X i ,Y i ,Z i ) and (X j ,Y j ,Z j ) are point cloud data p i With point cloud data j 3D coordinates in the 3D point cloud model of the tree;

[0125] Calculate each point p i The mean distance μ(p i ) and standard deviation σ(p i ), the formula is:

[0126]

[0127] Among them, |B(p i)| is the point cloud data p i The number of points in the neighborhood;

[0128] According to each point cloud data p i The mean distance μ(p i ) and standard deviation σ(p i ), determine the point cloud data p i Is it an outlier? If the following conditions are met, the point cloud data p is considered i For outliers:

[0129] d(p i ,p j )>μ(p i )+θ·σ(p i )

[0130] Among them, θ is the preset threshold factor;

[0131] The point cloud data judged as outliers are removed from the three-dimensional point cloud model of the tree to obtain a denoised and optimized three-dimensional point cloud model of the tree.

[0132] Embodiment 2:

[0133] The parts not mentioned in this embodiment are the same as those in Embodiment 1.

[0134] This embodiment provides a tree point cloud modeling method based on drone image acquisition and recognition. Figure 2 As shown, the following steps are included:

[0135] 1. UAV image collection

[0136] In the process of tree point cloud modeling, drones are needed to collect high-quality images to ensure that the images cover the entire target area and have sufficient details to support subsequent three-dimensional reconstruction.

[0137] First, flight planning should be carried out. The flight path of the drone should be designed according to the size, shape and distribution of trees in the target area. Common flight modes include grid mode, straight line mode and circular mode. For example, in a larger area, the grid mode can ensure that the trees are photographed from multiple angles to avoid missing shots. In a narrower area, the straight line mode is more efficient. In order to ensure sufficient overlap between images, the design of the flight path should try to ensure that the overlap rate of the camera shooting area is between 60% and 80%. Too low overlap may lead to insufficient matching of feature points and affect the reconstruction effect; too high overlap may increase the computational burden, so reasonable adjustments should be made according to the specific situation.

[0138] Then set up the camera. The camera on the drone should have sufficient resolution and wide-angle field of view. It is usually recommended to use a camera with 12MP or above for shooting. To ensure image details, a higher resolution should be selected when shooting, and the camera exposure parameters should be set appropriately. The automatic exposure mode is suitable for scenes with more dynamic lighting conditions, but if the light changes greatly, the manual exposure setting will be more stable. In addition, maintaining the consistency of the white balance of the image and avoiding color cast can also improve the accuracy of subsequent feature extraction.

[0139] During the shooting process, the overlap and viewing angle of the images need to be carefully controlled. It is recommended that the front-to-back overlap of each image be set between 70%-80%, and the side overlap be 50%-60%. This can effectively ensure the matching of feature points between images, thereby improving the accuracy of 3D reconstruction. In addition, the diversification of shooting angles is also very important. Avoid shooting only from a vertical perspective. Properly tilting the camera angle can ensure that key parts such as the trunk and crown can be better captured.

[0140] During the flight, the image quality should also be checked in real time. If conditions permit, the captured images can be checked regularly to ensure that they are not blurred, unevenly exposed, etc. The shooting frequency is usually set to 1-2 images per second, which can ensure image coverage and avoid overloading storage space.

[0141] 2. Semantic Extraction Using Neural Networks

[0142] In the process of tree point cloud modeling, semantic extraction is a key step to classify and segment the image content, with the goal of identifying trees and their related parts (such as trunks, leaves, etc.) from the original image. Neural networks, especially models for image segmentation, such as Mask R-CNN, play an important role in this task. By training these models, efficient semantic segmentation can be achieved, different parts of the tree can be extracted, and accurate regional information can be provided for subsequent point cloud reconstruction and model processing.

[0143] The following is the detailed operation process of this step:

[0144] 1. Data preprocessing

[0145] The effectiveness of neural network training depends on high-quality training data, so image preprocessing is a crucial step. The main preprocessing steps include denoising, cropping, and normalization.

[0146] ① Denoising: The original image may contain noise, which will affect the training effect of the neural network. Common denoising methods include median filtering and Gaussian filtering, which can reduce the impact of noise by smoothing the image.

[0147] ② Cropping: According to the size and position of the tree, crop the area of ​​the image that contains all parts of the tree. Cropping the image can reduce the amount of calculation during model training and improve training efficiency.

[0148] ③ Image enhancement: When training data is insufficient, data enhancement technology is very useful. The training data set can be expanded by rotating, flipping, cropping, adjusting colors, etc., making the model more robust and better able to adapt to images in different environments.

[0149] ④ Standardization: Standardizing the image to ensure that the pixel values ​​of the image are distributed within a certain range (such as normalizing the RGB values ​​to [0, 1] or [-1, 1]), helps accelerate the convergence of the network.

[0150] 2. Labeling Data

[0151] Before training, you first need to prepare labeled data. The labeling process involves collecting an image dataset containing objects such as tree trunks and leaves, and manually labeling each area in the image using the image labeling tool LabelMe. The parts of the tree trunk that are blocked by leaves are still labeled as leaves. Assign a label to each object and generate a corresponding segmentation mask. The labeled information is then converted into the format required by the neural network, usually a binary mask map (a label for each pixel) or a polygonal area (representing the outline of the object).

[0152] 3. Training the Neural Network

[0153] The goal of training a neural network is to learn how to classify pixels in an image into different semantic regions, such as trees and background, using existing labeled data. Commonly used neural network models include Mask R-CNN.

[0154] Mask R-CNN is an instance segmentation model based on Faster R-CNN. It can perform object detection and semantic segmentation at the same time. It is particularly suitable for problems that require detection and segmentation of different object instances (such as tree trunks and leaves). Unlike traditional image segmentation, Mask R-CNN generates accurate segmentation masks for each object while detecting the object. Mask R-CNN needs to annotate the tree trunks, leaves, etc. in each image and generate the corresponding segmentation masks. During the training process, the model will optimize the object detection and segmentation tasks at the same time, thereby outputting more refined segmentation results.

[0155] 4. Result Output

[0156] The trained model will perform inference based on the input image and output the category label of each pixel, that is, the semantic segmentation map of the image. In the application of tree point cloud modeling, each pixel in the semantic segmentation map will be classified into the following three categories:

[0157] Trunk: The trunk includes the main trunk of the tree and the primary branches. The primary branches of a tree are large branches that grow from the trunk and support more secondary branches.

[0158] Leaves: The leafy part of a tree. Leaves are usually found at the top of a tree or at the end of branches, forming the tree's crown.

[0159] Background: includes open areas, other objects, etc., usually marked as background categories.

[0160] The output segmentation map can help clarify the structure of each area and provide necessary semantic information for the subsequent segmentation processing after point cloud generation.

[0161] 3. Generate point cloud model based on image:

[0162] Using COLMAP for 3D reconstruction based on drone-collected images is one of the key steps in point cloud modeling. COLMAP is a powerful open source image reconstruction software that is widely used to generate sparse and dense 3D point cloud models from images. COLMAP combines information from multiple images through methods such as feature matching, relative pose estimation, and incremental reconstruction to generate high-quality 3D reconstruction results.

[0163] 1. Feature matching

[0164] The first step of 3D reconstruction is to extract feature points from the image that can be used for matching. The SIFT (Scale Invariant Feature Transform) algorithm is widely used in this step because it can stably identify key points in the image at different scales and rotations. The specific process is as follows:

[0165] ① Feature extraction: First, use the SIFT algorithm to extract local feature points from each image. These feature points have unique descriptors in different images and can be used to match between different images.

[0166] ② Feature matching: In COLMAP, the nearest neighbor matching algorithm (such as brute force matching or KD tree-based fast matching method) is usually used to find the same or similar feature points between images. The brute force matching method calculates the distance between each pair of feature points and selects the smallest distance as the matching result; while the KD tree matching method quickly finds the nearest neighbor feature points by constructing a spatial index of the feature points.

[0167] ③ Matching optimization: In order to improve the matching accuracy, COLMAP can use algorithms such as RANSAC (Random Sample Consensus) to eliminate mismatched points and ensure that the matching of feature points is more accurate.

[0168] 2. Relative pose calculation

[0169] By matching feature points, COLMAP can estimate the relative pose between cameras, that is, calculate the rotation matrix and displacement vector of the camera. This process is usually achieved through the following two main methods:

[0170] Five-point algorithm: The five-point algorithm is a commonly used algorithm for estimating the relative pose between cameras. It calculates the relative rotation and translation between cameras based on five pairs of matching points. By solving this five-point problem, the relative pose of the cameras can be obtained. Since there are multiple solutions in the calculation process, RANSAC is usually used to eliminate incorrect matching points to ensure the accuracy of the estimated pose.

[0171] Nonlinear optimization: After the initial estimation of the pose is completed, COLMAP will further refine the relative pose between cameras through nonlinear optimization (such as minimizing the reprojection error). The nonlinear optimization algorithm iteratively minimizes the reprojection error of each feature point so that the projected position of all matching points is as close as possible to the actual spatial position.

[0172] The accuracy of relative pose calculation directly affects the accuracy of subsequent 3D reconstruction. Therefore, the optimization step is crucial to improving the reconstruction effect.

[0173] 3. Incremental Reconstruction

[0174] Incremental reconstruction is one of the core technologies of COLMAP to generate point cloud models. This method usually starts with a preliminary relative pose estimation, and then gradually adds more images and continuously optimizes the model.

[0175] ① Selection of initial image pair: First, COLMAP selects a pair of images with more matching feature points from the image set as the initial image pair for 3D reconstruction. The relative pose of this pair of images is usually estimated through the above five-point algorithm and optimization steps.

[0176] ② Image incremental addition: After obtaining the preliminary 3D model, COLMAP will gradually add new images and try to match the new images with the feature points in the existing images. As more images are added, the 3D reconstruction gradually expands and optimizes the relative pose each time a new image is added.

[0177] ③Optimization and adjustment: COLMAP will continuously optimize the camera pose and point position during each new image addition. This process includes refining the camera parameters and 3D point cloud data through bundle adjustment. Bundle adjustment can effectively reduce the deviation caused by measurement errors, thereby improving the overall quality of the point cloud.

[0178] Incremental reconstruction gradually expands the 3D model and ensures that the results of each step are optimized, resulting in a high-quality point cloud.

[0179] 4. Point cloud generation

[0180] Through the above steps, COLMAP will output a preliminary sparse point cloud, which is the basic data for 3D reconstruction. The point cloud is a set of coordinates of matching feature points in the image in 3D space. The initially generated point cloud may have some noise and inaccuracy, but it provides a basis for further 3D reconstruction and optimization.

[0181] ① Sparse point cloud: In the early stage, the generated point cloud is sparse point cloud, that is, it contains a small number of high-quality feature points. Most of these points are unique feature points identified from the image, which form a preliminary model framework in three-dimensional space.

[0182] ②Point cloud quality optimization: As more images are added and optimized, the point cloud will gradually increase, and the connection between points will become more refined, eventually forming a more complete 3D point cloud structure. Through continuous incremental reconstruction and optimization, COLMAP will gradually improve the accuracy and details of the generated point cloud.

[0183] ③ Transition from sparse to dense: The initial sparse point cloud can be used as a basis for further refinement through the dense reconstruction algorithm in COLMAP. Dense reconstruction will perform a more detailed 3D reconstruction of each pixel point, generating a denser and more accurate point cloud.

[0184] 4. Processing of point cloud model:

[0185] After generating a point cloud model, further processing steps are usually required to ensure the quality of the point cloud data and remove excess noise. Especially in tree modeling, it is crucial to extract leaves and trunk parts from the original point cloud and remove environmental debris and noise. The following are two key steps for point cloud data processing: extracting tree parts from point clouds and using the SOR filter algorithm for noise reduction.

[0186] 1. Extract tree parts from point cloud

[0187] (1) Image segmentation results and point cloud data registration

[0188] Point cloud data is generated by COLMAP, and these point cloud data are associated with image data through the camera's perspective relationship, so it is necessary to align the image segmentation results with the point cloud data. The projection matrix of each image can be calculated using the camera's intrinsic and extrinsic parameters. The projection matrix projects points in three-dimensional space onto the two-dimensional image plane. The projection matrix is ​​usually: P = K[R|t], where K is the camera's intrinsic matrix, and [R|t] is the rotation matrix and displacement vector provided by the camera's extrinsic parameters.

[0189] The position of the pixel in three-dimensional space is calculated by back-projection. Combining the two-dimensional coordinates of the pixel (u, v) with the camera's intrinsic parameter matrix K, the corresponding ray direction can be obtained. This represents the direction of the ray in 3D space where the image pixel (u,v) lies. By combining it with the depth information in the point cloud (generated from COLMAP), if there is a point in the point cloud projected at the same position as the pixel, the 3D coordinates corresponding to the pixel can be determined.

[0190] (2) Mapping image segmentation labels to point clouds

[0191] In image segmentation, each pixel is assigned a label (such as "leaf", "trunk", "background"). These labels are back-projected into the point cloud through the projection matrix obtained earlier, ensuring that the corresponding points in the point cloud of pixels belonging to the same category in the image are also assigned corresponding labels. For each image pixel, its 3D coordinates are calculated through the projection matrix, its corresponding point in the 3D point cloud is found, and then the label of the pixel is passed to the point. In the end, each point in the point cloud will contain a label indicating that it belongs to a category such as trunk, leaf, or background.

[0192] (3) Semantic segmentation of point cloud data

[0193] Iterate through each point in the point cloud data, check the label of each point, if the label of the point is "tree", add the point to the tree area. If the label of the point is "leaf", add the point to the leaf area.

[0194] 2. Use SOR filtering algorithm to reduce noise and optimize point cloud

[0195] In the process of generating point cloud data, noise is often generated due to various factors (such as shooting angle, object occlusion, etc.). In order to improve the quality of point cloud, it is necessary to use SOR (Statistical Outlier Removal) filtering algorithm to filter outliers and remove unreliable data points.

[0196] The SOR filtering algorithm is a point cloud denoising method based on statistics. It determines whether a point is an outlier by calculating the distribution of points in its neighborhood. The specific steps are as follows:

[0197] ① Calculate the neighborhood of each point: For each point in the point cloud, first define its neighborhood, that is, other points within a certain radius around the point.

[0198] ②Calculate statistics: For each point, calculate the distances between all points in its neighborhood. These distances are usually measured using Euclidean distances. Then, calculate the mean and standard deviation of these distances.

[0199]

[0200] Among them, d(p i ,p j ) is point p i and point p j The Euclidean distance between is the average distance of the neighborhood points, is the standard deviation of the neighborhood distances.

[0201] ③ Outlier judgment: Then, based on the statistical information of the distance between each point and its neighboring points, determine whether the point is an outlier. Usually, by setting a distance threshold θ (such as the distance mean plus two standard deviations), if the distance between the point and the neighboring points is far from the mean, the point is considered an outlier.

[0202]

[0203] If the distance of a point exceeds a preset threshold, the point is considered an outlier.

[0204] ④ Remove outliers: For points judged as outliers, they are removed from the point cloud data, and the remaining points are valid and real point cloud data.

[0205] ⑤Update point cloud: By removing outliers, the denoised point cloud data is obtained.

[0206] 5. Generate mesh model from point cloud model

[0207] The purpose of generating a mesh model is to convert sparse point cloud data into a continuous 3D surface model so that the model can be used in 3D visualization and subsequent analysis. The Poisson reconstruction algorithm is a commonly used method in this process, which generates a smooth, watertight and continuous mesh by calculating the divergence of the normal vector field.

[0208] 1. Poisson reconstruction method

[0209] The Poisson reconstruction algorithm is based on the mathematical theory of the Poisson equation, and its goal is to derive a smooth three-dimensional surface from known point cloud data (including the position and normal vector of the point). The basic steps are as follows:

[0210] ① Normal vector estimation

[0211] If the normal vector is not directly provided in the point cloud data, the normal vector of each point needs to be estimated first. The general method is to use the K nearest neighbor algorithm (KNN) to find the neighborhood point set for each point, and then use these neighborhood points to fit the normal vector through principal component analysis (PCA).

[0212] K-nearest neighbor method (KNN): find k neighboring points around each point and use these points for local fitting.

[0213] PCA method: Perform PCA analysis on neighborhood points to obtain principal component vectors, where the principal direction is usually the normal vector.

[0214] ②Poisson Equation

[0215] The core of the Poisson reconstruction algorithm is to transform the normal vector field into a three-dimensional surface through the Poisson equation. The discrete form of the Poisson equation is:

[0216]

[0217] Where N is the normal vector field and ρ(x) is the density distribution of the point cloud. The goal of the algorithm is to calculate a smooth surface function φ(x) from the divergence field ρ(x), that is:

[0218] Δφ=ρ(x)

[0219] Where Δ is the Laplace operator, φ(x) represents the surface to be calculated, and finally the equation is solved by numerical method to obtain the three-dimensional surface of the point cloud.

[0220] ③Numerical solution and mesh generation

[0221] Poisson's equation is usually solved using numerical methods such as finite difference method or conjugate gradient method. Through these methods, we can estimate the surface position from the point cloud data. Then, the estimated surface is converted into a triangulated mesh using triangulation algorithm (such as Delaunay triangulation), which can be used for subsequent 3D rendering and analysis.

[0222] ④Grid optimization

[0223] The preliminary mesh obtained by Poisson reconstruction may contain some small isolated parts or the local details are not smooth enough, so it needs to be optimized. Common optimization methods include:

[0224] Remove small isolated polygons: Polygons with too small an area can be removed through connectivity analysis.

[0225] Smoothing: The mesh is further smoothed using a mesh smoothing algorithm (such as Laplacian smoothing).

[0226] 2. Octree Depth Control

[0227] In the Poisson reconstruction algorithm, the octree structure is used to manage the spatial distribution of point cloud data. The key role of the octree is to effectively organize point cloud data by recursively dividing the space into multiple subspaces, thereby improving the reconstruction efficiency. The octree is a hierarchical tree data structure that is mainly used to represent regions in three-dimensional space. Each octree node represents a cubic region, and the node is recursively divided into 8 child nodes. Each child node contains point data in the spatial region. By adjusting the depth of the octree, the fineness of the grid can be controlled. When the depth is large, the space is divided more finely, and each area of ​​the point cloud has a higher resolution, thereby generating a finer grid; when the depth is small, on the contrary, the generated grid is coarser and the amount of calculation is reduced.

[0228] Choosing the depth of the octree

[0229] ① Larger depth: For dense and complex point clouds, increasing the depth of the octree can ensure richer mesh details and adapt to complex geometric shapes. However, too large a depth will result in greater computational complexity and memory consumption.

[0230] ② Smaller depth: For sparse or simpler point cloud data, a smaller depth is enough to generate a rough but sufficiently accurate mesh. A smaller depth reduces the consumption of computing resources and is suitable for simpler scenes.

[0231] Reasonable depth selection needs to consider the density and quality of the point cloud as well as the accuracy requirements of the final application.

[0232] 6. Remove small independent pieces from the model

[0233] In the point cloud model, there may be isolated small fragments or noise, which need to be removed through connectivity analysis.

[0234] 1. Use PyMeshLab filters to remove isolated small fragments

[0235] In a point cloud model, there may be some isolated small fragments or noise, which usually do not contribute to the final 3D modeling and may even interfere with subsequent processing. PyMeshLab provides a filter that allows you to remove these small fragments by calculating the diameter of the connected components.

[0236] How to: Use PyMeshLab

[0237] The ms.meshing_remove_connected_component_by_diameter() function can determine whether to remove a small fragment based on the diameter of the component. The algorithm first identifies the connected components in the point cloud and calculates the maximum diameter of each component. If the diameter of a component is less than the set threshold (such as 0.5 meters), the component is considered to be an isolated small fragment or noise and is removed. This ensures that the parts that are closely related to the target structure are retained in the end.

[0238] 2. Use graph algorithms (depth-first search / breadth-first search) to identify connected components

[0239] In addition to directly using the functions provided by PyMeshLab, you can also use graph algorithms to analyze the connectivity in point cloud data. Considering the point cloud as a node in a graph, when the distance between each two points is less than a certain set threshold, it can be considered that there is an edge between them. Through the depth-first search (DFS) or breadth-first search (BFS) algorithm, you can identify the connected components in the graph.

[0240] How to do it: First, build a distance matrix of the point cloud and calculate the distance between every two points. Next, use the depth-first search (DFS) or breadth-first search (BFS) algorithm to traverse these points and identify all connected point sets. Then, for each connected component, calculate its maximum diameter (that is, the distance between the farthest points in the component). If the diameter of a component is less than the set threshold, the component is considered to be an isolated small fragment and is removed from the point cloud.

[0241] 3. Further simplify the model based on distance and diameter constraints

[0242] To further improve processing efficiency and remove redundant data, diameter restrictions can be added when calculating connectivity and component diameters. In this way, the model can be effectively streamlined to only retain those connected parts that are large enough.

[0243] How to do it: First, determine which points belong to the same connected component by calculating the distance between points. Then, based on the diameter limit of the component, remove those connected parts whose diameter is less than the threshold. In this way, only the truly meaningful parts of the point cloud in the target area are retained, reducing the number of invalid points.

[0244] These three methods help remove noise and isolated small fragments from point cloud models. The filter provided by PyMeshLab is the most direct tool, which removes small components through diameter thresholds; the graph algorithm further enhances connectivity analysis and can more finely identify and remove isolated parts; finally, by combining diameter restrictions and distance calculations, the model is further optimized to remove redundant data.

[0245] 7. Check the water tightness of the model

[0246] Ensuring that the generated mesh model is watertight means that the model surface is continuous, without holes or cracks, and is suitable for subsequent physical simulation, analysis, and virtual modeling. Watertightness is an important criterion for generating valid 3D models, especially when performing engineering simulations such as finite element analysis (FEA). Watertightness ensures that the model has no erroneous surfaces, boundaries, or breaks.

[0247] Here are the detailed steps:

[0248] 1. Use the compute_volume function in PyMeshLab to check the water tightness of the model

[0249] PyMeshLab provides a function for checking the watertightness of a mesh: compute_volume(). The core idea of ​​this function is to determine whether there are holes or cracks in the model by calculating the volume of the model. If there are holes in the mesh, the volume calculation will be wrong, usually returning an illogical negative value or zero. On the contrary, if the returned volume is positive and reasonable, it means that the mesh is closed and watertight.

[0250] The volume is calculated by integrating the geometric shapes on the model surface. In three-dimensional space, a closed mesh model represents a closed space. When calculating the volume, if the mesh surface is continuous and has no holes, the volume calculated by integration should be positive and reasonable. If there are cracks or openings in the mesh, the volume calculation will be wrong due to the missing gaps.

[0251] Steps:

[0252] ①Call the compute_volume function: First, use the compute_volume() function in PyMeshLab to calculate the volume of the model. PyMeshLab will automatically check the boundary conditions of the model.

[0253] If the returned volume is valid and positive, the model is watertight.

[0254] If the return value is negative or zero, it indicates that the model may have cracks or holes.

[0255] ② Volume check: The returned volume value should be consistent with the expected value. Generally speaking, a larger model should have a larger volume, and there should be no negative or zero values. At this point, further check whether the model has discontinuous parts.

[0256] 2. Check model boundaries

[0257] If the volume value returned by compute_volume() does not meet expectations (for example, a negative value or zero), further inspection of the surface of the model is required, especially the boundary of the mesh. The following are the inspection and repair steps:

[0258] Steps:

[0259] ① Visualize the model: Use 3D visualization tools (such as MeshLab, ParaView, etc.) to load the mesh model and carefully check it, focusing on the boundary area of ​​the model. Common inspection methods include: zooming in to see where cracks or holes may exist. Observe whether the surface of the model is smooth and has no sharp edges or protrusions.

[0260] ② Check for cracks and holes: By comparing the external surface and internal distribution of the mesh, check whether there are invisible tiny cracks or holes. These cracks may be caused by noise or incomplete acquisition during the reconstruction process.

[0261] ③Manually repair holes: If cracks or holes are detected, you can use mesh repair tools (such as the "Fill Holes" tool in MeshLab or the patch function in Blender) to repair them. When manually repairing, you can reinforce the boundaries of the model to ensure that each facet is correctly spliced.

[0262] ④ Refine the mesh: If there are large gaps or discontinuities on the model boundary, use a refined mesh or smoothing algorithm to further optimize the model surface and ensure surface continuity.

[0263] ⑤ Recalculate the volume: After repairing the model, call the compute_volume() function again to ensure that the modified model volume is valid and positive. If the volume calculation is normal, it means that the model has been repaired successfully and the watertightness is guaranteed.

[0264] 3. Further repair and optimize the model

[0265] After volume inspection and manual repair, the water tightness of the model may have been initially guaranteed, but further optimization may be required to ensure high-quality model output. The following methods can be used for reference:

[0266] ① Use closing operations: Some mesh repair tools have operations that automatically close the mesh. These tools can fill gaps on the mesh surface and repair gaps in the model.

[0267] ② Mesh simplification and smoothing: Through mesh simplification and smoothing operations, excessive details are removed and the edges of the mesh are smoothed to eliminate potential discontinuities.

[0268] ③ Reconstruct normal vectors and smooth surfaces: For the repaired model, the normal vectors need to be recalculated to ensure that the direction of the surface is consistent. By correctly repairing the normal vectors, simulation problems caused by reverse normal lines can be avoided.

[0269] 8. Mesh to Topology Conversion

[0270] Mesh-to-topology conversion is the process of converting the generated 3D mesh model into a topological structure suitable for use by finite element analysis (FEA) software, especially for importing into engineering simulation tools such as ABAQUS for physical simulation. The key to this process is to ensure the transition from a meshed geometric model to a mathematically accurate and physically realistic topological model for subsequent analysis. Common conversion methods include using the OpenCASCADE kernel for geometry processing and exporting the model to the STEP file format for reading by software such as ABAQUS.

[0271] 1. OpenCASCADE kernel conversion

[0272] OpenCASCADE is an open source CAD (computer-aided design) kernel that is widely used in geometric modeling, topological analysis and conversion tasks. Its main function is to extract accurate geometric surfaces and solid models from mesh models and convert these models into mathematical representations that are more suitable for finite element analysis. OpenCASCADE can effectively handle non-manifold structures (such as discontinuous surfaces or irregular geometries) in mesh models and convert them into clear solid models.

[0273] Conversion steps:

[0274] ① Conversion from mesh to geometric model: Mesh models are usually composed of discrete triangular mesh patches, which represent the approximate surface of the object. However, this surface may not fully conform to the strict geometric definition. Using OpenCASCADE, the discrete patches in the mesh model are first converted into continuous geometric surfaces through the BRep (boundary representation) function library. For example, OpenCASCADE can fit the meshed triangular patches into precise NURBS (non-uniform rational B-spline) surfaces or other suitable geometric representations.

[0275] ② Processing of non-manifold structures: Non-manifold structures in mesh models refer to some complex boundary conditions, such as multiple faces sharing an edge or multiple faces sharing a point, which may lead to geometric discontinuities or physically unsolvable structures. OpenCASCADE provides a variety of algorithms for automatically identifying and repairing these non-manifold parts. Through topological repair, OpenCASCADE can convert these irregular mesh structures into more standard geometric and topological representations.

[0276] ③Entity and surface extraction: OpenCASCADE converts meshed surfaces and solid structures into real mathematical geometric objects through entity extraction (such as basic geometric bodies such as cubes, cylinders, spheres, etc.) and surface extraction (such as the plane of a polyhedron) functions. These objects are more suitable for engineering analysis.

[0277] ④Conversion effect: Through the processing of OpenCASCADE, the final generated geometric model will meet the strict mathematical and physical requirements for the geometric model, and can perform higher-precision finite element analysis without being restricted by the discrete patches in the mesh model.

[0278] 2. Generate STEP file

[0279] STEP (Standard for the Exchange of Product model data) is an international standard data exchange format for transferring geometric and topological information between different CAD and CAE software. The STEP file format is widely used in engineering applications because it can accurately describe the geometric shape and topological structure of objects and supports various geometric representations such as NURBS, BRep, etc.

[0280] By exporting the processed geometric model as a STEP file, the compatibility of the model between different software platforms can be ensured, especially the model can be imported into simulation tools such as ABAQUS.

[0281] Steps to generate STEP file:

[0282] ①Convert to standardized geometry format: In OpenCASCADE, the geometry and topology have been converted to a standard mathematical representation. At this point, you can use the STEPWriter class provided by OpenCASCADE to save the converted geometry and topology data in STEP file format.

[0283] ② Export topology: Use the STEPWriter class to specify the output file path and export a STEP file containing geometric information and topology. This file will contain all the geometric data of the object (such as surfaces, entities) and the topological relationships between them (such as the connection of patches, the combination of entities, etc.).

[0284] ③ Ensure file compatibility: During the export process, the structure of the STE file should comply with the standard STEP file specifications (such as ISO 10303-21) to ensure that the file can be correctly read in simulation software such as ABAQUS. The generated STEP file will contain all the necessary geometric information, such as surfaces, boundaries, and entities, and is in a format that can be recognized and processed by ABAQUS.

[0285] ④ Check the output STEP file: Load the generated STEP file through CAD software or CAE tools to ensure that the geometry and topology information are correct. Check whether the geometric objects in the file are correct and whether there is any extra erroneous data. Make sure that the model does not have any errors or extra geometry.

[0286] ⑤Advantages of STEP file format: STEP file format has good cross-platform compatibility and is widely used in various engineering and simulation software. Since the STEP format can contain complex geometric information and topological structures, the exported model can be directly used in finite element analysis software such as ABAQUS without further conversion.

[0287] 3. Import into ABAQUS for finite element analysis

[0288] Once a standard STEP file is generated and its geometry and topology meet the requirements, it can be imported into ABAQUS for subsequent finite element analysis. ABAQUS will automatically identify the geometry in the STEP file and perform meshing and material property assignment based on the topological relationship of the model.

[0289] During the import process, ABAQUS will: automatically convert the geometric information in the STEP file into a finite element mesh suitable for simulation. Allow the user to further refine the mesh, adjust material properties, define boundary conditions, etc. as needed.

[0290] 9. Import STEP model into ABAQUS

[0291] Importing STEP models into ABAQUS for further modeling and simulation analysis is a crucial step in the entire engineering simulation process. By importing the converted STEP file into ABAQUS, the geometric model and topology structure can be seamlessly used for simulation analysis. The following are detailed step-by-step instructions.

[0292] 1. Import STEP file into ABAQUS

[0293] Importing STEP files into ABAQUS is the first step to ensure that the geometric model and topology can be loaded and recognized correctly. ABAQUS supports direct import of STEP format files, which can parse the geometric information of the model and construct a topology suitable for further analysis.

[0294] Steps:

[0295] ① Start ABAQUS: Open ABAQUS / CAE and create a new model database (File>New ModelDatabase).

[0296] ② Import STEP file: In ABAQUS / CAE, select File>Import>Part. In the pop-up file selection window, select the previously saved STEP file (.stp or .step format). ABAQUS will automatically read the file and parse the geometric data in it, including the model's surfaces, entities and their topological relationships.

[0297] ③ Check the model: After importing, ABAQUS will display the geometry of the model, check whether the model is loaded correctly, and ensure that all geometric elements (such as faces, edges, points, etc.) have been correctly identified. If any errors or warnings appear during the import process, you may need to manually repair or check whether the STEP file meets the standard or contains complex non-manifold structures.

[0298] 2. Assign material properties and loading conditions to the tree model

[0299] After importing the geometry model into ABAQUS, the next step is to assign appropriate material properties and loading conditions to the model. Correct material properties and loading conditions are the prerequisite for ensuring accurate simulation results.

[0300] Steps:

[0301] ① Define material properties: In ABAQUS, select the Property module and create a new material.

[0302] Assign appropriate material properties to the tree based on the physical properties of the tree model (such as density, elastic modulus, Poisson's ratio, etc.). For the material model of the tree, you can choose a simple elastic material model or define a more complex nonlinear model as needed. Elastic modulus (E): Related to the stiffness of the tree material. Poisson's ratio (ν): Describes the lateral deformation characteristics of the tree when stretched or compressed. Density (ρ): Affects the mass calculation of the tree and is applicable to problems such as gravity loading.

[0303] ② Apply loading conditions: In ABAQUS, select the Load module and apply appropriate loading conditions to the model. Apply static or dynamic loads according to simulation requirements. For example: force loads (such as wind force, gravity, etc.) act on the top of the tree or a specific area. Pressure loads, such as soil pressure at the roots or certain parts of the simulated tree.

[0304] ③ Select appropriate boundary conditions, such as fixing the bottom of the tree, constraining certain degrees of freedom of the trunk, etc., to ensure that the model can correctly simulate the actual stress conditions.

[0305] ④ Define contacts and constraints: If there are contacts between multiple parts (such as the contact between trees and the ground), you need to define contact pairs in ABAQUS. Select the corresponding contact pairs in the Interaction module and define parameters such as friction coefficient and contact mechanics to ensure that the interaction between trees and other parts is correctly simulated.

[0306] 3. Meshing and analysis

[0307] Meshing is the process of discretizing geometric models, with the goal of transforming continuous physical problems into finite element models. Appropriate meshing can ensure analysis accuracy and affect calculation efficiency.

[0308] Steps:

[0309] ① Create a mesh: In the Mesh module, select the model components that need to be meshed (such as the various parts of a tree).

[0310] ② Select the mesh type: Usually for tree models, the meshing is selected as tetrahedral mesh (Tet) or hexahedral mesh (Hex), etc., depending on the complexity and accuracy requirements of the model. Tetrahedral mesh is suitable for complex geometric shapes, but it is usually computationally inefficient. Hexahedral mesh is more accurate for regular shapes (such as cylinders), but complex models may need to be refined.

[0311] ③ Set the grid density: adjust the grid size according to the simulation accuracy requirements. A dense grid improves the calculation accuracy, but will significantly increase the calculation cost; a sparse grid may lead to insufficient accuracy.

[0312] ④ Check the mesh quality: After the mesh is divided, use the mesh quality check tool provided by ABAQUS to check the mesh quality to ensure that the mesh is not seriously distorted or excessively deformed. Appropriately adjust the size and shape of the mesh to ensure that the mesh is fine enough in key areas (such as stress concentration areas).

[0313] ⑤ Select the analysis type: According to the stress of the tree model, select the appropriate analysis type. For example, static analysis is suitable for studying the static deformation or damage of trees under external forces; while dynamic analysis is suitable for simulating the vibration, shaking or response of trees to dynamic loads (such as wind). Select the appropriate time step and solution method to set the analysis type.

[0314] ⑥ Run simulation analysis: Create a new simulation task in the Job module and set the solution parameters, select a suitable solver (such as linear static analysis solver, explicit dynamics solver, etc.). Submit the job and start the calculation analysis. ABAQUS will perform finite element solution and generate results based on the defined mesh and material model.

[0315] ⑦ Analysis results: After the analysis is completed, view the result file. Common results include deformation diagrams, stress fields, strain fields, displacements, etc. You can visualize the results in the Visualization module and analyze the behavior of the tree under different loads, such as the displacement of the tree, the location of stress concentration, etc.

[0316] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0317] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A tree modeling method based on drone image collection and image recognition, characterized in that: The following steps are involved: Step S1: planning the flight mode and flight path of the UAV according to the size, shape and distribution of trees in the target area; Step S2: using a drone to fly in a target area according to a planned flight mode and flight path to collect multi-angle image data of trees, wherein the multi-angle image data includes a set of tree images taken at multiple different viewing angles and heights; Step S3: preprocessing the multi-angle image data, including denoising, cropping, image enhancement, and standardization; Step S4: input the preprocessed multi-angle image data into a pre-trained neural network model for tree recognition and semantic segmentation, and output a semantic segmentation map of each tree image, in which each pixel in the semantic segmentation map is assigned a semantic label; Step S5: extract feature points and their descriptors from each tree image using the SIFT algorithm, match feature points between multiple tree images based on the feature points and their descriptors using the nearest neighbor matching algorithm, and remove mismatched feature points using the RANSAC algorithm; Step S6: Based on the matched feature points, a preliminary three-dimensional point cloud model is constructed by using COLMAP through an incremental reconstruction method; Step S7: Calculate the three-dimensional coordinates of each pixel in the semantic segmentation map obtained in step S4 in the preliminary three-dimensional point cloud model, determine the point cloud data corresponding to the pixel according to the three-dimensional coordinates of each pixel, and transfer the semantic label of each pixel to the corresponding point cloud data; Step S8: traverse each point cloud data in the preliminary three-dimensional point cloud model, segment the preliminary three-dimensional point cloud model according to the semantic labels of the point cloud data, and obtain a three-dimensional point cloud model of the tree; Step S9: Use the SOR filtering algorithm to perform noise reduction and optimization on the tree 3D point cloud model, use the Poisson reconstruction method to obtain the tree 3D mesh model of the tree after noise reduction and optimization, and construct the tree geometric model based on the tree 3D mesh model.

2. The tree modeling method based on drone image acquisition and image recognition according to claim 1 is characterized in that: The semantic labels include trunk, leaves and background.

3. The tree modeling method based on drone image acquisition and image recognition according to claim 1 is characterized in that: The neural network model is a Mask R-CNN model, and the training process of the Mask R-CNN model includes the following steps: Obtain a collection of tree images taken from multiple different perspectives and heights, and preprocess the tree image collection, including denoising, cropping, image enhancement, and standardization; Use the image annotation tool LabelMe to manually annotate each area in the preprocessed tree image set, annotate the parts of the tree trunk blocked by leaves as leaves, assign labels to each pixel in the tree image and generate corresponding segmentation masks, and convert the annotated tree image into a binary mask map; The preprocessed tree image set and the binary mask map are input into the Mask R-CNN model for training, and the binary mask map of the tree image is output. According to the output binary mask map and the annotated binary mask map, the loss of each pixel is calculated through the loss function. The calculated total loss value is used to update the model parameters through the back propagation algorithm.

4. A tree modeling method based on drone image acquisition and image recognition according to claim 2 or 3, characterized in that: The loss function of the Mask R-CNN model is: Where L is the loss function of the Mask R-CNN model, W is the loss weight matrix of the Mask R-CNN model, which represents the loss weight between each predicted semantic label and the true semantic label, p is the p-th pixel of the tree image, y(p) is the true semantic label of the p-th pixel, is the predicted semantic label of the p-th pixel output by the Mask R-CNN model, is the probability that the p-th pixel belongs to the predicted semantic label of the Mask R-CNN model, N is the total number of pixels in the tree image, the background label is 0, the trunk label is 1, and the leaf label is 2.

5. The tree modeling method based on drone image acquisition and image recognition according to claim 4 is characterized in that: The loss weight matrix W of the Mask R-CNN model is: Among them, the rows represent the true semantic labels, the columns represent the predicted semantic labels, and the values ​​represent the loss weights between the semantic labels.

6. The tree modeling method based on drone image acquisition and image recognition according to claim 1 is characterized in that: The method of using the nearest neighbor matching algorithm to match feature points between multiple tree images according to the local feature points of each tree image comprises the following steps: Extracting feature points and descriptors from each image, wherein the feature points are identified by the scale-invariant feature transform (SIFT) algorithm, and the descriptors are 128-dimensional vectors for each feature point; The descriptors in every two tree images are matched, the Euclidean distance between each pair of descriptors is calculated, and each feature point is matched according to the nearest Euclidean distance to obtain matching point pairs.

7. The tree modeling method based on drone image acquisition and image recognition according to claim 1 is characterized in that: The method of constructing a preliminary 3D point cloud model based on matched feature points by using COLMAP through an incremental reconstruction method includes the following steps: Step A1: Calculate the relative pose between images based on the matching feature point pairs; Step A2: Based on the relative position and matching feature point pairs, a preliminary three-dimensional point cloud is calculated by triangulation; Step A3: adding at least one image to the reconstruction process, and calculating the relative pose of the added image using matching feature point pairs between the added image and the existing 3D point cloud; Step A4: According to the pose and matching feature points of the newly added image, the 3D point cloud is updated by triangulation, and bundle adjustment is performed to optimize the 3D point cloud and camera pose; Step A5: Repeat steps A3 and A4 to gradually update and optimize the 3D point cloud and generate a preliminary 3D point cloud model.

8. The tree modeling method based on drone image acquisition and image recognition according to claim 1 is characterized in that: The step S7 comprises the following steps: According to the two-dimensional coordinates (u, v) of each pixel in the semantic segmentation map and the internal and external parameters of the camera, the three-dimensional coordinates of the pixel in the preliminary three-dimensional point cloud model are calculated by back-projection through the projection matrix. Wherein, (X, Y, Z) is the three-dimensional coordinate of the pixel in the preliminary three-dimensional point cloud model, Z(u, v) is the depth information of the pixel, and K is the intrinsic parameter matrix of the camera. The intrinsic parameter matrix K of the camera is: Among them, f x 、f y is the focal length of the camera, the scaling factor along the x and y axes, c x 、c y is the principal point coordinate of the camera; Find the point cloud data point closest to the three-dimensional coordinates of the pixel point in the preliminary three-dimensional point cloud model; Transfer the semantic label of a pixel to the closest point cloud data point.

9. The tree modeling method based on drone image acquisition and image recognition according to claim 1 is characterized in that: The method of traversing each point cloud data in the preliminary three-dimensional point cloud model and segmenting the preliminary three-dimensional point cloud model according to the semantic labels of the point cloud data to obtain a three-dimensional point cloud model of a tree includes the following steps: Traversing the point cloud data without semantic labels in the preliminary three-dimensional point cloud model, calculating the point cloud data closest to it, and setting the semantic label of the point cloud data closest to it as the semantic label of the currently traversed point cloud data without semantic labels; Each point cloud data in the preliminary 3D point cloud model is traversed, and the point cloud data with the semantic label as background is deleted to obtain the 3D point cloud model of the tree.

10. The tree modeling method based on drone image acquisition and image recognition according to claim 1 is characterized in that: The method of using the SOR filtering algorithm to perform noise reduction optimization on the tree three-dimensional point cloud model includes the following steps: For each point cloud data p in the tree 3D point cloud model i , define its neighborhood N(p i ), the neighborhood N(p i ) represents the point cloud data p i Other point cloud data sets within a certain radius; Calculate each point cloud data p i All points p in the neighborhood of j With point cloud data i The Euclidean distance d(p i , p j ), the formula is: Among them, (X i , Y i , Z i ) and (X j , Y j , Z j ) are point cloud data p i With point cloud data j 3D coordinates in the 3D point cloud model of the tree; Calculate each point p i The mean distance μ(p i ) and standard deviation σ(p i ), the formula is: Among them, |N(p i )| is the point cloud data p i The number of points in the neighborhood; According to each point cloud data p i The mean distance μ(p i ) and standard deviation σ(p i ), determine the point cloud data p i Is it an outlier? If the following conditions are met, the point cloud data p is considered i For outliers: d(p i ,p j )>μ(p i )+θ·σ(p i ) Among them, θ is the preset threshold factor; The point cloud data judged as outliers are removed from the three-dimensional point cloud model of the tree to obtain a denoised and optimized three-dimensional point cloud model of the tree.

Images