Automatic extraction method of multi-level branch structure of trees based on natural images

Abstract

The present invention discloses a method for automatically extracting multi-level branch structures of trees based on natural images, and belongs to the field of computer graphics technology. The present invention first performs tree image annotation processing on the collected tree images to generate branch annotation data. Then, the branch annotation data are processed to generate vector data with the same length that meets the requirements of the neural network model. Then, the neural network model is trained using the tree image data and the processed branch annotation data, and the trained neural network model is output and saved. Furthermore, the loaded neural network model file and the tree image file are processed, and the results are calculated and output. Finally, the above output results are read to generate the trunk and the first-level side branch structure, and the results are stored in a file. Compared with the prior art, the method proposed by the present invention has a high degree of automation, is easy for users to operate, has a stronger ability to process complex background images, and generates a higher quality branch structure.

Description

Technical Field

[0001] The invention relates to the technical field of computer graphics, and in particular to a method for automatically extracting multi-level branch structures of trees based on natural images. Background Art

[0002] The automatic extraction of tree branches is widely used in agriculture and forestry, such as obstacle avoidance of drones under forests, branch positioning of intelligent picking robots in orchards, mechanized pruning, and monitoring of tree growth. The existing related technologies are mainly divided into two categories: 1) image-based branch extraction; 2) three-dimensional point cloud-based branch extraction. Among them, the image-based branch extraction methods mainly include methods based on traditional image processing technology and methods based on deep learning technology. Based on the traditional image processing technology method, its general workflow is: first, the tree image is obtained through an RGB camera, and then the main trunk area of ​​the tree is segmented or edge extracted according to the color information of different pixels in the image. The method based on deep learning technology generally adopts the general target detection neural network model in the field of computer vision to directly calculate the detection box (rectangular box) of the area containing branches in the tree image. Its general workflow includes: data collection and processing, transfer learning of neural network models, and deployment and application of neural network models. The output of this type of method is generally a series of rectangular detection boxes containing the target. The branch extraction method based on 3D point cloud mainly uses LiDAR to obtain 3D point cloud, and then identifies and segment the trunk point cloud. This type of method mainly uses clustering method to identify the trunk point cloud. The general steps are: calculation of local features of point cloud, selection of threshold of trunk point cloud features, clustering and segmentation of trunk point cloud. The selection of threshold of trunk point cloud features is the core of this technology, which mainly includes manual selection through experiments and automatic learning of thresholds using machine learning technology.

[0003] The main problems and shortcomings of existing image-based tree branch extraction technology are:

[0004] (1) Based on traditional image processing technology, when the background in the image is complex, the segmentation and contour extraction errors of the branch area are large. The reason for this shortcoming is that this type of technology is mainly based on the color information of the pixel for segmentation. When the background pixel color is difficult to distinguish, it will cause pixel misjudgment.

[0005] (2) Based on traditional image processing technology, it can generally only extract the trunk of a tree, but often cannot extract the side branches of a tree. The reason for this shortcoming is that the occlusion between the leaves and branches of a tree cannot be simply processed by information such as pixel color, resulting in the failure of extracting the side branches of a tree.

[0006] (3) Existing image-based methods can only output the visible area of ​​the tree trunk that is not blocked by leaves. The reasons are: 1) the blocked area cannot be identified by the color information of the pixels, so the trunk cannot be extracted; 2) the neural network model relies on the corresponding dataset for training, but there is currently a lack of such image datasets that contain complete tree trunk structure labels.

[0007] (4) The existing image-based methods extract the trunk area of ​​the tree as either a set of pixels containing the trunk or a rectangular frame containing the trunk, and cannot obtain the overall geometric structure of the trunk (such as the location of branch nodes on the trunk, the connection relationship between nodes, etc.). The reasons are: 1) The trunk extraction method based on pixel color information can only classify pixels in the end, so the output of this type of method is a set of pixels, and the geometric structure of the trunk cannot be extracted; 2) The method based on deep learning technology uses the common target detection model in computer vision. The output of this type of model is a set of rectangular detection frames, rather than the specific geometric structure of the trunk; 3) The geometric structures of branches and trunks of different trees vary greatly, and existing technologies cannot handle these differences to support the training of the deep general network model.

[0008] The main problems of existing tree branch extraction technology based on 3D point cloud are:

[0009] (1) LiDAR is more expensive and complicated to operate than RGB cameras;

[0010] (2) The volume of 3D point cloud data is large, the computational overhead is high, and it is difficult to achieve real-time processing;

[0011] (3) The tree trunk obtained based on 3D point cloud segmentation is a set of discrete points, which cannot directly obtain the geometric structure of the tree trunk.

[0012] In order to solve the above problems, the present invention proposes a method for automatically extracting multi-level branch structures of trees based on natural images. Summary of the invention

[0013] The purpose of the present invention is to propose a method for automatically extracting multi-level branch structures of trees based on natural images to solve the following problems existing in the prior art:

[0014] (1) The problem of extracting tree branch structures in complex background images;

[0015] (2) The problem of extracting the multi-level branch structure of trees;

[0016] (3) The problem of extracting the geometric structure of tree branches.

[0017] In order to achieve the above object, the present invention adopts the following technical solutions:

[0018] The automatic extraction method of multi-level branch structure of trees based on natural images uses a deep neural network model to extract the multi-level branch structure of trees from images, which specifically includes the following contents:

[0019] S1. Constructing a dataset: Collect tree images and branch annotation data to construct a dataset for training deep neural networks, which further includes the following:

[0020] S1.1. Tree image acquisition: tree images are obtained by crawling the Internet and taking pictures with a mobile phone. The tree images are all RGB images and stored in JPEG format;

[0021] S1.2, tree image annotation: use annotation software to annotate the branch and trunk structure in the tree image obtained in S1.1;

[0022] S1.3, branch labeling data processing: Process the branch labeling data obtained in S1.2 to construct data with unified dimensions;

[0023] S2. Construct a deep neural network model: The deep neural network model adopts a structure of a skeleton network and a branch network, wherein the skeleton network adopts a BotNet grid structure, which is composed of a convolutional residual module and a multi-head self-attention module; the branch network is composed of a fully connected module; construct a loss function of the deep neural network model, and train the model;

[0024] S3. Construct a multi-level tree trunk structure based on the output results of the deep neural network model: The input of the deep neural network model is a tree image, and the output is a vector composed of a series of branch node coordinate values. Based on the output results, the tree trunk structure is constructed.

[0025] Preferably, the S1.2 specifically includes the following contents:

[0026] S1.2.1. Select tree images and visualize the selected images using annotation software;

[0027] S1.2.2. The user selects the level of the branches to be marked: the main trunk corresponds to level 0, the side branches directly connected to the main trunk correspond to level 1, the secondary side branches directly connected to the above level 1 side branches correspond to level 2, and so on;

[0028] S1.2.3. The user directly clicks on the tree image to mark the nodes that make up the branches, and the marking software automatically obtains and records their two-dimensional coordinates and the corresponding branch level; when the user completes the marking of all nodes of a branch, the marking software automatically connects these nodes to construct a point-line diagram of the branch and saves it;

[0029] S1.2.4. The user repeats the above steps S1.2.2 to S1.2.3 until the marking of m branch levels is completed; m represents the number of levels pre-specified by the user.

[0030] Preferably, the annotation data generated by the annotation software is uniformly saved in XML format. Different branch levels correspond to a group of annotation data, and each group of annotation data contains two types of labels: nodes and branches. The node label stores the two-dimensional coordinates of all branch nodes at the same level. Each group can contain multiple branch labels, and each branch label stores the coordinates of all nodes that constitute the branch.

[0031] Preferably, the S1.3 specifically includes the following contents:

[0032] S1.3.1. Set the number of branch levels m in the data set to 2. Each data point in the branch annotation data contains two values ​​(x, y). Normalize the coordinate values:

[0033] x'=x / W

[0034] y'=y / H

[0035] Among them, W and H are the width and height of the image respectively;

[0036] S1.3.2. Based on the normalized branch annotation data, the following three types of distribution information are counted: the number of trunk annotation points, the number of primary side branches, and the total number of annotation points of all primary side branches;

[0037] S1.3.3, based on the statistical information obtained in S1.3.2, determine the number of trunk annotation points N0 and the total number of annotation points of all first-level side branches N1;

[0038] S1.3.4, using the natural cubic spline interpolation method, for the trunk and all first-level side branches, according to the coordinates of their marked nodes, a smooth curve with continuous second-order derivative is generated;

[0039] S1.3.5, the smooth curves obtained by fitting the trunk and all the first-level side branches are calculated according to N0, N1 and the number of side branches N. L Resample the nodes;

[0040] S1.3.6. The branch annotation information of each tree image is uniformly represented as a one-dimensional vector of length D = (N0+N1)*2+1, where N0+N1 represents the total number of branch nodes; multiplying by 2 means that each node contains two coordinate values ​​(x, y); plus 1 means that an additional value is required to store the number of first-level side branches.

[0041] Preferably, S1.3.4 specifically includes the following contents:

[0042] Assume there are N+1 labeled nodes, where the coordinates of the i-th node are labeled P i , whose tangent vector is denoted by P i ', 0≤i≤N, then the construction steps of the curve are as follows:

[0043] ①Solve the linear equation system A*b=c and calculate the tangent vector P at each node i ',in

[0044]

[0045] ② Every two adjacent nodes (P i , P i '), (P i+1 , P i+1 ') Construct a cubic spline curve, the formula is as follows:

[0046] C i (u)=H1(u)P i +H2(u)P i+i +H3(u)P′ i +H4(u)P′ i+1 (2)

[0047] in:

[0048] H1(u)=(2u 3 -3u 2 +1); H2(u)=(-2u 3 +3u 2 ); H3(u)=(u 3 -2u 2 +u); H4(u)=(u 3 -u 2 );0≤u≤1.

[0049] Preferably, S1.3.5 specifically includes the following contents:

[0050] ① Trunk resampling: Assuming that there are N+1 labeled nodes on the trunk, it contains N segments of cubic spline curves. For each segment of curve C i (u), by uniformly sampling N0 / N values ​​of the parameter u in the interval [0, 1], to calculate the corresponding node coordinates;

[0051] ②First-level side branch resampling: Assuming there are N L First-level side branches, each side branch needs to sample N1 / N L points; the sampling method of each side branch is consistent with the trunk resampling method (i.e., the natural cubic spline fitting curve is uniformly sampled).

[0052] Preferably, the loss function for constructing the deep neural network model mentioned in S2 specifically includes the following contents:

[0053] The neural network has three branch networks, each branch network corresponds to an independent loss function, and the total loss function formula is as follows:

[0054]

[0055] Where x represents the input tree image; y represents the tree branch node coordinate vector of length D, which is composed of three sub-vectors (y1, y2, y3); y1 is the trunk node coordinate vector; y2 is the branch node coordinate vector; y3 is the number of side branches; Θ represents the model parameters of the neural network; Loss is the total loss function, which is composed of three sub-loss functions and the regular constraint term on the parameter Θ, and each loss function corresponds to the output of a branch network; parameters α, β and γ are the weights of each sub-loss function, which are used to control its influence on the total loss function;

[0056] (3) In the formula, loss1 corresponds to the loss function of the trunk node coordinate prediction result, and its calculation formula is as follows:

[0057]

[0058] Loss2 corresponds to the loss function of the first-level side branch node coordinate prediction result, and its calculation formula is as follows:

[0059]

[0060] Loss3 corresponds to the loss function of the prediction result of the number of primary side branches, and its calculation formula is as follows:

[0061]

[0062] In the above formula Calculation results corresponding to the three-branch network.

[0063] Preferably, the model training mentioned in S2 specifically includes the following contents:

[0064] S2.1, use 80% of the data set as the training set and 20% as the validation set;

[0065] S2.2. Perform data enhancement on the images in the training set. The main operations include random cropping and color transformation.

[0066] S2.3, load the data augmented dataset in batches and use the stochastic gradient descent method to train the neural network;

[0067] S2.4. Select the best model and save it.

[0068] Preferably, S3 specifically includes the following contents:

[0069] The length of the vector composed of the branch node coordinate values ​​output by the deep neural network model is recorded as D = (N0 + N1) * 2 + 1. Assuming that the branch node coordinate vector output by the deep neural network model is recorded as y, then the y[1:2N0] part corresponds to the trunk node coordinates, y[2N0 + 1:2N1] corresponds to the first-level branch node coordinates, and y[D] corresponds to the number of first-level branches. Based on the above data, the steps to construct the tree trunk structure are as follows:

[0070] ① Trunk construction: Apply the natural cubic spline curve fitting method to fit and generate the trunk curve according to the coordinate data in y[1:2N0];

[0071] ② Construction of primary side branches: First, the number of nodes on each side branch is calculated according to the formula N1 / y[D]. Then, the vector y[2N0+1:2N1] is divided equally according to the number of side branches, and the natural cubic spline curve fitting method is applied to each equal part to generate the branch-trunk curve.

[0072] ③ Save the above branch curve data into a universal OBJ format file.

[0073] Compared with the prior art, the present invention provides a method for automatically extracting multi-level branch structures of trees based on natural images, which has the following beneficial effects:

[0074] (1) Aiming at the problem of extracting the branch structure of trees in complex background images, the method proposed in the present invention can process tree images taken in natural environments. Such images contain rich backgrounds, such as buildings, roads, pedestrians, and other plants. The branch structure of trees can be directly extracted from the images without the need for specific preprocessing.

[0075] (2) Aiming at the problem of extracting the multi-level branch structure of trees, the method proposed in the present invention can not only extract the main trunk of the tree, but also extract the side branches of the tree, which is impossible in the prior art;

[0076] (3) Regarding the problem of extracting the geometric structure of tree branches, the branch structure extracted by the method proposed in the present invention is not a set of pixels or a rectangular detection frame, but a geometric structure including the location of branch nodes and the edges connecting nodes. To this end, it is necessary to solve the problem of unified mathematical representation of the multi-level branch structure of trees.

[0077] In summary, the method for automatic extraction of multi-level branch structures of trees based on natural images proposed in the present invention has a high degree of automation, is easy for users to operate, has a stronger ability to process complex background images, and generates branch structures with higher quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a flow chart of the method for automatically extracting multi-level branch structures of trees based on natural images proposed by the present invention;

[0079] Figure 2 Schematic diagram of the overall structure of the neural network in the method for automatically extracting the multi-level branch structure of trees based on natural images proposed in this invention

[0080] Figure 3 It is a flowchart of tree image annotation in the method for automatically extracting multi-level branch structure of trees based on natural images proposed by the present invention;

[0081] Figure 4 This is a flowchart of labeling data processing in the method for automatically extracting multi-level branch structures of trees based on natural images proposed by the present invention;

[0082] Figure 5 This is a flowchart of neural network training in the method for automatically extracting multi-level branch structures of trees based on natural images proposed by the present invention;

[0083] Figure 6 A histogram of the number of trunk sampling points in the method for automatically extracting multi-level branch structures of trees based on natural images proposed by the present invention;

[0084] Figure 7 A histogram of the number of sampling points of the first-level side branches in the method for automatically extracting the multi-level branch structure of trees based on natural images proposed by the present invention;

[0085] Figure 8 It is the convergence curve of the model loss function training set in the method for automatically extracting multi-level branch structures of trees based on natural images proposed by the present invention;

[0086] Fig. 9 This is the convergence curve on the validation set of the model loss function in the method for automatic extraction of multi-level branch structures of trees based on natural images proposed in the present invention. DETAILED DESCRIPTION

[0087] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0088] The present invention proposes a method for automatically extracting multi-level branch structures of trees based on natural images, which mainly includes two parts: deep neural network model construction and deep neural network model application. The deep neural network model construction includes tree image annotation, annotation data processing and deep neural network model training operations. The specific process is: first, the collected tree images will be firstly processed for tree image annotation to generate branch annotation data; then, these branch annotation data are processed to generate vector data with consistent length that meets the requirements of the neural network model; finally, the tree image data and the processed branch annotation data are used to train the neural network model, and the trained neural network model is output and saved. The deep neural network model application includes deep neural network model calculation and multi-level branch structure construction. The specific process is: first, the loaded neural network model file and the tree image file are processed, and the results are calculated and output; then, the above output results are read to generate the trunk and the first-level side branch structure, and the results are stored in the file.

[0089] Based on the above description, the method for automatically extracting multi-level branch structures of trees based on natural images proposed in the present invention specifically includes the following contents:

[0090] See also Figure 1-2 , an automatic extraction method of multi-level branch structure of trees based on natural images, adopts a deep neural network model to achieve the goal of extracting the multi-level branch structure of trees from images. Its working process is as follows:

[0091] (1) Dataset construction

[0092] The dataset is mainly used to train deep neural network models. It mainly consists of tree images and branch annotation data. Its construction process goes through the following steps:

[0093] (1.1) Tree image collection

[0094] All tree images are RGB images, which are collected mainly through two channels: web crawling and mobile phone photography. All images are saved in JPEG format.

[0095] (1.2) Tree Image Annotation

[0096] For labeling images of trees and their branches, see Figure 3 , the working process is:

[0097] A. Select the tree image, and the annotation software will visualize the selected image;

[0098] B. The user selects the level of the branches to be marked: the main trunk corresponds to level 0, the side branches directly connected to the main trunk correspond to level 1, the secondary side branches directly connected to the above level 1 side branches correspond to level 2, and so on;

[0099] C. The user directly clicks on the tree image to mark the nodes that make up the branches, and the marking software automatically obtains and records its two-dimensional coordinates and the corresponding branch level; when the user completes the marking of all nodes of a branch, the marking software automatically connects these nodes to construct a point-line diagram of the branch and saves it;

[0100] D. The user repeats the above BC steps until the labeling of m branch levels is completed; here m is the number of levels pre-specified by the user.

[0101] The annotation data generated by the annotation software is uniformly saved in XML format. Different branch levels correspond to a group of annotation data, and each group of annotation data contains two types of labels: node and branch. The node label stores the two-dimensional coordinates of all branch nodes at the same level. Each group can contain multiple branch labels, and each branch label stores the coordinates of all nodes that constitute the branch.

[0102] (1.3) Branch labeling data processing

[0103] The dimensions of the labeled data for different tree branches vary and cannot be directly used for training deep neural network models. Figure 4 , the data set constructed by the present invention first sets the number of branch levels m to 2, and then uses the following processing flow to construct data with unified dimensions:

[0104] A. Each data point in the branch annotation data contains two values ​​(x, y). The coordinate values ​​are normalized:

[0105] x'=x / W y'=y / H

[0106] W and H in the above formula are the width and height of the image.

[0107] B. Based on the normalized branch annotation data, the following three types of distribution information are counted: the number of trunk annotation points, the number of first-level side branches, and the total number of annotation points of all first-level side branches;

[0108] C. Based on the above statistical information, determine the number of trunk annotation points N0 and the total number of annotation points of all primary side branches N1; N0 and N1 are key parameters of the present invention. In the implementation process, they are mainly obtained by statistically analyzing the annotation data. For the data in the present invention, the corresponding statistical histogram is shown in the attached figure. Figure 6 and Figure 7 . It can be found from the figure that when the value range of N0 is between [7,11] and the value range of N1 is between [30,70], most of the data samples will be covered. In the implementation of the present invention, N0=10, N1=60, and the corresponding deep neural network model, when α=0.01, β=0.01, γ=0, the loss function The convergence curve on the training data set is shown in Figure 8 As shown, the loss function convergence curve on the validation data set is shown in Fig. 9 As shown in the figure, in a specific implementation, when these parameters take the above values, the corresponding results can be obtained.

[0109] D. Use the natural cubic spline interpolation method to fit the trunk and all first-level side branches according to their marked node coordinates to generate a smooth curve with continuous second-order derivatives, including the following:

[0110] Assume there are N+1 labeled nodes, where the coordinates of the i-th node are labeled P i , whose tangent vector is denoted by P i ', 0≤i≤N. Then the construction steps of the curve are as follows:

[0111] D1. Solve the linear equation system A*b=c and calculate the tangent vector P at each node i ',in

[0112]

[0113] D2, every two adjacent nodes (P i , P i '), (P i+1 , P i+1 ') Construct a cubic spline curve, the formula is as follows:

[0114] C i (u)=H1(u)P i +H2(u)P i+1 +H3(u)P′ i +H4(u)P′ i+1 (2)

[0115] in:

[0116] H1(u)=(2u 3 -3u 2 +1), H2(u)=(-2u 3 +3u 2 ),

[0117] H3(u)=(u 3 -2u 2 +u), H4(u)=(u 3 -u 2 ),

[0118] 0≤u≤1.

[0119] E. The smooth curves obtained by fitting the trunk and all the first-level side branches are calculated based on N0, N1 and the number of side branches N. LResample the nodes:

[0120] E1. Trunk resampling: Assuming that there are N+1 labeled nodes on the trunk, it contains N segments of cubic spline curves. For each segment of curve C i (u), by uniformly sampling N0 / N values ​​of the parameter u in the interval [0,1] to calculate the corresponding node coordinates;

[0121] E2, first-level side branch resampling: Assuming there are N L First-level side branches, each side branch needs to sample N1 / N L points; the sampling method for each side branch is consistent with the trunk resampling method, that is, the natural cubic spline fitting curve is uniformly sampled.

[0122] F. Finally, the branch annotation information of each tree image is uniformly represented as a one-dimensional vector of length D = (N0+N1)*2+1. Here, N0+N1 represents the total number of branch nodes, multiplied by 2 means that each node contains two coordinate values ​​(x, y), and plus 1 means that an additional value is needed to store the number of first-level side branches.

[0123] (2) Construction of deep neural network model

[0124] The deep neural network in the present invention adopts the structure of a skeleton network and a branch network. The skeleton network adopts the BotNet (Bottleneck Transformer Network) network structure, which is composed of a convolutional residual module and a multi-head self-attention module, and the branch network is composed of a fully connected module. The neural network in the present invention has three branch networks, each branch network corresponds to an independent loss function, and the total loss function formula is as follows:

[0125]

[0126] In the above formula, x represents the input tree image; y represents the tree branch node coordinate vector of length D, which is composed of three sub-vectors (y1, y2, y3), y1 is the trunk node coordinate vector, y2 is the branch node coordinate vector, and y3 is the number of side branches; Θ represents the model parameters of the neural network; Loss is the total loss function, which is composed of three sub-loss functions and regular constraints on the parameter Θ, and each loss function corresponds to the output of a branch network; parameters α, β and γ are the weights of each sub-loss function, which are used to control their influence on the total loss function. Among them, loss1 corresponds to the loss function of the trunk node coordinate prediction result, and its calculation formula is as follows:

[0127]

[0128] In addition, loss2 corresponds to the loss function of the first-level side branch node coordinate prediction result, and its calculation formula is as follows:

[0129]

[0130] Finally, loss3 corresponds to the loss function of the prediction result of the number of first-level side branches, and its calculation formula is as follows:

[0131]

[0132] In the above formula Calculation results corresponding to the three-branch network.

[0133] See also Figure 5 , the training process of the deep neural network model based on the above loss function is as follows:

[0134] A. Use 80% of the data set as a training set and 20% as a validation set;

[0135] B. Perform data augmentation on the images in the training set. The main operations include random cropping and color transformation.

[0136] C. Load the data augmented dataset in batches and train the neural network using stochastic gradient descent.

[0137] D. Select the best model and save it.

[0138] (3) Construction of multi-level tree trunk structure based on the output of deep neural network model

[0139] The input of the deep neural network model is a tree image, and the output is a vector consisting of a series of branch node coordinate values, whose length is D = (N0 + N1) * 2 + 1. Assume that the branch node coordinate vector output by the neural network model is y, then the y[1:2N0] part corresponds to the main trunk node coordinates, y[2N0 + 1:2N1] corresponds to the first-level branch node coordinates, and y[D] corresponds to the number of first-level branches. Based on the above data, the steps to construct the tree trunk structure are as follows:

[0140] A. Backbone construction: Apply the natural cubic spline curve fitting method in 1.3-D above to fit the backbone curve according to the coordinate data in y[1:2N0]

[0141] B. Construction of primary side branches: First, the number of nodes on each side branch is calculated according to the formula N1 / y[D]. Then, the vector y[2N0+1:2N1] is divided equally according to the number of side branches, and the natural cubic spline curve fitting method in 1.3-D mentioned above is applied to each equal part to generate the branch-trunk curve.

[0142] C. Save the above branch curve data into a general OBJ format file.

[0143] In combination with the above description, the present invention includes the following specific implementation modes:

[0144] Embodiment 1:

[0145] Simple application method:

[0146] This method mainly performs Figure 1 The neural network model application operation on the right. First, read the trained neural network model file and build the corresponding neural network model; then, input the tree image to be processed, call the neural network model, calculate the branch node coordinates and the number of first-level side branches; finally, based on the output data of the previous step, complete the multi-level branch structure construction, generate the final branch curve and save it to the file.

[0147] Embodiment 2:

[0148] Complete build method:

[0149] This method will Figure 1 The complete process in the embodiment of the present invention is used to implement all the processes involved in the present invention. First, execute Figure 1 The main steps of the neural network model construction process are:

[0150] 1. Collect tree image data and follow the process in tree image annotation operation (see Appendix Figure 3 ), complete the labeling of branch data;

[0151] 2. For the branch and trunk labeling data, follow the process in the labeling data processing operation (see Appendix Figure 4 ), complete the processing of labeled data;

[0152] 3. Based on the tree image data and the processed branch annotation data, follow the process in the neural network model training operation (see Appendix Figure 5 ), complete the construction of the neural network model and save it.

[0153] 4. According to the process in the above embodiment 1, the final branch structure is generated.

[0154] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for automatically extracting multi-level branch structures of trees based on natural images, characterized in that: A deep neural network model is used to extract the multi-level branch structure of trees from images, including the following: S1. Constructing a dataset: Collect tree images and branch annotation data to construct a dataset for training deep neural networks, which further includes the following: S1.

1. Tree image acquisition: tree images are obtained by crawling the Internet and taking pictures with mobile phones. The tree images are all RGB images and stored in JPEG format; S1.

2. Tree image annotation: Use annotation software to annotate the branch and trunk structure in the tree image obtained in S1.1; specifically include the following contents: S1.2.

1. Select tree images and visualize the selected images using annotation software; S1.2.

2. The user selects the level of the branches to be marked: the main trunk corresponds to level 0, the side branches directly connected to the main trunk correspond to level 1, the secondary side branches directly connected to the above level 1 side branches correspond to level 2, and so on; S1.2.

3. The user directly clicks on the tree image to mark the nodes that make up the branches, and the marking software automatically obtains and records their two-dimensional coordinates and the corresponding branch level; when the user completes the marking of all nodes of a branch, the marking software automatically connects these nodes to construct a point-line diagram of the branch and saves it; S1.2.

4. The user repeats the above steps S1.2.2 to S1.2.3 until the marking of m branch levels is completed; m represents the number of levels pre-specified by the user; S1.3, branch labeling data processing: Process the branch labeling data obtained in S1.2 to construct data with unified dimensions; S2. Construct a deep neural network model: The deep neural network model adopts a structure of a skeleton network and a branch network, wherein the skeleton network adopts a BotNet grid structure, which is composed of a convolutional residual module and a multi-head self-attention module; the branch network is composed of a fully connected module; construct a loss function of the deep neural network model, and train the model; S3. Construct a multi-level tree trunk structure based on the output results of the deep neural network model: The input of the deep neural network model is a tree image, and the output is a vector composed of a series of branch node coordinate values. Based on the output results, the tree trunk structure is constructed.

2. The method for automatically extracting multi-level tree branch structures based on natural images according to claim 1 is characterized in that: S1.3 specifically includes the following contents: S1.3.

1. Set the number of branch levels m in the data set to 2. Each data point in the branch annotation data contains two values ​​(x, y). Normalize the coordinate values: x' = x / W y' = y / H Among them, W and H are the width and height of the image respectively; S1.3.

2. Based on the normalized branch annotation data, the following three types of distribution information are counted: the number of trunk annotation points, the number of primary side branches, and the total number of annotation points of all primary side branches; S1.3.3, based on the statistical information obtained in S1.3.2, determine the number of trunk annotation points N0 and the total number of annotation points of all first-level side branches N1; S1.3.4, using the natural cubic spline interpolation method, for the trunk and all first-level side branches, according to the coordinates of their marked nodes, a smooth curve with continuous second-order derivative is generated; S1.3.5, for the smooth curves obtained by fitting the trunk and all the first-level side branches, resample the nodes according to N0, N1 and the number of side branches NL; S1.3.

6. The branch annotation information of each tree image is uniformly represented as a one-dimensional vector of length D=(N0+N1)*2+1, where N0+N1 represents the total number of branch nodes; multiplying by 2 means that each node contains two coordinate values ​​(x, y); adding 1 means that an additional value is required to store the number of first-level side branches.

3. The method for automatically extracting multi-level tree branch structures based on natural images according to claim 2 is characterized in that: S1.3.4 specifically includes the following contents: Assume that there are N+1 labeled nodes, where the coordinates of the i-th node are denoted as Pi, and its tangent vector is denoted as Pi', 0 ≤ i ≤N, then the steps for constructing the curve are as follows: ①Solve the linear equation system A*b=c and calculate the tangent vector Pi' at each node, where （1） ② Every two adjacent nodes (P i , P i '), (P i+1 , P i+1 ') Construct a cubic spline curve with the following formula: （2） in: ；0 ≤ u ≤ 1。 4. The method for automatically extracting multi-level tree branch structures based on natural images according to claim 2 is characterized in that: S1.3.5 specifically includes the following contents: ① Trunk resampling: Assuming that there are N+1 labeled nodes on the trunk, it contains N segments of cubic spline curves. For each segment of the curve Ci(u), the corresponding node coordinates are calculated by uniformly sampling N0 / N values ​​of the parameter u in the interval [0,1]; ② First-level side branch resampling: Assuming there are NL first-level side branches, each side branch needs to sample N1 / NL points; the sampling method of each side branch is consistent with the trunk resampling method.

5. The method for automatically extracting multi-level tree branch structures based on natural images according to claim 1, characterized in that: The loss function for constructing the deep neural network model mentioned in S2 specifically includes the following contents: The neural network has three branch networks, each branch network corresponds to an independent loss function, and the total loss function formula is as follows: （3） Where x represents the input tree image; y represents the tree branch node coordinate vector of length D, which is composed of three sub-vectors (y1, y2, y3); y1 is the trunk node coordinate vector; y2 is the branch node coordinate vector; y3 is the number of side branches; Θ represents the model parameters of the neural network; Loss is the total loss function, which is composed of three sub-loss functions and the regular constraint term on the parameter Θ, and each loss function corresponds to the output of a branch network; parameters α, β and γ are the weights of each sub-loss function, which are used to control its influence on the total loss function; (3) In the formula, loss1 corresponds to the loss function of the trunk node coordinate prediction result, and its calculation formula is as follows: （4） Loss2 corresponds to the loss function of the first-level side branch node coordinate prediction result, and its calculation formula is as follows: （5） Loss3 corresponds to the loss function of the prediction result of the number of primary side branches, and its calculation formula is as follows: （6） In the above formula Calculation results corresponding to the three-branch network.

6. The method for automatically extracting multi-level tree branch structures based on natural images according to claim 1, characterized in that: The model training mentioned in S2 specifically includes the following contents: S2.1, use 80% of the data set as the training set and 20% as the validation set; S2.

2. Perform data enhancement on the images in the training set. The main operations include random cropping and color transformation. S2.3, load the data augmented dataset in batches and use the stochastic gradient descent method to train the neural network; S2.

4. Select the best model and save it.

7. The method for automatically extracting multi-level tree branch structures based on natural images according to claim 1, characterized in that: The S3 specifically includes the following contents: The length of the vector composed of the branch node coordinate values ​​output by the deep neural network model is recorded as D=(N0+N1)*2+1. Assuming that the branch node coordinate vector output by the deep neural network model is recorded as y, then the y[1:2N0] part corresponds to the trunk node coordinates, y[2N0+1:2N1] corresponds to the first-level branch node coordinates, and y[D] corresponds to the number of first-level branches. Based on the above data, the steps to construct the tree trunk structure are as follows: ① Trunk construction: Apply the natural cubic spline curve fitting method to fit the trunk curve according to the coordinate data in y[1:2N0]; ② Construction of primary side branches: First, the number of nodes on each side branch is calculated according to the formula N1 / y[D]. Then, the vector y[2N0+1: 2N1] is divided equally according to the number of side branches, and the natural cubic spline curve fitting method is applied to each equal part to generate the branch-trunk curve. ③ Save the above branch curve data into a universal OBJ format file.

Images