Tree branch single-view point cloud reconstruction method and system in real scene

Abstract

The invention discloses a single-view point cloud reconstruction method for tree branches in a real scene. The method comprises the following steps: acquiring a single RGB image of the tree branches in the real scene; performing single-view depth estimation based on loop prediction on the single RGB image of the tree branch to obtain a plurality of depth images; performing depth image-to-point cloud conversion on the plurality of depth images to obtain sparse point clouds; carrying out sparse point cloud densification on the sparse point cloud to obtain a dense point cloud; performing point cloud coordinate fine correction on the dense point cloud to obtain a reconstruction result; and reconstructing the tree branch point cloud in the real scene based on the reconstruction result. The invention further discloses a corresponding system, electronic equipment and a computer readable storage medium. The problem that the depth information of the shielded branches cannot be effectively estimated is solved through single-view depth estimation based on loop prediction; the tree morphological feature analysis-based point cloud denseness algorithm and the point cloud correction method based on point-by-point correction solve the problem that the preliminarily obtained tree branch point cloud is sparse and inaccurate.

Description

Technical Field

[0001] The present invention relates to the technical field of target point cloud reconstruction, and in particular to a method and system for reconstructing a single-view point cloud of a tree trunk in a real scene. Background Art

[0002] In recent years, with the rise of digital orchards, the demand for efficient, automated, and digital orchard management has increased. Among them, tree trunk reconstruction technology has emerged as one of the important means of digital orchard management. This technology aims to meet the needs of tree morphological analysis, growth monitoring, and intelligent pruning in orchard management, and provide more data support for orchard managers. The continuous development of deep learning and computer vision provides strong technical support for tree trunk reconstruction. Using these technologies, researchers can reconstruct the three-dimensional structure of tree trunks from the collected data.

[0003] At present, some scholars have started to study the tree trunk reconstruction technology. However, in the process of tree trunk reconstruction in actual scenarios, due to the mutual occlusion between leaves, branches and other organs and branches, some tree trunks are blocked and cannot be observed. The occlusion problem makes tree trunk reconstruction more complicated and difficult. Solving the occlusion problem is of great significance for the complete reconstruction of tree structure, which helps to achieve more comprehensive tree morphological analysis, allowing orchard managers to better understand the growth status of trees, and can also provide strong support for management decisions such as tree growth monitoring and intelligent pruning, and promote orchard management to develop in a more efficient and intelligent direction.

[0004] The reconstruction problem of the occluded part of tree branches can be divided into the following three categories according to the different original data used: reconstruction based on LiDAR point cloud, reconstruction based on RGB image and depth image, and reconstruction based on RGB image. Among them, the reconstruction method based on RGB image has obvious advantages over other methods in data acquisition and processing. Specifically, the reconstruction method based on RGB image does not require image data from multiple perspectives, which reduces the complexity and cost of data acquisition. Therefore, the present invention will focus on the tree branch reconstruction method based on RGB image for in-depth research. The reconstruction based on LiDAR point cloud belongs to the method of converting residual point cloud to fine point cloud, which has a low relevance to the present invention, so the present invention provides the progress of the latter two studies.

[0005] Reconstruction research based on RGB images and depth images usually uses RGBD cameras to collect data and obtain RGB images and depth images at the same time. Due to the complexity of actual orchard scenes, the collected depth images are usually inaccurate. Therefore, it is necessary to use image processing and deep learning techniques to deal with the problem of inaccurate depth information caused by occlusion. For example, Geckeler C et al. used RGBD cameras to collect RGB images and depth images of trees. They found that the depth images of the collected tree depth images had depth faults due to occlusion. They proposed a U-Net-based network model to optimize the depth images of occluded branches. Experimental results show that this method can effectively improve the reconstruction effect of occluded areas. Kok E et al. proposed a Unet++-based model for the problem of branch occlusion in natural orchards. This method can restore the three-dimensional structure of the occluded trees from the RGB images and depth images captured by the RGBD camera. Experimental results show that the proposed method can effectively reconstruct the spatial structure of occluded branches. Kim CH et al. proposed a method to estimate the tree skeleton hidden inside the tree crown from RGB images and depth images to solve the problem that pruning robots equipped with RGBD cameras cannot accurately perceive the tree skeleton hidden inside the tree crown, and thus cannot effectively prune. The authors conducted experiments on simulated data and real data, respectively, and proved that the proposed method can effectively estimate the occluded tree skeleton structure. Lin G et al. proposed a three-dimensional reconstruction method to solve the problem that the depth image collected by RGBD camera has depth faults, resulting in poor reconstruction effect. The method first uses micro Mask R-CNN to detect branches and trunks, converts the detected branches and trunks into 3D point clouds, then uses 3D cylinders to simulate the shape of branches, and finally realizes the reconstruction of branches and trunks based on the cylinder segment fitting method of principal component analysis. The experimental results show that the proposed method can effectively solve the problem of poor reconstruction of branches in occluded areas.

[0006] Reconstruction research based on RGB images can be divided into point cloud reconstruction methods based on multi-views and point cloud reconstruction methods based on single view.

[0007] 1) Point cloud reconstruction method based on multi-view: Structure from Motion and Multi View Stereo (SfM-MVS). Gené-Mola J et al. used SfM-MVS technology to reconstruct the branches of apple trees in three dimensions. The results showed that compared with other reconstruction methods, SfM-MVS technology can better deal with the problem of poor reconstruction effect caused by occlusion. Nader KB et al. proposed a multi-view reconstruction method for grape branches. By controlling the different shooting angles of the camera, it effectively solved the problem of poor effect caused by occlusion when reconstructing grape branches. He W et al. used multi-view images obtained by smartphones to reconstruct the three-dimensional point cloud of soybean plants. Then the detection and segmentation of soybean point clouds were realized. The author proposed that by using smartphones, a mobile micro device, the shooting angle can be flexibly adjusted to obtain unobstructed soybean plant reconstruction point clouds. G et al. conducted a comprehensive analysis of the processing parameters, image redundancy, and acquisition geometry in the SfM-MVS technology. The authors proposed that by adjusting the processing parameters and image acquisition angles in the SfM-MVS technology, the reconstruction accuracy of the occluded area can be effectively improved. The main reason why multi-view reconstruction can solve the occlusion problem is that by adding cameras or shooting positions, the image data of the tree can be obtained from different perspectives. When a part is occluded, other perspectives can usually capture the information of that part, thereby making up for the lack of the occluded part. Through image data from multiple perspectives, more comprehensive and complete tree structure information can be obtained, thereby achieving higher-precision tree branch reconstruction. However, this type of research requires high data acquisition costs, requires the use of multiple cameras or devices for image acquisition, and the data processing and calculation are relatively cumbersome. In addition, this method has high requirements for algorithms and computing power, resulting in a relatively slow reconstruction process.

[0008] 2) Point cloud reconstruction method based on single view: The point cloud reconstruction method based on single view uses a single RGB image to reconstruct the branches of trees. It usually learns the mapping from RGB images to depth images by training a depth estimation network, and then converts the depth image into a 3D point cloud to complete the 3D reconstruction. Guénard J et al. proposed a single-view tree reconstruction method. This method inputs a single background-free image of a tree, combines the prior knowledge of tree morphology, first roughly estimates the skeleton of the tree, and then obtains the 3D structure of the tree by changing the parameter values ​​of the generative model (the main branch structure and leaves of the tree). Experimental results show that the proposed single-view tree reconstruction method can effectively reconstruct the 3D structure of the tree. Liu Z et al. used conditional generative adversarial networks (cGAN) to extract the edge information of the tree from the image, and combined with the 2D skeleton strokes manually drawn by the user to infer the 3D shape of the tree. Experimental results show that the proposed method is effective in reconstructing 3D tree models from a single image. The innovation of this method is that it is the first to use a single image to store the front and back layers of depth information of the tree crown, so that the envelope shape of the occluded area on the back of the tree crown can be reconstructed. Since the research on point cloud reconstruction based on a single view only requires one RGB image as input data, data collection and processing are simpler and less expensive than RGBD camera reconstruction and multi-view reconstruction technology. At present, there are relatively few studies on the reconstruction of tree branches using point cloud reconstruction technology based on a single view, and solving the occlusion problem is a major challenge to be solved in the field of point cloud reconstruction.

[0009] In summary, the tree branch point cloud reconstruction method based on a single view has obvious advantages over the multi-view point cloud reconstruction method in data acquisition and processing, that is, the single view method does not require image data from multiple perspectives, which reduces the complexity and cost of data acquisition; at the same time, during the processing, the single view method simplifies the data processing process and improves the processing efficiency. In view of these advantages, the present invention will focus on the tree branch reconstruction method based on a single view to carry out research. In the tree branch reconstruction technology, the reconstruction method based on a single view has the advantage of simpler data acquisition and processing process compared with the multi-view method because it only requires one RGB image as input data. However, the current research on tree branch reconstruction based on a single view mainly focuses on the point cloud reconstruction of tree branches in virtual simulation scenes. The research is usually conducted in relatively simple scenes without background interference, and the complexity of tree growth and occlusion interference in real scenes are not considered enough. In actual scenes, the mutual occlusion between branches increases the complexity and difficulty of reconstruction. Solving the occlusion problem is crucial for completely reconstructing the structure of tree branches, which not only helps to achieve more accurate tree morphology analysis, but also supports management decisions such as tree growth monitoring and intelligent pruning. Therefore, it is particularly important to study new point cloud reconstruction methods to achieve complete reconstruction of tree branches in real scenes, that is, in view of the complex growth state of trees and the existence of occlusion interference, it is necessary to study new point cloud reconstruction methods to achieve complete reconstruction of tree branches in real scenes. Summary of the invention

[0010] In order to solve the problems existing in the prior art, a single-view point cloud reconstruction method and system for tree branches in a real scene are provided. In the method and system, firstly, a single-view depth estimation method based on cyclic prediction is proposed, which solves the problem that the traditional point cloud reconstruction method cannot effectively estimate the depth information of occluded branches; secondly, a point cloud density algorithm based on tree morphological feature analysis and a point cloud correction method based on point-by-point correction are proposed, which solve the problem that the tree branch point cloud initially obtained in the single-view depth estimation method is sparse and inaccurate.

[0011] The reconstruction of tree branches can usually be achieved through two methods: point cloud reconstruction based on a single view and point cloud reconstruction based on multiple views. The multi-view method can provide more comprehensive and accurate reconstruction results, but it requires a higher data acquisition cost and increases the complexity of data processing and calculation. Therefore, the present invention chooses to use point cloud reconstruction based on a single view to achieve tree branch reconstruction. The idea of ​​point cloud reconstruction based on a single view is to rely on a single RGB image to predict its corresponding depth image to achieve reconstruction. However, due to the limitation of the viewing angle during reconstruction, the same branch can only reconstruct the surface of one side, and some branches may also be blocked by other branches. The essential reason for the occlusion problem is that a single depth image cannot represent the three-dimensional structural information of the entire tree branch. In this case, the depth image predicted by the depth estimation network can only represent the depth information of the visible side of the corresponding position of the RGB image, but cannot represent the depth information of the occluded side. For the above problem, the present invention considers using multiple depth images to store depth information in layers, and then restores the three-dimensional structural information of the occluded part through a solution idea of ​​cyclic prediction. Specifically, the present invention builds a depth prediction network, uses the known RGB image and the previous depth image to predict the next depth image, and iterates until all depth images are predicted. This idea of ​​layered storage and cyclic prediction of depth can effectively solve the problem of poor reconstruction effect caused by occlusion.

[0012] The embodiment of the present invention adopts a strategy of layered cyclic prediction of depth information, which effectively solves the problem of poor prediction effect of the occluded part of the tree branches. Secondly, a recurrent pixel-to-pixel network (R-pix2pix) is proposed for cyclic prediction of tree branch depth information. Then, a normal and uniformly distributed point cloud dense algorithm (NUD) is proposed, which can achieve the densification effect on sparse tree point clouds. Finally, a point cloud correction network is proposed, which can correct the errors caused in the tree branch point cloud reconstruction process.

[0013] On the one hand, the present invention provides a method for reconstructing a single-view point cloud of a tree trunk in a real scene, comprising:

[0014] S1, obtain a single RGB image of tree branches in a real scene;

[0015] S2, performing single-view depth estimation based on cyclic prediction on the single RGB image of the tree branches to obtain multiple depth images;

[0016] S3, converting the multiple depth images into point clouds, so as to obtain a sparse point cloud;

[0017] S4, densifying the sparse point cloud to obtain a dense point cloud;

[0018] S5, performing fine correction on the point cloud coordinates of the dense point cloud to obtain a reconstruction result;

[0019] S6, reconstructing a single-view point cloud of tree branches in a real scene based on the reconstruction result.

[0020] Preferably, S2 includes:

[0021] S21, dividing the point cloud into grids based on a hierarchical storage strategy for point cloud three-dimensional information, and hierarchically storing the center of gravity information in the grids into multiple two-dimensional images;

[0022] S22, establishing a cyclic pixel-to-pixel conversion network, wherein the cyclic pixel-to-pixel conversion network is used to implement cyclic prediction, and when predicting each layer of depth image, the previous layer of depth image is used as a conditional input, so as to generate a depth image sequence layer by layer; each neuron in the cyclic pixel-to-pixel conversion network is composed of a generator and a discriminator, wherein each neuron adopts the basic structure of a conditional generative adversarial network, and a mechanism in which the generator and the discriminator compete with each other and evolve together;

[0023] S23, inputting the single RGB image of the tree branches into the cyclic pixel-to-pixel conversion network, and cyclically predicting the depth information of each layer of the tree branches.

[0024] Preferably, a pair of N mapping relationships is set in S21, that is, one RGB image corresponds to N depth images; during network training and testing, the next depth image is predicted using the known RGB image and the previous depth image, and it is iterated until all depth images are predicted, including the following steps:

[0025] Step a, meshing the depth axis of the three-dimensional tree; setting the depth axis as the y-axis, setting the number of grids to M, and the size of each grid being max(depth) / M, where max(depth) is the maximum value of the coordinates of the points on the y-axis;

[0026] Step b, calculating the relative center of gravity position of the points in each grid, and calculating the average relative center of gravity of all points in each grid by counting the point cloud data in each grid, so as to obtain the representative position of the points in each grid;

[0027] Step c: Encode the average relative center of gravity position information of the M grids into the RGB channel value of each pixel in the depth image; convert the point cloud data in the three-dimensional space into the pixel value of the two-dimensional image, and obtain a total of N depth images.

[0028] Preferably, the specific structure of the cyclic pixel-to-pixel conversion network is as follows:

[0029] (1) Input, output and training process of the cyclic pixel-to-pixel conversion network:

[0030] The recurrent pixel-to-pixel conversion network receives a single RGB image X RGB As input, the depth image of the next layer is output; during the training process, the depth image of the first layer is predicted by the pix2pix_0 network Then, X RGB and As a conditional input, use the pix2pix_1 network to predict the depth image of the second layer This iterative cycle continues until the depth image of the N-1th layer is predicted. Thus, the depth images of all layers are predicted;

[0031] (2) Each neuron in the cyclic pixel-to-pixel conversion network consists of two parts: a generator and a discriminator. The generator uses a U-Net structure to convert the input RGB image into a corresponding depth image. The discriminator is used to determine the authenticity of the generated image. Through a series of convolutional and pooling layer operations, the features of the input image are extracted. During the downsampling process, the size of the feature matrix gradually decreases, and the number of channels gradually increases, and finally a 1*1 feature vector is obtained. The feature vector is converted into a score representing the authenticity of the image through a sigmoid function, which is used to optimize the adversarial loss between the generator and the discriminator. The Huber loss function is introduced in the generator.

[0032] Preferably, the S4 includes: based on the tree morphological feature analysis, the sparse point cloud is densified to obtain a dense point cloud, including:

[0033] S41, establishing a NUD algorithm, wherein the NUD algorithm classifies sparse point clouds according to the morphological characteristics of tree branches, and uses corresponding densification strategies and densification parameters for sparse point clouds of tree branches of different categories; wherein the densification strategies include normal and uniform distribution; and the densification parameters include one or more of the number of points in the normal distribution, the mean and the variance, the number of points in the uniform distribution, and the upper and lower bounds of the distribution;

[0034] S42, densifying the sparse point cloud based on the NUD algorithm to obtain a dense point cloud.

[0035] Preferably, the S42 includes:

[0036] (1) The NUD algorithm classifies sparse point clouds according to the morphological characteristics of tree branches. Specifically, when traversing to a certain point, the algorithm will determine whether there are other points in the front and back grids of the point; according to the existence of points in the front and back grids, the current sparse point cloud can be divided into 9 categories, including:

[0037] Category ①: There are no points in the front and back grids;

[0038] Category ②: The point exists in the previous grid and is located close to the first 1 / 2 of the grid, but does not exist in the next grid;

[0039] Category ③: The point exists in the previous grid and is located close to the back 1 / 2 of the grid, but does not exist in the next grid;

[0040] Category ④: The point does not exist in the previous grid, but exists in the next grid, and its position is close to the first 1 / 2 of the grid;

[0041] Category ⑤: The point does not exist in the previous grid, but exists in the next grid, and its position is close to the back 1 / 2 of the grid;

[0042] Category ⑥: There are points in both the front and back grids, and the position of the point in the front grid is close to the front 1 / 2 of the grid, and the position of the point in the back grid is close to the front 1 / 2 of the grid;

[0043] Category ⑦: There are points in both the front and back grids, and the position of the point in the front grid is close to the front 1 / 2 of the grid, and the position of the point in the back grid is close to the back 1 / 2 of the grid:

[0044] Category ⑧: There are points in both the front and back grids, and the point in the front grid is close to the back 1 / 2 of the grid, and the point in the back grid is close to the front 1 / 2 of the grid;

[0045] Category ⑨: There are points in both the front and back grids, and the position of the point in the front grid is close to the back 1 / 2 of the grid, and the position of the point in the back grid is close to the back 1 / 2 of the grid;

[0046] (2) Based on the classified sparse point cloud, different densification strategies are adopted, including:

[0047] For categories ①, ②, ⑤, and ⑦, the densification method of normal distribution is adopted, where the mean of the normal distribution adopts the centroid value of the points in the current grid, and the variance and the number of generated points are dynamically adjusted according to the altitude of the point cloud;

[0048] For category ③, a uniformly distributed densification method is used. The lower bound of the uniform distribution is the centroid value of the point in the previous grid, and the upper bound is the centroid value of the point in the current grid. The number of generated points is also dynamically adjusted according to the altitude of the point cloud.

[0049] For category ④, the uniform distribution densification method is adopted, but the lower and upper bounds of the uniform distribution are adjusted to the centroid value of the points in the current grid and the centroid value of the points in the next grid respectively;

[0050] For category ⑥, a uniformly distributed densification method is used for the points in the current grid. The lower bound is the centroid value of the points in the current grid, and the upper bound is the centroid value of the points in the next grid. The number of generated points is still dynamically adjusted according to the altitude of the point cloud.

[0051] For category ⑧, a uniformly distributed densification method is used, with the lower bound being the starting position value of the current grid and the upper bound being the end position value of the current grid. The number of generated points is still dynamically adjusted according to the altitude of the point cloud.

[0052] For category ⑨, the densification method of normal distribution is adopted. The mean of the normal distribution adopts the centroid value of the points in the current grid, and the variance and the number of generated points are dynamically adjusted according to the altitude of the point cloud.

[0053] Preferably, S5 includes:

[0054] S51, establishing a point cloud correction network, which is composed of a convolutional layer, a BN layer, and a fully connected layer, takes the coordinates of the original rough point cloud as input, learns and adjusts the coordinate positions, and outputs the corrected coordinates at the corresponding positions, thereby making a more precise correction to the existing point cloud coordinates to reduce the cumulative error and improve the accuracy of the overall reconstruction;

[0055] S52, inputting the coordinates of the original rough point cloud into the point cloud correction network, and outputting the coordinates of the corresponding points at the corresponding positions after correction.

[0056] The second aspect of the present invention is to provide a single-view point cloud reconstruction system for tree branches in a real scene, which is used to implement the method of the first aspect, including:

[0057] Image acquisition module, which acquires a single RGB image of tree branches in real scenes;

[0058] A single-view depth estimation module based on cyclic prediction is used to obtain multiple depth images after performing single-view depth estimation based on cyclic prediction on the single RGB image of the tree branches;

[0059] A sparse point cloud conversion module, used for converting the multiple depth images from depth images to point clouds, thereby obtaining a sparse point cloud;

[0060] A point cloud densification module based on tree morphological feature analysis is used to densify the sparse point cloud to obtain a dense point cloud;

[0061] A point cloud correction module based on point-by-point correction is used to perform fine correction on the point cloud coordinates of the dense point cloud to obtain a reconstruction result;

[0062] The point cloud reconstruction module is used to reconstruct a single-view point cloud of tree branches in a real scene based on the reconstruction results.

[0063] The present invention has the following beneficial technical effects:

[0064] In order to verify the effectiveness and practicality of the single-view point cloud reconstruction method proposed in the present invention, the present invention is compared with the traditional single-view point cloud reconstruction method and the multi-view point cloud reconstruction method. The experimental results show that the single-view point cloud reconstruction method proposed in the present invention shows good performance, and is superior to the traditional single-view point cloud reconstruction method in terms of reconstruction effect, and can effectively solve the problem that the occluded area cannot be effectively reconstructed in the traditional single-view point cloud reconstruction method; compared with the multi-view reconstruction method, the single-view point cloud reconstruction method proposed in the present invention shows significant advantages in the reconstruction effect of branch details; the single-view point cloud reconstruction technology proposed in the present invention brings innovative ideas and practical methods to the research in the field of tree branch reconstruction, specifically:

[0065] (1) A layered and cyclic prediction strategy for depth information was adopted to effectively solve the problem of poor prediction of parts blocked by tree branches.

[0066] (2) The proposed recurrent pixel-to-pixel network (R-pix2pix) is used to cyclically predict the depth information of tree branches, aiming to solve the problem that traditional depth estimation networks cannot effectively estimate the depth information of occluded areas. The network combines the hierarchical storage strategy of point cloud three-dimensional information, first storing the three-dimensional information of tree branches in multiple two-dimensional images, and then cyclically predicting the depth information layer by layer through the improved pixel-to-pixel conversion network. Through the layer-by-layer prediction processing method, the network can more accurately capture the structural information of the occluded area, improving the accuracy of depth estimation.

[0067] (3) The Normal and Uniformly Distributed point cloud dense algorithm (NUD) can achieve the densification effect on sparse tree point clouds, aiming to solve the problem of sparse tree branch point clouds synthesized after the depth image is predicted by the single-view depth estimation method. The algorithm analyzes the morphological characteristics of tree branches and densifies the sparse point clouds using the normal and uniform distribution generation method.

[0068] (4) The point cloud correction network can correct the errors caused in the single-view point cloud reconstruction process of tree branches, aiming to correct the errors caused by the R-pix2pix network and the NUD algorithm. The present invention adopts a hierarchical cyclic prediction method when performing depth prediction. This method causes the error to gradually accumulate in the network layer, making the final depth estimation result inaccurate. In addition, when the point cloud is densified, due to the complex structure of the tree branches, the densified tree details are not fine enough. In order to reduce the errors caused by depth estimation and densification, the present invention proposes a point cloud correction network. The network inputs the coordinates of the point cloud and outputs the corrected coordinates at the corresponding position. The role of the correction network is to make more precise corrections to the existing point cloud coordinates to reduce the accumulated errors and improve the accuracy of the overall reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flow chart of the method for reconstructing a single-view point cloud of a tree trunk in a real scene according to the present invention;

[0070] Figure 2 The schematic diagram of the method for single-view point cloud reconstruction of tree branches in a real scene described in the present invention; wherein the three gray dashed boxes are one of the innovative points of the present invention, the present invention proposes a point cloud three-dimensional information hierarchical storage strategy and an R-pix2pix network in a single-view depth estimation module based on cyclic prediction to solve the problem of poor reconstruction of occluded parts in single-view point cloud reconstruction of tree branches; a NUD algorithm is proposed in a point cloud densification module based on tree morphological feature analysis to achieve a densification effect of sparse point clouds; a point cloud correction network is proposed in a point cloud optimization module based on point-by-point correction to achieve refined correction of point cloud coordinates;

[0071] Figure 3 This is a schematic diagram of the hierarchical storage of the three-dimensional information of tree branches as multiple depth images according to the present invention, Figure 3 The middle left column shows the overall display effect of the 3D point cloud, the upper right dotted box shows the details of the points in the tree branch point cloud in the y-axis direction, and the lower right dotted box shows the final 8 depth images. They represent 8 depth images obtained by layering;

[0072] Figure 4 This is the R-pix2pix network architecture diagram of the present invention. Figure 4 Medium X RGB represents the RGB image of tree branches input to the R-pix2pix network, Represents the depth image of the tth layer, t∈{0, 1, …, 7}, G(x) represents the depth map generated by the generator. The upper dotted box in the figure represents the overall architecture of the R-pix2pix network, where 8 depth maps are cyclically predicted layer by layer through 8 neurons composed of pix2pix. The lower dotted box represents the specific architecture of each neuron. The upper area in the figure is the innovation of the R-pix2pix network. This embodiment adopts the idea of ​​cyclic prediction to predict the depth image layer by layer, and also introduces the Huber loss function, which significantly improves the prediction effect of the network;

[0073] Figure 5 This is a diagram of the point cloud correction network architecture of the present invention;

[0074] Figure 6 This is a diagram of the architecture of the single-view point cloud reconstruction system for tree branches in a real scene according to the present invention;

[0075] Figure 7 It is a schematic diagram for visual comparison of depth image prediction performance under MAE, MSE and Huber loss function introduced in the present invention;

[0076] Figure 8 It is a schematic diagram of quantitative comparison results of eight pix2pix networks under different loss functions under MAE, MSE and the Huber loss function introduced in the present invention under PSNR and SSIM indicators;

[0077] Fig. 9 It is a schematic diagram of the visual comparison results of the method of the present invention and the traditional single-view point cloud reconstruction in a real scene;

[0078] Fig.10 This is an example of an RGB image of a tree trunk taken from a bird's-eye view by a drone-mounted camera. The area indicated by the arrow in the image is the pole that stretches the trunk, which blocks the branches below.

[0079] Fig.11 A schematic diagram showing the visual comparison results of the reconstruction details of the occluded area between this method and the traditional single-view point cloud reconstruction;

[0080] Fig.12 A visualization diagram showing the comparison results between the method of the present invention and the multi-view reconstruction method in a real scene;

[0081] Fig.13 A comparison view of the results of multi-view reconstruction using different numbers of images and the single-view point cloud reconstruction technology proposed in the present invention;

[0082] Fig.14 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0083] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0084] Embodiment 1

[0085] like Figure 1-2 As shown, this embodiment provides a method for reconstructing a single-view point cloud of a tree trunk in a real scene, comprising:

[0086] S1, obtain a single RGB image of tree branches in a real scene;

[0087] In the single-view point cloud reconstruction method, the depth estimation network plays a vital role. It is responsible for establishing the mapping relationship between RGB images and depth images. This step directly determines the quality of reconstruction. However, due to the limitations of single-view point cloud reconstruction, a single RGB image can often only predict the corresponding single depth image. The limitation of a single depth image is that it cannot accurately characterize the depth information of the occluded areas in the tree branches, which leads to the inability to effectively reconstruct the three-dimensional structure of these occluded areas. This becomes an unavoidable problem when reconstructing tree branches using a single-view point cloud reconstruction method. In order to solve this problem, the present invention stores the three-dimensional structural information of the tree branches in multiple depth images in layers. Through this strategy, the present invention successfully overcomes the limitation that a single depth image cannot fully express the depth information of the occluded areas in the tree branches. Then the present invention adopts the idea of ​​cyclic prediction, using the known RGB image and the previous depth image to predict the next depth image, and iterates until all depth images are predicted. This idea of ​​cyclic prediction enables the present invention to effectively predict the three-dimensional structure of the occluded areas of the tree branches.

[0088] S2, performing single-view depth estimation based on cyclic prediction on the single RGB image of the tree branches to obtain multiple depth images;

[0089] S3, converting the multiple depth images into point clouds, so as to obtain a sparse point cloud;

[0090] S4, densifying the sparse point cloud to obtain a dense point cloud;

[0091] S5, performing fine correction on the point cloud coordinates of the dense point cloud to obtain a reconstruction result;

[0092] S6, reconstructing a single-view point cloud of tree branches in a real scene based on the reconstruction result.

[0093] As a preferred embodiment, the S2 includes:

[0094] S21, based on the hierarchical storage strategy of point cloud three-dimensional information, the point cloud is divided into grids, and the centroid information in the grids is hierarchically stored into multiple two-dimensional images, thereby solving the problem that the three-dimensional information of the occluded area cannot be stored in the two-dimensional image, resulting in the inability to perform depth prediction;

[0095] In traditional single-view depth estimation networks, network training usually adopts a one-to-one form, that is, an RGB image of a tree branch is input, and the network outputs a tree branch depth image corresponding to the RGB image. However, a single depth image cannot represent the three-dimensional structural information of the entire tree branch, because the three-dimensional information cannot be completely stored in a two-dimensional image. To solve this problem, this embodiment proposes a point cloud three-dimensional information hierarchical storage strategy, which stores the point cloud information in multiple two-dimensional images in layers.

[0096] In this preferred embodiment, a one-to-eight mapping relationship is set, that is, one RGB image corresponds to eight depth images. During network training and testing, the known RGB image and the previous depth image are used to predict the next depth image, and the iteration is performed until all depth images are predicted. This hierarchical storage strategy can effectively solve the problem of poor reconstruction effect caused by occlusion.

[0097] Figure 3 This is a schematic diagram of storing the three-dimensional information of tree branches in layers as eight depth images. The specific step S21 includes the following steps:

[0098] Step a, meshing the depth axis of the three-dimensional tree. In this embodiment, the depth axis is set as the y-axis, and the number of grids is set to 24, and the size of each grid is max(depth) / 3, where max(depth) is the maximum value of the coordinates of the points on the y-axis;

[0099] Step b, calculate the relative center of gravity position of the points in each grid, and calculate the average relative center of gravity of all points in each grid by counting the point cloud data in each grid, so as to obtain the representative position of the points in each grid.

[0100] This step can further simplify the data while retaining the main features of the point cloud data. In this embodiment, the centroid value in each grid is unified into the interval [0, 255] so as to be stored as a pixel value in the image.

[0101] Step c: Encode the average relative center of gravity position information of the 24 grids into the RGB channel value of each pixel in the depth image.

[0102] Through this method, the point cloud data of the three-dimensional space is converted into the pixel values ​​of the two-dimensional image, and a total of 8 depth images are obtained.

[0103] Through the above steps, this embodiment stores the main three-dimensional information of tree branches in multiple two-dimensional images in layers, solving the problem that the three-dimensional information of the occluded part cannot be stored in the two-dimensional image, which leads to the inability to perform depth prediction.

[0104] S22, establishing a recurrent pixel-to-pixel network (R-pix2pix network for short in this embodiment), wherein the recurrent pixel-to-pixel network is used to implement recurrent prediction, and when predicting each layer of depth image, the previous layer of depth image is used as a conditional input, thereby generating a depth image sequence layer by layer; each neuron in the recurrent pixel-to-pixel network is composed of a generator and a discriminator, wherein each neuron adopts the basic structure of a conditional generative adversarial network, and a mechanism in which the generator and the discriminator compete with each other and evolve together;

[0105] In step S21, this embodiment stores the point cloud information of tree branches into eight depth images to solve the problem that a single depth image in traditional single-view point cloud reconstruction cannot store the depth information of occluded areas. Through the layered storage strategy of point cloud three-dimensional information, this embodiment obtains training pairs of a single RGB image of tree branches and eight depth images. In traditional single-view point cloud reconstruction schemes, a single RGB image can only predict the corresponding single depth image and cannot complete a one-to-many prediction task. Therefore, this embodiment designs a new depth estimation network that can complete a one-to-many prediction task.

[0106] As a preferred embodiment, Figure 4 The figure shows the architecture diagram of the recurrent pixel-to-pixel network (R-pix2pix for short). The R-pix2pix network draws on the core concepts of recurrent neural networks and conditional generative adversarial networks, and introduces the idea of ​​cyclic iterative prediction into the depth estimation task for the prediction of multi-layer depth images. When predicting each layer of depth images, the previous layer of depth images is used as conditional input to generate a sequence of depth images layer by layer. This idea of ​​cyclic iterative prediction can effectively improve the prediction accuracy of depth images. In each neuron, the embodiment of the present invention draws on the basic structure of the conditional generative adversarial network, introduces a mechanism in which the generator and the discriminator compete and evolve together, thereby improving the prediction ability of depth images.

[0107] As a preferred implementation, the specific structure of the R-pix2pix network is as follows:

[0108] (1) Input, output and training process of R-pix2pix network:

[0109] R-pix2pix network receives a single RGB image X RGB As input, it outputs the depth image of the next layer.

[0110] During the training process, the depth image of the first layer is predicted by the pix2pix_0 network Then, X RGB and As a conditional input, use the pix2pix_1 network to predict the depth image of the second layer This iterative cycle continues until the seventh layer of depth image is predicted. This is how the depth images of all layers are predicted.

[0111] (2) Each neuron in the R-pix2pix network consists of two parts: the generator and the discriminator.

[0112] The generator uses a U-Net structure to convert the input RGB image into the corresponding depth image, while the discriminator is used to judge the authenticity of the generated image.

[0113] Specifically, in this embodiment, the generator receives two images of size 256*256*3 as input, and performs downsampling and upsampling operations through the U-Net structure. In the downsampling stage, the input image is first processed by a series of convolutional layers and pooling layers, and the size of the feature matrix gradually decreases, while the number of channels gradually increases. Specifically, starting from the initial 256*256*3 matrix, it is downsampled to 128*128*64, 64*64*128, 32*32*256, 16*16*512, 8*8*512 and 4*4*512 in sequence, until a feature matrix of size 2*2*512 is finally obtained. In this process, the network extracts high-dimensional features of the image layer by layer, providing necessary information for subsequent upsampling operations. In the upsampling stage, the generator uses a transposed convolution layer to amplify the feature matrix. First, the 2*2*512 feature matrix is ​​upsampled by transposed convolution, and then spliced ​​with the feature matrix of the same size in the previous layer. This transposed convolution plus feature concatenation method enables the network to fuse feature information of different dimensions, thereby restoring finer image details. Through layer-by-layer upsampling, the size of the feature matrix gradually increases, the number of channels gradually decreases, and finally a 256*256*3 depth image of the same size as the input image is obtained.

[0114] The discriminator is used to judge the authenticity of the depth image output by the generator. It receives two images of size 256*256*3 as input, including an RGB image and a depth image output by the generator or a real depth image. The discriminator extracts the features of the input image through a series of convolutional layers and pooling layers. During the downsampling process, the size of the feature matrix gradually decreases, the number of channels gradually increases, and finally a 1*1 feature vector is obtained. The feature vector is converted into a score representing the authenticity of the image through the sigmoid function, which is used to optimize the adversarial loss between the generator and the discriminator.

[0115] (3) Loss function: This embodiment introduces the Huber loss function in the generator, and the specific formula is shown in formula (1). The generator of the traditional Pix2Pix network uses MAE (Mean Absolute Error) and MSE (MeanSquared Error) loss functions to achieve the mapping effect of RGB images to depth images. Among them, the MAE loss function has the disadvantage of being sensitive to outliers, which makes it easy to be affected by outliers during the training process, resulting in the problem that the generated image is not fine enough. The MSE loss function is not sensitive to outliers, but it is easy to cause the problem of gradient explosion, and the network convergence may be unstable during the training process. In contrast, the Huber loss function combines the advantages of MAE and MSE loss functions, has good robustness to outliers, and performs well when the network approaches the optimal solution. It can more effectively balance the network's processing of outliers and accurate values. The present invention introduces the Huber loss function to solve the problems caused by the MAE and MSE loss functions of the traditional pix2pix network, so that the generation effect of the pix2pix network is more refined.

[0116]

[0117] In formula (1), y is the true value, f(x) is the predicted value output by the network; the hyperparameter δ determines how the network handles outliers. When the residual is greater than δ, the Huber loss uses L1 loss (i.e., linear loss), which is less sensitive to outliers; and when the residual is less than δ, L2 loss (i.e., square loss) is used for optimization. The hyperparameter δ in this embodiment is set to 0.5.

[0118] S23, inputting the single RGB image of the tree branches into the cyclic pixel-to-pixel conversion network, and cyclically predicting the depth information of each layer of the tree branches.

[0119] Through the processing method of cyclic prediction, the network can more accurately capture the structural information of the occluded area, and then restore the three-dimensional structural information of the tree branches.

[0120] As a preferred implementation, the S4 includes: densifying the sparse point cloud based on tree morphological feature analysis to obtain a dense point cloud.

[0121] As the skeleton and support of trees, tree branches have rich and diverse morphological characteristics, which not only reflect the growth habits and environmental adaptability of trees, but also are an important basis for the overall morphology of the crown. The shape of the trunk is usually upright and thick. The central trunk is the part extending upward from the trunk, reaching the top of the crown. It is usually slenderer than the trunk, providing support and balance for various parts of the crown. The main branches are large branches that grow directly from the central trunk, which constitute the main skeleton of the crown. The side branches are small branches growing on the main branches, which further subdivide the crown to form a more complex structure. However, because the tree branches may be blocked, the real scene environment is complex, and the accuracy of the acquisition equipment itself is affected, the collected tree branch point cloud is usually sparse. This sparsity is mainly manifested in the small number of points and the low point cloud density. This makes the branch shape and details not rich enough in the point cloud data, resulting in information loss. In addition, sparse branch point clouds often cannot accurately capture the fine features and details of the branches, such as the bifurcation and bending of the branches. This makes point cloud data limited in applications such as morphological analysis and 3D reconstruction. It also leads to incomplete overall spatial structure of trees, especially at the junction of branches and at the edge of the crown. This incompleteness affects the accurate understanding of tree morphology and structure.

[0122] Point cloud densification is a necessary and important task to address various problems that may arise due to the sparsity of the above-mentioned tree branch point cloud. Point cloud densification aims to increase the density of point cloud data through algorithms and technical means to more accurately reflect the morphology and structure of tree branches. Traditional point cloud densification algorithms mainly include methods based on spatial patch diffusion and methods based on contour constraints. These methods are usually based on certain spatial geometric rules and constraints, and densify existing sparse point cloud information through deduction and calculation. However, when dealing with complex scenes, such as point clouds of tree branches with complex morphological characteristics, these methods cannot fully consider their morphological characteristics and structural information, resulting in unsatisfactory densification effects.

[0123] Therefore, this embodiment proposes a point cloud densification algorithm NUD based on the analysis of tree morphological characteristics, and deeply analyzes the morphological characteristics of tree branches. By preprocessing and classifying the point cloud data, for different types of points, the algorithm will use different densification methods and parameters according to their location and density changes. This classification processing method enables the algorithm to better adapt to the complexity and diversity of tree branch morphology. This helps to improve the accuracy and reliability of the densification results, making the densified point cloud data closer to the actual tree branch morphology.

[0124] As a preferred implementation, the S4 includes:

[0125] S41, establish a NUD algorithm, the NUD algorithm classifies sparse point clouds according to the morphological characteristics of tree branches, and uses corresponding densification strategies (normal and uniform distribution) and densification parameters (the number of points in the normal distribution, the mean and variance, the number of points in the uniform distribution, the upper and lower bounds of the distribution, etc.) for different categories of tree branch sparse point clouds. By fully considering the morphological characteristics of tree branches, the NUD algorithm can achieve a dense effect of tree branch point clouds.

[0126] S42, densifying the sparse point cloud based on the NUD algorithm to obtain a dense point cloud.

[0127] In this embodiment, the S42 includes:

[0128] (1) The NUD algorithm classifies sparse point clouds according to the morphological characteristics of tree branches. Specifically, when traversing to a certain point, the algorithm determines whether there are other points in the grids before and after the point.

[0129] In this embodiment, according to the existence of points in the front and rear grids, the current sparse point cloud can be divided into the following 9 categories, which correspond to the 9 classification situations in Table 1 respectively:

[0130] Category ①: There are no points in the front and back grids;

[0131] Category ②: The point exists in the previous grid and is located close to the first 1 / 2 of the grid, but does not exist in the next grid;

[0132] Category ③: The point exists in the previous grid and is located close to the back 1 / 2 of the grid, but does not exist in the next grid;

[0133] Category ④: The point does not exist in the previous grid, but exists in the next grid, and its position is close to the first 1 / 2 of the grid;

[0134] Category ⑤: The point does not exist in the previous grid, but exists in the next grid, and its position is close to the back 1 / 2 of the grid;

[0135] Category ⑥: There are points in both the front and back grids, and the position of the point in the front grid is close to the front 1 / 2 of the grid, and the position of the point in the back grid is close to the front 1 / 2 of the grid;

[0136] Category ⑦: There are points in both the front and back grids, and the point in the front grid is close to the front 1 / 2 of the grid, and the point in the back grid is close to the back 1 / 2 of the grid;

[0137] Category ⑧: There are points in both the front and back grids, and the point in the front grid is close to the back 1 / 2 of the grid, and the point in the back grid is close to the front 1 / 2 of the grid;

[0138] Category ⑨: There are points in both the front and back grids, and the position of the point in the front grid is close to the back 1 / 2 of the grid, and the position of the point in the back grid is close to the back 1 / 2 of the grid.

[0139] (2) Based on the classified sparse point cloud, different densification strategies are adopted.

[0140] In this embodiment, the densification strategies corresponding to specific categories are shown in Table 1. The details are as follows:

[0141] For categories ①, ②, ⑤, and ⑦, these categories correspond to the situation in which the tree branch point cloud only exists in the current grid and does not exist in the adjacent grids. Therefore, the algorithm adopts the densification method of normal distribution. Among them, the mean of the normal distribution adopts the centroid value of the points in the current grid, while the variance and the number of generated points are dynamically adjusted according to the altitude of the point cloud.

[0142] For category ③, it corresponds to the case where the tree branch point cloud exists in the current grid and the previous adjacent grid, but not in the subsequent adjacent grid. In this case, the algorithm uses a uniformly distributed densification method. The lower bound of the uniform distribution is the centroid value of the point in the previous grid, and the upper bound is the centroid value of the point in the current grid. The number of generated points is also dynamically adjusted according to the altitude of the point cloud.

[0143] For category ④, it corresponds to the case where the tree branch point cloud exists in the current grid and the next adjacent grid, but not in the previous adjacent grid. The densification strategy is similar to category ③, but the lower and upper bounds of the uniform distribution are adjusted to the centroid value of the points in the current grid and the centroid value of the points in the next grid, respectively.

[0144] For category ⑥, the point cloud corresponding to the tree branches exists in the current grid, the front adjacent grid and the back adjacent grid. In this case, there are two branches, one in the front adjacent grid and the other in the current grid and the back adjacent grid. In this case, the algorithm uses a uniformly distributed densification method for the points in the current grid. The lower bound is the centroid value of the points in the current grid, and the upper bound is the centroid value of the points in the next grid. The number of generated points is still dynamically adjusted according to the altitude of the point cloud.

[0145] For category ⑧, the point cloud corresponding to the tree branches exists in the current grid, the previous adjacent grid, and the next adjacent grid. However, in this case, there is only one branch, which spans the previous adjacent grid, the current grid, and the next adjacent grid. In this case, a uniformly distributed densification method is adopted. The lower bound is the starting position value of the current grid and the end position value of the current grid. The number of generated points is still dynamically adjusted according to the altitude of the point cloud.

[0146] For category ⑨, the point cloud corresponding to the tree branches exists in the current grid, the previous adjacent grid, and the next adjacent grid. In this case, there are three branches, located in the previous adjacent grid, the current grid, and the next adjacent grid. In this case, the densification method of normal distribution is adopted. The mean of the normal distribution adopts the centroid value of the points in the current grid, and the variance and the number of generated points are dynamically adjusted according to the altitude of the point cloud.

[0147] Table 1 The specific classification of each point in the tree branch point cloud by the NUD algorithm and the corresponding densification strategy adopted

[0148]

[0149]

[0150] As a preferred implementation, S5 includes:

[0151] S51, establishing a point cloud correction network, which is composed of a convolutional layer, a BN layer, and a fully connected layer, takes the coordinates of the original rough point cloud as input, learns and adjusts the coordinate positions, and outputs the corrected coordinates at the corresponding positions, thereby making a more precise correction to the existing point cloud coordinates to reduce the cumulative error and improve the accuracy of the overall reconstruction;

[0152] S52, inputting the coordinates of the original rough point cloud into the point cloud correction network, and outputting the coordinates of the corresponding points at the corresponding positions after correction.

[0153] When the present invention uses the R-pix2pix network for depth estimation, due to the use of a layered and cyclic training method, the error gradually accumulates in the layer-by-layer cycle, resulting in the final depth estimation result being inaccurate. At the same time, in the process of point cloud densification, the tree branch point cloud obtained by the NUD algorithm is not fine enough. These two problems together affect the accuracy of the overall point cloud reconstruction, which is specifically manifested in the poor reconstruction effect of the tree branch point cloud details and the deviation of the point cloud coordinates of the tree branch details. Therefore, an effective method is needed to realize the correction of point cloud coordinates.

[0154] Traditional point cloud coordinate correction methods mainly use registration algorithms, which aim to adjust the coordinates of point clouds through mathematical transformation or model learning to eliminate deviations caused by acquisition equipment errors, environmental noise and other factors. The registration algorithm calculates the similarities and differences between different point clouds, and uses transformations such as rotation and translation to make the point cloud data achieve the best alignment in space, thereby correcting the coordinate deviation. However, since the point cloud data of tree branches has complex and changeable shapes and structures, the registration algorithm often finds it difficult to find accurate matching points when processing point cloud data with complex shapes, resulting in unsatisfactory correction effects. In addition, such methods can usually only adjust and optimize the point cloud coordinates as a whole, and it is difficult to make fine corrections to local details, thereby affecting the accuracy and effect of the correction. Therefore, although traditional point cloud coordinate correction methods can correct coordinate deviations to a certain extent, their effects when processing the details of tree branches still need to be improved. The present invention needs to explore more suitable methods and techniques to improve the accuracy and efficiency of point cloud coordinate correction.

[0155] In order to solve the above problems, the present invention proposes a new point cloud correction network. The core idea of ​​the network is to correct the existing point cloud coordinates point by point to reduce the cumulative error and improve the accuracy of the overall reconstruction.

[0156] Specifically, an embodiment of the present invention designs a deep learning network, which takes the coordinates of the original rough point cloud as input, and outputs the corrected coordinates at the corresponding position by learning and adjusting the coordinate positions. In this way, the tree branch point cloud obtained by the single-view point cloud reconstruction method can be finely corrected to correct errors and imperfections. The advantage of the point cloud correction network is that it can make fine adjustments to each point instead of uniformly transforming or optimizing the entire point cloud. This enables the network to better handle local details and complex structures, and improve the accuracy of point cloud reconstruction. The following is a detailed introduction to the network.

[0157] Figure 5The network model is shown in Figure 1. The network consists of convolutional layers, BN layers, and fully connected layers. The data vector of size 1*4 passes through Conv-32, Conv-64, Conv-128, and Conv-256 layers in sequence, with BN layers interspersed to prevent the risk of gradient disappearance. Then it passes through three fully connected layers, FC-512, FC-256, and FC-1, and finally outputs a value of size 1*1, which is the corrected y coordinate. The core part of the network consists of multiple convolutional layers, which gradually extract the deep features of the point cloud data in a progressive manner. Specifically, the data vector of size 1*4 first passes through the Conv-32 layer to initially capture the local features of the data. Subsequently, the data passes through Conv-64, Conv-128, and Conv-256 in sequence to learn more abstract and advanced feature representations. Between the convolutional layers, the network is interspersed with batch normalization (BN) layers. The main function of the BN layer is to solve the gradient vanishing problem that may occur during the training process by normalizing each batch of data, thereby accelerating the convergence speed of the network and improving the stability of the network. After a series of convolution and BN layer processing, the data is passed to the fully connected layer for further processing. The function of the fully connected layer is to globally integrate the features extracted previously and map them to the output space. In this network, three fully connected layers, FC-512, FC-256 and FC-1, are set up in sequence. Through layer-by-layer dimensionality reduction, the feature vector is gradually mapped to the final output space. After being processed by the fully connected layer, the network outputs a feature vector of size 256*4. Finally, this feature vector is output as a value of size 1*1, that is, the corrected y coordinate.

[0158] Embodiment 2

[0159] like Figure 6 As shown, this embodiment provides a single-view point cloud reconstruction system for tree branches in a real scene, which is used to implement the method of embodiment 1, including:

[0160] An image acquisition module 101 acquires a single RGB image of a tree branch in a real scene;

[0161] A single-view depth estimation module 102 based on cyclic prediction is used to obtain multiple depth images after performing single-view depth estimation based on cyclic prediction on the single RGB image of the tree branches;

[0162] A sparse point cloud conversion module 103, used to convert the multiple depth images from depth images to point clouds, so as to obtain a sparse point cloud;

[0163] A point cloud densification module 104 based on tree morphological feature analysis is used to densify the sparse point cloud to obtain a dense point cloud;

[0164] A point cloud correction module 105 based on point-by-point correction is used to perform fine correction on the point cloud coordinates of the dense point cloud to obtain a reconstruction result;

[0165] The point cloud reconstruction module 106 is used to reconstruct a single-view point cloud of tree branches in a real scene based on the reconstruction result.

[0166] Experimental results and analysis: The proposed method for single-view point cloud reconstruction of tree branches in real scenes is evaluated, including experimental results of R-pix2pix network effect evaluation and reconstruction result evaluation.

[0167] (I) Experimental environment and parameter settings

[0168] The computer equipment used in the experiments of the embodiments of the present invention has the following configuration: Intel(R) Core(TM) i7-12700F@2.1GHz, NVIDlA GeForce GTX 4090 graphics card, Python 3.8.0 (64-bit) and PyTorch 1.7.1.

[0169] In addition, for the single-view depth estimation method based on cyclic prediction, this embodiment sets the batch_size to 32 to balance training efficiency and resource consumption. At the same time, in order to meet the training requirements of image input network structures with different characteristics, this embodiment sets the learning rate of the discriminator to 1e-6 and the learning rate of the generator to 1e-5. Such a learning rate configuration is intended to ensure the stability and convergence of the training process.

[0170] (II) Evaluation indicators

[0171] When evaluating the method of the present invention, this embodiment uses multiple evaluation indicators to ensure a comprehensive and objective measurement of its performance. These indicators cover the performance of the R-pix2pix network and the quality of the overall reconstruction results. The specific reasons for their selection and their advantages are as follows:

[0172] 1. First, for the performance evaluation of the R-pix2pix network, the present invention uses the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index (SSIM). PSNR is an important indicator for measuring image quality. It reflects the accuracy of reconstruction by comparing the pixel differences between the original image and the reconstructed image. The PSNR value ranges from [0, +∞). The higher the PSNR value, the smaller the distortion of the network in the process of correcting the point cloud, and the better the performance. SSIM focuses on the structural information of the image and can evaluate the network's ability to maintain the integrity of the image structure. The SSIM value ranges from [0, 1]. The closer the SSIM value is to 1, the better the network performance. By combining PSNR and SSIM, the present invention can comprehensively evaluate the performance of the R-pix2pix network in the task of predicting the depth of tree branches.

[0173] 2. Secondly, in order to comprehensively evaluate the overall reconstruction results, the embodiments of the present invention introduce Hausdorff distance and chamfer distance. Hausdorff distance is an important indicator to measure the similarity between two point sets, which can reflect the maximum mismatch between the reconstructed point cloud of tree branches and the real point cloud. By calculating the Hausdorff distance, the present invention can determine the deviation between the reconstructed point cloud and the real point cloud, thereby evaluating the accuracy of the tree branch reconstruction results. The chamfer distance focuses on the overall distribution of the tree branch reconstruction point cloud, and can measure the average distance between the point pairs of the reconstructed point cloud and the real point cloud. The smaller the chamfer distance, the closer the reconstruction result is to the real tree branch point cloud as a whole.

[0174] In addition to the above quantitative indicators, the present invention also visualizes the reconstruction results so as to more intuitively observe the differences in reconstruction results between the present method and other methods.

[0175] (III) Experimental results and analysis

[0176] 1. Performance of R-pix2pix network under different loss functions

[0177] In order to verify the performance of the depth estimation network proposed in the present invention, the embodiment of the present invention evaluates the effect of the R-pix2pix network in a real scene. The embodiment of the present invention experimentally tests the performance of the R-pix2pix network under different loss functions, and uses two common indicators, PSNR and SSIM, as well as visualization results to measure the performance of the network.

[0178] The experimental results are as follows Figure 7As shown in the figure. Each row in the figure represents the prediction effect of the depth image at different layers, and each column represents the prediction effect under different loss functions and the difference result with the true value. The cleaner the difference image is, the better the effect of the loss function in the depth estimation task is. The results show that the three loss functions have their own characteristics in the depth image prediction tasks at different layers. When close to the edge layer (i.e., the first and second layers and the seventh and eighth layers), the effects of MAE, MSE and Huber loss functions are relatively close, and there is no obvious difference between the advantages and disadvantages. However, when focusing on the depth image prediction close to the middle layer, the differences between the three begin to emerge. Specifically, the Huber loss function performs the best and can generate more detailed and accurate depth images; the MAE loss function and the MSE loss function are both poor, and their sensitivity to outliers causes the generated images to be not detailed enough in some areas. This is because the Huber loss function cleverly combines the advantages of MAE and MSE. It behaves as MSE when the error is small, which can better approach the optimal solution; when the error is large, it behaves as MAE, which has good robustness to outliers. This feature enables the Huber loss function to more effectively balance the network's processing of outliers and accurate values ​​in deep image prediction tasks, thereby obtaining more accurate and stable prediction results.

[0179] like Figure 8Shown are the quantitative comparison results of 8 pix2pix networks under different loss functions under MAE, MSE and the Huber loss function introduced by the present invention under PSNR and SSIM indicators. In the figure, the horizontal axis represents the pix2pix network of each layer, and the horizontal axis represents the values ​​of PSNR and SSIM. The experimental results show that the R-pix2pix network shows a trend of first decreasing and then increasing in both PSNR and SSIM indicators. The emergence of this trend can be attributed to the distribution characteristics of depth information at different levels in the depth image. There is less depth information stored in the depth image close to the edge layer, so the prediction task of the R-pix2pix network at these levels is relatively easy, thereby achieving higher performance. However, as the depth goes deeper into the middle layer, the depth information stored in the depth image gradually increases, the amount of information that the network needs to process increases, and the prediction task becomes more complex and difficult, resulting in a decrease in performance indicators. The comparison results of the three loss functions show that in terms of overall performance, the Huber loss function is generally better than MAE and MSE. When the performance of neurons at different levels was analyzed in depth, it was found that the MSE loss function performed better in neurons close to the edge layer. This is because depth images near the edge layer often contain less depth information and have more blank space, while the MSE loss function has stronger performance for images with more obvious edge features. MSE evaluates the loss by calculating the sum of the squares of the difference between the predicted value and the true value, and is more sensitive to subtle changes in the edge, so it can achieve better results in this case. However, in neurons close to the middle layer, the depth image stores more depth information and has less blank space. In this case, the MAE loss function performs better. MAE evaluates the loss by calculating the absolute value of the difference between the predicted value and the true value, and has stronger performance for images with rich foreground features. Since the middle layer contains more depth information, MAE can better capture these features, thereby achieving more accurate depth estimation.

[0180] 2. Comparison of reconstruction effects in occluded areas

[0181] In order to comprehensively evaluate the reconstruction effect of the occluded area of ​​the method proposed in the present invention, the embodiment of the present invention adopts a single-view point cloud reconstruction method as a comparison algorithm for in-depth comparative analysis. The comparison algorithm is a general single-view point cloud reconstruction method proposed by Hassner T et al., which realizes the depth prediction task of RGB images by constructing a mapping network from RGB images to depth images, and then reconstructs the three-dimensional structure of the object. The effectiveness of this method has been proved by experiments. Since this method abandons the complex parameterized network, it shows higher flexibility and can effectively adapt to objects of different shapes and structures. At the same time, this method shows high accuracy in depth estimation. Therefore, in order to fully verify the effectiveness of the single-view point cloud reconstruction method proposed in the present invention in the reconstruction of tree branches, the preferred embodiment of the present invention selects this method as a comparison algorithm, and conducts a comparative experiment with the method of the present invention to further highlight the advantages of the method of the present invention. Fig. 9 Comparison results with traditional single-view point cloud reconstruction.

[0182] like Fig. 9As shown, from the analysis of the visualization results, the embodiment of the present invention can clearly observe the difference between the traditional single-view point cloud reconstruction method and the method proposed in the present invention in reconstructing point clouds. Specifically, under the xoy perspective, the reconstruction results obtained by the traditional single-view point cloud reconstruction method and the method proposed in the present invention are not much different from the true value in morphology, and there are slight differences. In contrast, under the xoz and yoz perspectives, the point cloud reconstructed by the method proposed in the present invention is very close to the true value, with almost no significant difference, while the results obtained by the traditional single-view point cloud reconstruction method show obvious differences compared with the true value, and the structures of the main branches and secondary branches are somewhat different from the true value. There are two reasons for this result: First, it may be because the RGB image of the tree branches in the data is taken under the xoy perspective. Under the xoy perspective, the method of the embodiment of the present invention adopts a cyclic depth prediction strategy. Compared with the traditional single-view point cloud reconstruction that predicts all depth information at one time, the depth information obtained by the method of the embodiment of the present invention is more accurate. Second, it is because the similarity between the tree branches and the background environment in the real RGB image is too high. Due to this similarity, the traditional single-view point cloud reconstruction method has encountered great difficulties in distinguishing the foreground from the background, resulting in differences in some structures in the reconstruction results. This difference is particularly obvious under the xoy perspective, further highlighting the limitations of traditional methods in dealing with such complex scenes. For example, in the actual application process of distinguishing the foreground and background, taking the peach tree with a specific planting method as an example, a pole is usually set in the center of the tree, and the tree branches are stretched with ropes. This processing method is intended to ensure the growth form and stability of the tree. However, since the RGB image used in the present invention is taken by a camera carried by a drone, the perspective is mainly a bird's-eye view. From this perspective, the middle pole will inevitably block part of the tree's branches, such as Fig.10 shown.

[0183] Fig.11 The visual comparison results of the reconstruction effects of the point cloud reconstruction method proposed in the present invention and the traditional point cloud reconstruction method in the occluded area are shown in detail. The first column in the figure is the result of the traditional point cloud reconstruction comparison algorithm, the second column is the reconstruction result of this method, and the third column is the true value result. Each row represents the reconstruction result of the detailed part of the occluded area of ​​each algorithm. The black box is the comparison result of the reconstruction of the occluded area. When using the traditional single-view point cloud reconstruction method, the embodiment of the present invention found that the tree branches obscured by the poles could not be effectively reconstructed. This limitation is mainly due to the deficiencies of the traditional method in dealing with occlusion problems. In contrast, the single-view point cloud reconstruction method proposed in the present invention can overcome this problem. Fig.11It can be seen that even branches blocked by poles can be completely reconstructed by the method of the present invention. More importantly, the reconstructed branches are almost completely consistent with the branches at the corresponding positions in the true value, thus ensuring the high accuracy and authenticity of the reconstruction results. However, the traditional single-view point cloud reconstruction method cannot reconstruct the branches of trees in the blocked area. Fig.11 It can be seen that in the reconstruction results of the traditional single-view point cloud reconstruction method, the tree branches in the occluded area appear incomplete.

[0184] It can be seen from the visualization results that the single-view point cloud reconstruction method proposed in the present invention has significant advantages in dealing with occlusion problems and can more accurately restore the branch structure of the occluded part.

[0185] 3. Comparison of global reconstruction effects

[0186] Fig.12 A visualization diagram showing the comparison results between the multi-view point cloud reconstruction method and the single-view point cloud reconstruction method proposed in this invention. idis the identification code of the tree. This comparison aims to explore the differences in reconstruction effects of different methods. When performing multi-view reconstruction, the present invention selects 5, 100, and 200 RGB images as input data for reconstruction. These RGB images are randomly selected from all available images, reflecting the impact of different numbers of inputs on the reconstruction results. From the visualization results, the present invention can observe the following obvious trends. First, when 5 RGB images are used for multi-view reconstruction, the reconstructed point cloud is very sparse, and even the structure of the main branches has a large error compared with the true value. This shows that the small number of input images leads to insufficient reconstruction information. Secondly, as the number of input images increases, the reconstruction effect gradually improves. When 100 RGB images are used for reconstruction, the main branches of the point cloud gradually become clear, closer to the branch structure of the true value, but the overall point cloud is still relatively sparse. Finally, when the number of input images reaches 200, although the structure of the main branches is very clear, the overall point cloud still appears to be relatively sparse, and the secondary branches can hardly be effectively reconstructed. The reason for this phenomenon may be due to the characteristics of the multi-view reconstruction technology itself. In SfM-MVS, feature extraction is usually performed using algorithms such as SIFT. However, in the real scene used in the present invention, the foreground and background in the RGB image of the tree branches are too similar, which makes it difficult to effectively extract the features of the secondary branch area, thereby affecting the recovery of the secondary branch structure during the reconstruction process. In contrast, the single-view point cloud reconstruction method proposed in the present invention can overcome this problem. By using the image-to-point cloud angle registration method proposed in this section, even subtle areas such as secondary branches can capture their depth information features. Therefore, the reconstruction results of the method of the present invention are denser and more detailed, and can more accurately restore the detailed structure in the real scene.

[0187] Fig.13 The results of multi-view reconstruction using different numbers of images are compared with the single-view point cloud reconstruction technology proposed in the present invention. Through this comparison, the present invention aims to explore how many RGB images are equivalent to the single-view point cloud reconstruction technology proposed in the present invention in terms of effect.

[0188] In order to make a quantitative comparison, the present invention uses the Hausdorff distance as the evaluation indicator. The reason for choosing the Hausdorff distance instead of the chamfer distance is that the Hausdorff distance focuses on the maximum distance between the nearest points between two point sets, which can better reflect the similarity of the overall structure. Specifically, the Hausdorff distance is calculated by calculating the distance between each point in one point set and all points in another point set, then taking the minimum distance value, and then selecting the largest of these minimum distance values ​​as the final Hausdorff distance. The chamfer distance is a weighted distance metric that focuses more on measuring the local similarity between two point clouds or shapes. Considering that the goal of the present invention is to evaluate the performance of single-view point cloud reconstruction technology from the overall structure, the Hausdorff distance is selected as the evaluation indicator.

[0189] according to Fig.13 The experimental results show that the single-view point cloud reconstruction technology proposed in the present invention is equivalent to multi-view reconstruction using about 200 RGB images. This result shows that the single-view point cloud reconstruction technology proposed in the present invention has reached a very high level in performance and can effectively approach or even exceed the effect of multi-view reconstruction using a large number of images.

[0190] From the above visualization and quantitative comparison results, it can be seen that the single-view point cloud reconstruction technology proposed in the present invention has shown excellent performance. It is superior to the traditional single-view point cloud reconstruction method in terms of reconstruction effect, and can effectively solve the problem that occluded areas cannot be effectively reconstructed in traditional single-view point cloud reconstruction. Comparison with multi-view reconstruction shows that the single-view point cloud reconstruction technology proposed in the present invention has shown excellent performance. Specifically, its reconstruction effect is equivalent to multi-view reconstruction using about 200 images, showing excellent performance. The single-view point cloud reconstruction technology proposed in the present invention provides new ideas and methods for research in the field of tree branch reconstruction, and is expected to promote technological progress and application expansion in the future.

[0191] (IV) Conclusion

[0192] In order to fully verify the effectiveness and practicality of the single-view point cloud reconstruction method proposed in the present invention, the present invention has been experimentally verified on a real peach tree dataset. At the same time, the present invention has comprehensively compared the proposed method with traditional single-view point cloud reconstruction and multi-view reconstruction techniques. The experimental results show that the single-view point cloud reconstruction technology proposed in the present invention exhibits excellent performance. In terms of reconstruction effect, the method of the present invention is significantly better than the traditional single-view point cloud reconstruction method, especially when dealing with occluded areas, it can effectively solve the problem that traditional methods cannot effectively reconstruct. At the same time, compared with the multi-view reconstruction technology, the single-view point cloud reconstruction method of the present invention is equivalent to using about 200 images for multi-view reconstruction in terms of reconstruction effect, and is also better than the multi-view reconstruction technology in the reconstruction effect of the details of tree branches and trunks, which fully demonstrates its excellent performance and practicality. The single-view point cloud reconstruction technology proposed in the present invention provides new ideas and methods for research in the field of tree branch reconstruction, and is expected to promote technological progress and application expansion in the future.

[0193] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method described in the first embodiment.

[0194] like Fig.14 As shown, the present invention also provides an electronic device, including a processor 301 and a memory 302 connected to the processor 301, wherein the memory 302 stores a plurality of instructions, which can be loaded and executed by the processor so that the processor can execute the method described in Example 1.

[0195] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for reconstructing a single-view point cloud of tree branches in a real scene, characterized in that: include: S1, obtain a single RGB image of tree branches in a real scene; S2, performing single-view depth estimation based on cyclic prediction on the single RGB image of the tree branches to obtain multiple depth images; S3, converting the multiple depth images into point clouds, so as to obtain a sparse point cloud; S4, densifying the sparse point cloud to obtain a dense point cloud; S5, performing fine correction on the point cloud coordinates of the dense point cloud to obtain a reconstruction result; S6, reconstructing a single-view point cloud of tree branches in a real scene based on the reconstruction result.

2. According to the method for reconstructing a single-view point cloud of tree branches in a real scene in claim 1, it is characterized in that: The S2 includes: S21, dividing the point cloud into grids based on a hierarchical storage strategy for point cloud three-dimensional information, and hierarchically storing the center of gravity information in the grids into multiple two-dimensional images; S22, establishing a cyclic pixel-to-pixel conversion network, wherein the cyclic pixel-to-pixel conversion network is used to implement cyclic prediction, and when predicting each layer of depth image, the previous layer of depth image is used as a conditional input, so as to generate a depth image sequence layer by layer; each neuron in the cyclic pixel-to-pixel conversion network is composed of a generator and a discriminator, wherein each neuron adopts the basic structure of a conditional generative adversarial network, and a mechanism in which the generator and the discriminator compete with each other and evolve together; S23, inputting the single RGB image of the tree branches into the cyclic pixel-to-pixel conversion network, and cyclically predicting the depth information of each layer of the tree branches.

3. The method for reconstructing a single-view point cloud of tree branches in a real scene according to claim 2, characterized in that: In the S21, a mapping relationship of one pair to N is set, that is, one RGB image corresponds to N depth images; during the network training and testing process, the next depth image is predicted using the known RGB image and the previous depth image, and it is iterated until all the depth images are predicted, including the following steps: Step a, meshing the depth axis of the three-dimensional tree; setting the depth axis as the y-axis, setting the number of grids to M, and the size of each grid being max(depth) / M, where max(depth) is the maximum value of the coordinates of the points on the y-axis; Step b, calculating the relative center of gravity position of the points in each grid, and calculating the average relative center of gravity of all points in each grid by counting the point cloud data in each grid, so as to obtain the representative position of the points in each grid; Step c: Encode the average relative center of gravity position information of the M grids into the RGB channel value of each pixel in the depth image; convert the point cloud data in the three-dimensional space into the pixel value of the two-dimensional image, and obtain a total of N depth images.

4. The method for reconstructing a single-view point cloud of tree branches in a real scene according to claim 3, characterized in that: The specific structure of the cyclic pixel-to-pixel conversion network is as follows: (1) Input, output and training process of the cyclic pixel-to-pixel conversion network: The recurrent pixel-to-pixel conversion network receives a single RGB image X RGB As input, output the depth image of the next layer; During the training process, the depth image of the first layer is predicted by the pix2pix_O network Then, X RGB and As a conditional input, use the pix2pix_1 network to predict the depth image of the second layer This iterative cycle continues until the depth image of the N-1th layer is predicted. Thus, the depth images of all layers are predicted; (2) Each neuron in the cyclic pixel-to-pixel conversion network consists of two parts: a generator and a discriminator. The generator adopts a U-Net structure to convert the input RGB image into a corresponding depth image. The discriminator is used to judge the authenticity of the generated image. Through a series of convolutional layers and pooling layers, the features of the input image are extracted. During the downsampling process, the size of the feature matrix gradually decreases and the number of channels gradually increases. Finally, a 1*1 feature vector is obtained. The feature vector is converted into a score representing the authenticity of the image through a sigmoid function, which is used to optimize the adversarial loss between the generator and the discriminator. The Huber loss function is introduced in the generator.

5. The method for reconstructing a single-view point cloud of tree branches in a real scene according to claim 4, characterized in that: The S4 comprises: based on the tree morphological feature analysis, the sparse point cloud is densified to obtain a dense point cloud, including: S41, establishing a NUD algorithm, wherein the NUD algorithm classifies sparse point clouds according to the morphological characteristics of tree branches, and uses corresponding densification strategies and densification parameters for sparse point clouds of tree branches of different categories; wherein the densification strategies include normal and uniform distribution; and the densification parameters include one or more of the number, mean, and variance of points in the normal distribution, the number of points in the uniform distribution, and the upper and lower bounds of the uniform distribution: S42, densifying the sparse point cloud based on the NUD algorithm to obtain a dense point cloud.

6. The method for reconstructing a single-view point cloud of tree branches in a real scene according to claim 5, characterized in that: The S42 includes: (1) The NUD algorithm classifies sparse point clouds according to the morphological characteristics of tree branches. Specifically, when traversing to a certain point, the algorithm will determine whether there are other points in the front and back grids of the point; according to the existence of points in the front and back grids, the current sparse point cloud can be divided into 9 categories, including: Category ①: There are no points in the front and back grids; Category ②: The point exists in the previous grid and is located close to the first 1 / 2 of the grid, but does not exist in the next grid; Category ③: The point exists in the previous grid and is located close to the back 1 / 2 of the grid, but does not exist in the next grid; Category ④: The point does not exist in the previous grid, but exists in the next grid, and its position is close to the first 1 / 2 of the grid; Category ⑤: The point does not exist in the previous grid, but exists in the next grid, and its position is close to the back 1 / 2 of the grid; Category ⑥: There are points in both the front and back grids, and the position of the point in the front grid is close to the front 1 / 2 of the grid, and the position of the point in the back grid is close to the front 1 / 2 of the grid; Category ⑦: There are points in both the front and back grids, and the point in the front grid is close to the front 1 / 2 of the grid, and the point in the back grid is close to the back 1 / 2 of the grid; Category ⑧: There are points in both the front and back grids, and the point in the front grid is close to the back 1 / 2 of the grid, and the point in the back grid is close to the front 1 / 2 of the grid; Category ⑨: There are points in both the front and back grids, and the position of the point in the front grid is close to the back 1 / 2 of the grid, and the position of the point in the back grid is close to the back 1 / 2 of the grid; (2) Based on the classified sparse point cloud, different densification strategies are adopted, including: For categories ①, ②, ⑤, and ⑦, the densification method of normal distribution is adopted, where the mean of the normal distribution adopts the centroid value of the points in the current grid, and the variance and the number of generated points are dynamically adjusted according to the altitude of the point cloud; For category ③, a uniformly distributed densification method is used. The lower bound of the uniform distribution is the centroid value of the point in the previous grid, and the upper bound is the centroid value of the point in the current grid. The number of generated points is also dynamically adjusted according to the altitude of the point cloud. For category ④, the uniform distribution densification method is adopted, but the lower and upper bounds of the uniform distribution are adjusted to the centroid value of the points in the current grid and the centroid value of the points in the next grid respectively; For category ⑥, a uniformly distributed densification method is used for the points in the current grid. The lower bound is the centroid value of the points in the current grid, and the upper bound is the centroid value of the points in the next grid. The number of generated points is still dynamically adjusted according to the altitude of the point cloud. For category ⑧, a uniformly distributed densification method is used, with the lower bound being the starting position value of the current grid and the upper bound being the end position value of the current grid. The number of generated points is still dynamically adjusted according to the altitude of the point cloud. For category ⑨, the densification method of normal distribution is adopted. The mean of the normal distribution adopts the centroid value of the points in the current grid, and the variance and the number of generated points are dynamically adjusted according to the altitude of the point cloud.

7. The method for reconstructing a tree trunk single view point cloud in a real scene according to claim 6, characterized in that: The S5 includes: S51, establishing a point cloud correction network, which is composed of a convolutional layer, a BN layer and a fully connected layer, takes the coordinates of the original rough point cloud as input, learns and adjusts the coordinate positions, and outputs the corrected coordinates at the corresponding positions, thereby making a more precise correction to the existing point cloud coordinates to reduce the cumulative error and improve the accuracy of the overall reconstruction; S52, inputting the coordinates of the original rough point cloud into the point cloud correction network, and outputting the coordinates of the corresponding points at the corresponding positions after correction.

8. A single-view point cloud reconstruction system for tree branches in real scenes, used to implement the method according to any one of claims 1 to 7, characterized in that: include: An image acquisition module (101) acquires a single RGB image of tree branches in a real scene; A single-view depth estimation module (102) based on cyclic prediction is used to obtain multiple depth images after performing single-view depth estimation based on cyclic prediction on the single RGB image of the tree branches; A sparse point cloud conversion module (103), used for converting the plurality of depth images from depth images to point clouds, thereby obtaining a sparse point cloud; A point cloud densification module (104) based on tree morphological feature analysis is used to densify the sparse point cloud to obtain a dense point cloud; A point cloud correction module (105) based on point-by-point correction is used to perform fine correction on the point cloud coordinates of the dense point cloud, thereby obtaining a reconstruction result; The point cloud reconstruction module (106) is used to reconstruct the point cloud of tree branches in the real scene based on the reconstruction result.

9. An electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions, characterized in that: The processor is configured to read the instruction and execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a plurality of instructions, characterized in that: The plurality of instructions may be read by a processor and execute the method according to any one of claims 1 to 7.

Images