Farm point cloud three-dimensional reconstruction method based on LGSDF deep learning model

Through the improved LGSDF deep learning model, combined with global and local feature information, the problem of insufficient accuracy in farm three-dimensional reconstruction is solved, and high-precision farmland scene reconstruction is realized, supporting the intelligent and unmanned operation of agricultural equipment.

CN118918243APending Publication Date: 2024-11-08LONGMEN LAB
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410886839.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-03
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing three-dimensional reconstruction algorithm has limitations in farm scenarios and cannot be used for reconstruction of various land objects. In complex environments, parallax calculation errors are caused due to the nonlinearity of repeated textures and stereo matching, making it difficult to achieve high-precision farmland. Digital representation of the scene.

Method used

The three-dimensional reconstruction method of farm point cloud based on LGSDF deep learning model is adopted. By improving the DeepSDF model, local decoding operations are added, point cloud feature extraction is used using the PointNet network, and LGSDF deep learning model is constructed for training and optimization, combining global and local feature information. To improve reconstruction accuracy.

Benefits of technology

It realizes high-precision three-dimensional reconstruction in farmland scenarios, effectively removes noise, improves the reconstruction accuracy of land features, and performs excellently in local details reconstruction, supporting the intelligent and unmanned operations of agricultural equipment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918243A_ABST
    Figure CN118918243A_ABST
Patent Text Reader

Abstract

The invention provides a farm point cloud three-dimensional reconstruction method based on an LGSDF deep learning model, and the method comprises the following steps: collecting data, and obtaining point cloud data; performing statistical filtering and mean filtering preprocessing on the point cloud data to obtain a point cloud data set; an LGSDF deep learning model is constructed through an improved DeepSDF model; carrying out training optimization on the LGSDF deep learning model; and in combination with the point cloud data set, performing three-dimensional reconstruction by applying the trained LGSDF deep learning model. According to the implicit curved surface reconstruction method based on deep learning, the improvement of the information amount of data feature extraction is taken as an entry point, and the local context structure information is effectively utilized, so that the precision of farmland scene reconstruction by using the LGSDF model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of farm point cloud three-dimensional reconstruction, and specifically to a farm point cloud three-dimensional reconstruction method based on an LGSDF deep learning model. Background Art

[0002] Agriculture is the cornerstone of the national economy. With the rapid development of information technology, "unmanned farms" have emerged. This model provides a fully automated solution for farm production and management by integrating cutting-edge technologies such as the Internet of Things, big data, artificial intelligence, 5G, and robots. It realizes uninterrupted, full-space automated agricultural operations through remote control and autonomous decision-making of intelligent systems, effectively reducing the waste of rural land resources.

[0003] With the advancement of sensor technology and computer vision technology, three-dimensional scene visualization has developed rapidly. Three-dimensional reconstruction technology can convert the structural morphology and position information of the target object into a mathematical model that can be recognized by the computer, which is conducive to the machine to better understand the structural characteristics of the three-dimensional scene and realize efficient information interaction. Reconstructing farmland scene objects from sparse or partial point clouds is a basic requirement for the development of intelligent agriculture. It can determine the 3D structure of different objects in any outdoor scene, provide information such as obstacle location, field roads, and farmland object characteristics, which helps agricultural machinery to achieve accurate autonomous navigation and operation in field scenes. However, the existing three-dimensional reconstruction algorithms have their limitations in farm scene reconstruction: first, the existing three-dimensional reconstruction methods are mostly for the reconstruction of specific objects, and cannot be used for the reconstruction of various types of objects in farmland; second, in complex farmland environments, due to the existence of repeated textures in images and the nonlinearity of stereo matching, a single pixel often corresponds to multiple matching points, resulting in parallax calculation errors.

[0004] Therefore, how to adopt efficient reconstruction methods to achieve high-precision digital representation of farmland scenes has become a key issue in farmland environment perception. Summary of the invention

[0005] In order to solve the deficiencies in the prior art, the present invention provides a farm point cloud 3D reconstruction method based on the LGSDF deep learning model, improves the DeepSDF (a learnable signed continuous distance function SDF, implicit surface reconstruction method) network structure, proposes the LGSDF (Local Global Signed Distance Function) deep learning model, and uses the LGSDF model to reconstruct farmland scenes. The implicit surface reconstruction method based on deep learning of the present invention takes improving the amount of information extracted from data features as the starting point, and effectively utilizes local contextual structure information to improve the accuracy of model reconstruction.

[0006] In order to achieve the above object, the specific scheme adopted by the present invention is: A farm point cloud 3D reconstruction method based on the LGSDF deep learning model mainly includes the following steps: S1, collect data and obtain point cloud data; S2, performing statistical filtering and mean filtering preprocessing on the point cloud data to obtain a point cloud data set; S3. Construct the LGSDF deep learning model by improving the DeepSDF model. The LGSDF deep learning model uses the PointNet network to extract point cloud features and adds local decoding operations. The network extracts global features through three convolutional layers and a maximum pooling layer, and uses a fully connected layer to output feature vectors. The feature vector and the original point cloud are concatenated as the input of the global decoder, and processed using 8 consecutive 512-dimensional fully connected layers. Each fully connected layer uses the Relu activation function to output global features. The global features are fused with the features extracted by PointNet as the input of the local decoder to complete the connection between global and local feature information. The SDF value and the corresponding gradient vector are output after further processing through 8 512-dimensional fully connected layers. Finally, the global decoder adjusts the SDF value by the value output by the local decoder. S4, training and optimizing the LGSDF deep learning model; S5. Combined with the point cloud dataset, the trained LGSDF deep learning model is used for 3D reconstruction.

[0007] Furthermore, in step S1, the specific method for obtaining farm point cloud data is: S11. Use drones equipped with cameras and lidar; Aerial photography and photogrammetry using cameras to obtain high-resolution images; Perform laser scanning through LiDAR to obtain scanning data for 3D reconstruction and environmental perception; S12, input the high-resolution image, scanning data and base station data into the processing software Cropress equipped with AU20; S13. Calculate the UAV track by integrating photogrammetry, inertial navigation and measurement adjustment technology to eliminate the error and offset after the point cloud overlap matching; S14. Use the post-processing dynamic technology PPK GNSS to further calibrate the geographic coordinates and implement the synchronous alignment of the point cloud and the image to color the point cloud; S15. Merge data from multiple acquisition projects to obtain point cloud data.

[0008] Furthermore, in step S4, the farm is divided into 40 local areas, point cloud data of no less than 10 local areas are selected as training sets, and the training set is used to train and optimize the LGSDF deep learning model.

[0009] Furthermore, in step S4, during the training phase, the loss value loss is calculated by calculating the mean of the Euclidean distance between the predicted point cloud and the real point cloud, and the Adam optimizer is used to minimize the loss function to update the network. After several iterations, the network weight parameters are updated to select the optimal deep learning model parameters.

[0010] The present invention provides a farm point cloud 3D reconstruction method based on LGSDF deep learning model, which has the following features: Beneficial effects:

[0011] 1. The problem of lack of farm data sets is solved. The present invention uses drones equipped with lidar and cameras to take highly adaptable photos of farm scenes, formulates data collection strategies, determines the most effective scene point cloud data collection method, and realizes efficient and automated raw data acquisition, providing high-quality input for subsequent in-depth processing and analysis.

[0012] 2. Pre-processing techniques such as statistical filtering and mean filtering can effectively remove noise and outliers in point cloud data, thus improving data processing efficiency.

[0013] 3. Through the application of the LGSDF deep learning model, the present invention can achieve higher accuracy in the local 3D reconstruction of farm point cloud data. This method effectively utilizes the structural information of local context and global context, improves the reconstruction accuracy of ground features, and especially performs well in reproducing local details, which greatly improves the problems of low reconstruction accuracy and insufficient detail recovery of farm scenes.

[0014] 4. The reconstructed three-dimensional model of the farm scene can effectively guide agricultural equipment to carry out field operations, promote the development of intelligent and unmanned farms, and enable more abandoned farms to be reused. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is the overall distribution map of the farm research area of ​​the present invention.

[0016] Figure 2 This is the visualization result after point cloud filtering.

[0017] Figure 3 Encode shape map for DeepSDF.

[0018] Figure 4 This is the DeepSDF network architecture diagram.

[0019] Figure 5 This is the LGSDF network structure diagram in the present invention.

[0020] Figure 6 This is the reconstruction result of the rod-shaped object.

[0021] Figure 7 This is the resulting image of the building reconstruction.

[0022] Figure 8 This is the road reconstruction result map.

[0023] Fig. 9 This is the result of farmland reconstruction.

[0024] Fig.10 This is the tree reconstruction result map. DETAILED DESCRIPTION

[0025] The technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] The present invention provides a farm point cloud 3D reconstruction method based on the LGSDF deep learning model, and each step thereof is described in detail below.

[0027] S1. Data collection, obtaining farm point cloud data.

[0028] The step S1 includes the following contents:

[0029] S11. The collection area selected by the present invention includes information such as farmland, roads, artificial buildings, vegetation, etc., which can accurately reflect the point cloud data used in the real farm scene. The point cloud data includes RGB color information, XYZ coordinate information and reflection intensity information. The overall distribution of the area is as follows Figure 1 The collection area of ​​plot 1 is located in the X farmland base, the collection areas of plots 2, 3 and 4 are all located in the Y farmland base, the collection area of ​​plot 5 is located in the Z farmland base, and the collection areas of plots 6 and 7 are located in the M farmland base.

[0030] S12. The present invention uses a Bumblebee BB4 industrial-grade quad-rotor drone (Huace Navigation Technology Co., Ltd., Shanghai, China), equipped with an AU20 laser radar and a C5 orthophoto measurement system camera to obtain scanning data and high-resolution images. The AU20 laser radar measurement system includes an advanced laser ranging and positioning attitude system, which is known for its strong penetration, high precision and high efficiency, and is very suitable for high-quality point cloud data collection in multiple scenarios. The collection process of scanning data: First, the route is planned according to the satellite image of the collection area, the flight altitude is set to 120 meters, the side overlap of the point cloud is 50%, and the side overlap rate and heading overlap rate of the camera are 60% and 80% respectively. According to the altitude, width and aircraft endurance, the flight speed is controlled at 8m / s. The drone is equipped with a laser radar and a camera, and starts the collection task after an empty load test, and monitors the altitude, speed, satellite signal and power in real time until the data collection of a flight is completed. At the same time, the i83 multi-constellation GNSS system is always placed in a standardized farmland scene as a base station. The base station configuration needs to receive at least GPS and BDS observation data, and the total number of available satellites is not less than 18;

[0031] S13. After the drone aerial survey is completed, the high-resolution images, scanning data and base station data are input into the AU20 equipped processing software Cropress. Then, the drone track is calculated by integrating photogrammetry, inertial navigation and measurement adjustment technology to eliminate the errors and offsets after the point cloud overlap matching. The post-processing dynamic technology PPK GNSS is used to calibrate the geographic coordinates, implement the synchronous alignment of the point cloud and the image to color the point cloud, and finally, the point cloud data from multiple acquisition projects are merged to obtain the point cloud data and output the las point cloud format file of the scene.

[0032] S2. Perform statistical filtering and mean filtering preprocessing on the farm point cloud data to obtain a point cloud data set.

[0033] Mean filtering calculates the average coordinates of all points in the neighborhood around each point in the point cloud and replaces the coordinates of the original point with this average, thereby effectively removing noise and smoothing the point cloud data. The statistical filtering method calculates the average distance between each point in the point cloud and its neighborhood points. If the distance exceeds a preset threshold, the point is removed to achieve the purpose of removing noise. Specifically, the present invention pre-processes the collected point cloud data through statistical filtering and mean filtering to remove noise and outliers. Figure 2 The visualization results show that the filtered point cloud data expresses the morphological characteristics of the objects more clearly.

[0034] S3. Construct the LGSDF deep learning model by improving the DeepSDF model.

[0035] 3D reconstruction based on SDF function is a common method for generating continuous 3D models from discrete point cloud data or voxel data. The DeepSDF network obtains the latent vector by encoding the shape of the object and inputs the latent vector and the coordinates of the point into the function. The multi-layer fully connected neural network f is trained on the training set. θ The parameter θ is such that f θ It becomes a good approximator of the given SDF in the target domain, thereby achieving the fitting of the SDF function, as shown in the following formula (1). fθ(zi,x)≈SDFi(x)(1) In the above formula, i represents a certain shape, f θ is the latent vector z i and the query point coordinates x, which outputs an approximate SDF value for the shape.

[0036] The DeepSDF network uses a deep learning network to generate a continuous SDF representation of a certain type of object. Unlike traditional SDF methods, it can generate implicit vectors of various shapes, thereby achieving efficient encoding of object shapes. Figure 3 shown.

[0037] The network uses a self-decoding network structure to achieve SDF estimation without the need to train the encoder's decoder. At the beginning of training, the network first randomly generates an initial latent vector to represent the object shape, and continuously optimizes the latent vector through back-propagation training. After the iteration is completed, the modified shape latent vector and the densely sampled points in the boundary space are passed to the decoder to predict the spatial properties of all points. The DeepSDF network architecture is shown in the figure. Figure 4 As shown in the figure. Its decoder is a feedforward network composed of 8 fully connected layers (FC), each of which uses the Relu activation function and the Dropout layer. In the fourth hidden layer, a jump connection is used to accelerate the convergence speed of the loss function, and the last layer uses the Tanh activation function to obtain the SDF value of the coordinate point. The latent vector is obtained by the automatic decoder of DeepSDF, which contains the most important feature information in the original image, which facilitates better feature learning. When processing new geometric shape inputs, the encoder network can generate corresponding latent vectors by learning new data, showing strong generalization ability. In contrast, the self-decoder network performs poorly in this regard. In the generation stage of the network, the DeepSDF model cannot directly obtain the latent vector from the input shape, and some points need to be trained through the true value and loss function. In this process, the parameters of the decoder remain unchanged, so that the most suitable latent vector can be quickly found. Subsequently, the latent vector is input into the implicit neural network together with the query points in the entire space for complete shape prediction.

[0038] However, the DeepSDF network has not yet effectively integrated the global and local context information of point cloud data, which may lead to insufficient details of local reconstruction when processing complex agricultural scenes. Therefore, the present invention improves on this basis and proposes a LGSDF (Local Global Signed Distance Function) network structure.

[0039] like Figure 5 As shown in the figure, the LGSDF network uses the PointNet network for point cloud feature extraction and adds local decoding operations. The network extracts global features through three convolutional layers and a maximum pooling layer, and uses a fully connected layer to output a feature vector. The feature vector and the original point cloud are concatenated as the input of the global decoder and processed using 8 consecutive 512-dimensional fully connected layers, each of which uses a Relu activation function. The intermediate features are fused with the features extracted by PointNet as the input of the local decoder to complete the connection between global and local feature information. It is further processed through 8 fully connected layers (512 dimensions) to output the SDF value and the corresponding gradient vector. Finally, the global decoder adjusts the SDF value by the value output by the local decoder.

[0040] S4. Train and optimize the LGSDF deep learning model.

[0041] The farm is divided into 40 local areas, and point cloud data of no less than 10 local areas are selected as training sets, which are used to train and optimize the LGSDF deep learning model.

[0042] The present invention uses a loss function in the model training process, and the loss function is mainly applied to two parts: the local decoder and the global decoder. The loss function of the local decoder calculates the average value of the Euclidean distance between the target point cloud and the point cloud generated by the local decoder, so that the local decoder can generate a point cloud as close to the target point cloud as possible; the loss function of the global decoder calculates the average value of the Euclidean distance between the global target point cloud and the point cloud generated by the global decoder, fine-tunes the global decoder, and further optimizes the quality of the generated point cloud.

[0043] In the training phase, the loss value loss is calculated by calculating the mean of the Euclidean distance between the predicted point cloud and the real point cloud, and the Adam optimizer is used to minimize the loss function to update the network. After several iterations, the network weight parameters are updated and the optimal deep learning model parameters are selected. By deepening the network layer, it helps to improve the model's sensitivity to the input point cloud data and enhance the model's ability to capture local shape changes and details.

[0044] S5. Three-dimensional reconstruction.

[0045] Combined with the point cloud dataset, the trained LGSDF deep learning model is applied for 3D reconstruction.

[0046] The present invention uses an Inter(R) Core(TM) i7-9700 processor, an NVIDIA GeForce graphics card, and a computer with 16G video memory, and builds a CUDA 11.4, CUDNN11.0, and Tensorflow 2.5 environment under the Ubuntu 18.04 operating system and Python 3.6 programming language to conduct experiments. When performing three-dimensional model reconstruction training, the present invention sets the learning rate to 0.0001.

[0047] S6. Evaluation of reconstruction results.

[0048] In order to better evaluate the reconstruction effect of the LGSDF model, the present invention selects two classic reconstruction methods, AtlasNet method and Poisson reconstruction, to compare with the LGSDF model in the present invention. The selected reconstruction categories are common farm objects, such as poles, buildings, roads, farmland, and trees. The reconstruction results are shown in Figure 2. Figures 6 to 10 shown.

[0049] Depend on Figures 6 to 10 It can be seen that the AtlasNet method has weak generalization ability when dealing with various types of objects, especially when reconstructing objects with sharp edges and thin-walled structures, such as poles and trees, which are difficult to accurately restore the basic shape. When dealing with objects such as buildings and farmland with planar regular structures, the AtlasNet method often produces more redundant patches. This is mainly because the structure of the new data is very different from the training data, and this method cannot adapt well to the new data. The Poisson reconstruction method can effectively reconstruct the features of the object. However, there will be many holes inside the object, resulting in poor performance in reconstructing the surface feature details of the pole-like object. In contrast, the proposed LGSDF method has better reconstruction effects on various objects.

[0050] The present invention selects Chamfer distance, F2-score and normal consistency score as reconstruction evaluation indicators. Figures 6 to 10 Each type of object in was tested, and evaluation indicators such as Chamfer distance, F2-score and normal consistency were calculated, and the average value of the same category was taken as the reconstruction result. Among them, the smaller the Chamfer distance, the better, while the higher the F-score and normal consistency results, the better. The quantitative reconstruction results are shown in Table 1.

[0051] Table 1 Quantitative results based on different surface reconstruction algorithms

[0052] As shown in Table 1, the LGSDF method has the smallest Chamfer distance calculated in the reconstruction of poles and road objects, which are 0.227 and 0.325 respectively. The Poisson reconstruction method has the smallest Chamfer distance in the reconstruction of buildings, farmlands and trees (0.844, 0.058 and 0.182 respectively), which is because it is good at generating smooth planes and has good adaptability to the facades of buildings, the flat surfaces of farmlands and the continuous curved surfaces of trees. In contrast, the LGSDF method shows the highest F2-score and normal consistency in the reconstruction of different categories, reaching an average of 70% and 66%, indicating that it is superior in dealing with complex geometric structures and diverse scenes. In contrast, the LGSDF method proposed in the present invention has better reconstruction effects for various objects. This is mainly due to the fact that the LGSDF method based on local feature learning has a good effect in local reconstruction and can learn the geometric distribution in a large amount of data, thereby achieving accurate prediction. In the process of surface reconstruction, in addition to paying attention to performance evaluation indicators, it is also necessary to ensure that the reconstruction method can accurately restore the details of the geometric shape and produce a correct and smooth plane. Taking all indicators into consideration, LGSDF shows better reconstruction results compared with the other two methods.

[0053] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any equivalent changes or modifications made according to the essence of the present invention should be included in the protection scope of the present invention.

Claims

1. A farm point cloud 3D reconstruction method based on LGSDF deep learning model, characterized in that: The main steps are as follows: S1, collect data and obtain point cloud data; S2, performing statistical filtering and mean filtering preprocessing on the point cloud data to obtain a point cloud data set; S3. The LGSDF deep learning model is constructed by improving the DeepSDF model. The LGSDF deep learning model uses the PointNet network to extract point cloud features and adds local decoding operations. The network extracts global features through three convolutional layers and a maximum pooling layer, and uses a fully connected layer to output feature vectors. The feature vector and the original point cloud are concatenated as the input of the global decoder and processed using 8 consecutive 512-dimensional fully connected layers. Each fully connected layer uses the Relu activation function to output global features. The global features are fused with the features extracted by PointNet as the input of the local decoder to complete the connection between the global and local feature information. They are further processed through 8 512-dimensional fully connected layers to output the SDF value and the corresponding gradient vector. Finally, the global decoder adjusts the SDF value through the value output by the local decoder. S4, training and optimizing the LGSDF deep learning model; S5. Combined with the point cloud dataset, the trained LGSDF deep learning model is used for 3D reconstruction.

2. According to claim 1, a farm point cloud 3D reconstruction method based on LGSDF deep learning model is characterized in that: In step S1, the specific method for obtaining farm point cloud data is: S11. Use drones equipped with cameras and lidar; Aerial photography and photogrammetry using cameras to obtain high-resolution images; Perform laser scanning through LiDAR to obtain scanning data for 3D reconstruction and environmental perception; S12, input the high-resolution image, scanning data and base station data into the processing software Cropress equipped with AU20; S13, integrating photogrammetry, inertial navigation and survey adjustment technology to calculate the UAV track to eliminate the error and offset after point cloud overlap matching; S14. Use post-processing dynamic technology PPK GNSS to further calibrate the geographic coordinates and implement synchronous alignment of point cloud and image to color the point cloud; S15. Merge data from multiple acquisition projects to obtain point cloud data.

3. According to the method of claim 1, the farm point cloud 3D reconstruction method based on the LGSDF deep learning model is characterized in that: In step S4, the farm is divided into 40 local areas, and point cloud data of no less than 10 local areas are selected as training sets, and the training sets are used to train and optimize the LGSDF deep learning model.

4. The farm point cloud 3D reconstruction method based on the LGSDF deep learning model according to claim 1 is characterized in that: In step S4, during the training phase, the loss value loss is calculated by calculating the mean of the Euclidean distance between the predicted point cloud and the real point cloud, and the Adam optimizer is used to minimize the loss function to update the network. After several iterations, the network weight parameters are updated to select the optimal deep learning model parameters.

Citation Information

Cited By

  • Multi-mode integrated quick acquisition, analysis and calculation method for river sand excavation data

    CN119919599A

  • Complex spacecraft target geometric modeling method and device based on modular implicit neural representation

    CN121959756A

  • Point cloud curved surface implicit reconstruction method and device based on surface precision optimization strategy

    CN122089948A