Orchard three-dimensional reconstruction and fruit tree semantic segmentation method based on neural radiation field

Abstract

The invention relates to the technical field of agricultural informatization, in particular to an orchard three-dimensional reconstruction and fruit tree semantic segmentation method based on a neural radiation field, and the method comprises the following steps: carrying out the multi-view shooting of an orchard through an unmanned plane, and obtaining an original image data set for three-dimensional reconstruction; and preprocessing the original image data set, including camera pose estimation and distortion removal, image down-sampling, monocular depth estimation and semantic annotation, and generating image data of RGB images, depth maps and semantic maps suitable for neural radiation field network training. According to the method, under the condition of processing a large-scale outdoor environment, especially complex illumination conditions and variable terrains, through optimized unmanned aerial vehicle multi-view reconstruction, an improved NeRF algorithm and an introduced semantic field, the limitation of a traditional method can be overcome when the method faces a large-scale orchard scene, and the method is suitable for large-scale orchard scenes. And faster, finer and more accurate three-dimensional reconstruction is realized.

Description

Technical Field

[0001] The present invention relates to the field of agricultural information technology, and in particular to a method for orchard three-dimensional reconstruction and fruit tree semantic segmentation based on neural radiation fields. Background Art

[0002] In modern agriculture, 3D reconstruction technology is widely used to monitor crop growth, analyze environmental conditions, and predict yields. As an important innovative means, 3D reconstruction effectively promotes refined management and monitoring by generating high-precision three-dimensional models. Combined with virtual reality (VR) technology, complex unstructured agricultural scenes can be reconstructed in real time, thereby providing remote assistance support for agricultural robots to perform various complex tasks in orchards or farmlands.

[0003] In orchards, 3D reconstruction relies on visual data such as images and point clouds to create digital models that accurately depict trees, fruits, and terrain, which helps to present the spatial structure and distribution characteristics of the environment, thereby supporting crop phenotyping and yield estimation. In addition, by integrating semantic information into 3D scenes, we can better understand the distribution and status of various objects in the orchard, such as helping robots to automatically pick fruits or using drones for precise pesticide spraying, thereby promoting agricultural automation.

[0004] Drones are widely used in crop monitoring due to their low cost, high resolution, flexible operation and high degree of automation. Combined with 3D reconstruction technology, drones can quickly and accurately obtain 3D data of large orchards, providing unique technical support for the refined management, planning and design of orchards, and the assessment of the health status of fruit trees.

[0005] At present, point cloud-based 3D reconstruction technology has been widely used in the agricultural field, but when dealing with the complex structural details of plants, it is often limited by sensor performance and data acquisition conditions. The emergence of Neural Radiance Field (NeRF) technology has brought new breakthroughs in high-fidelity 3D reconstruction. NeRF technology uses deep learning methods to generate high-precision 3D models from 2D images, greatly improving the accuracy and detail of reconstruction.

[0006] However, in actual use, NeRF technology is currently mainly used in small-scale agricultural scenarios. When drones are used for large-scale 3D reconstruction of orchards, NeRF technology still faces technical bottlenecks such as reduced reconstruction quality under complex lighting conditions, slower speed, and high requirements for data collection. Summary of the invention

[0007] The purpose of the present invention is to provide a method for orchard 3D reconstruction and fruit tree semantic segmentation based on neural radiation field, so as to solve the problems of reduced reconstruction quality, slow speed and high data acquisition requirements under complex lighting conditions.

[0008] To achieve the above object, the present invention provides the following technical solution: a method for orchard 3D reconstruction and fruit tree semantic segmentation based on neural radiation field, comprising the following steps:

[0009] S1. Image acquisition: Use drones to take multi-view photos of the orchard to obtain the original image dataset for 3D reconstruction;

[0010] S2. Preprocessing: Preprocessing the original image data set, including camera pose estimation and dedistortion, image downsampling, monocular depth estimation and semantic annotation, to generate image data of RGB images, depth maps and semantic maps suitable for neural radiation field network training;

[0011] S3. Neural radiation field network construction and training:

[0012] S31. Based on the original neural radiation field network model, the input sampling point coordinate shrinkage module, multi-resolution hash coding and compact MLP architecture, appearance embedding vector, transparency estimation network and semantic MLP module are integrated to construct an improved neural radiation field network model;

[0013] S32, inputting the image data set preprocessed in step S2 into the improved neural radiation field network model in step S31 for training, generating a three-dimensional reconstruction and semantic segmentation of the orchard scene;

[0014] S4. Loss function design: using a composite loss function including reconstruction loss, depth loss, semantic loss, visibility loss, distortion loss and inter-layer loss to train the improved neural radiation field network model in step S31, so as to further optimize the effects of 3D reconstruction and semantic segmentation;

[0015] S5, performance evaluation: using peak signal-to-noise ratio, structural similarity index, perceived image similarity, rendering frames per second, intersection-over-union ratio, average intersection-over-union ratio, single-category pixel accuracy and total pixel accuracy indicators to perform performance evaluation on the 3D reconstruction and semantic segmentation results in step S32;

[0016] S6. 3D visualization and rendering: Apply the improved neural radiation field network model trained in step S4 to the 3D visualization tool to achieve rendering of the orchard scene and visualization of the segmentation results.

[0017] Preferably, in step S31:

[0018] The input sampling point coordinate shrinkage module is used to reduce the computational burden and improve the modeling accuracy;

[0019] The multi-resolution hash coding and compact MLP architecture are used to capture global and local information of the scene;

[0020] The appearance embedding vector is used to encode the appearance information of the image;

[0021] The transparency estimation network is used to optimize the transparency estimation of the model;

[0022] The semantic MLP module is used to predict the semantic label of each point in the scene.

[0023] Preferably, the composite loss function in step S4 is specifically:

[0024] The reconstruction loss is used to measure the difference between the generated image and the real image, based on the color rendering loss;

[0025] The depth loss uses the depth map of monocular depth estimation as a priori and is calculated using mean square error;

[0026] The semantic loss measures the difference between the predicted semantic label and the true label through the cross entropy loss;

[0027] The visibility loss removes floating objects in the scene and optimizes the scene geometry by penalizing high density values ​​in invisible areas in the training view.

[0028] The distortion loss is used to penalize the weight distribution of distant points, ensuring that the weights are concentrated at closer locations, ensuring that the density estimation is more accurate and localized;

[0029] The inter-layer loss is used to supervise the voxel density distribution at different levels to ensure the consistency of the prediction results of the transparency estimation network and the NeRF network.

[0030] Preferably, the performance evaluation index in step S5 is specifically:

[0031] The peak signal-to-noise ratio is used to measure the similarity between the generated image and the real image;

[0032] The structural similarity index is used to evaluate the fidelity of image structural information and reflects the structural similarity between the reconstructed image and the reference image;

[0033] The perceived image similarity: calculating the perceived distance of the images through a deep learning network;

[0034] The rendering frames per second is used to measure the rendering efficiency of the three-dimensional scene;

[0035] Intersection-over-Union: Evaluates the degree of overlap between the segmented area predicted by the model and the true area.

[0036] Average Intersection over Union (IoU): The IoU of all categories is averaged to measure the accuracy of the overall semantic segmentation.

[0037] Single category pixel accuracy: calculate the correct pixel ratio for each category and evaluate the accuracy of segmentation;

[0038] Total pixel accuracy: Calculates the proportion of pixels predicted correctly among all pixels, reflecting the overall segmentation performance of the model on the entire image.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. When dealing with large-scale outdoor environments, especially those with complex lighting conditions and changeable terrain, the present invention overcomes the limitations of traditional methods and achieves faster, more precise and accurate three-dimensional reconstruction when facing large-scale orchard scenes through optimized drone multi-perspective reconstruction, improved NeRF algorithm and introduced semantic fields. Multi-resolution hash coding technology is used to achieve efficient scene representation. By fusing multi-perspective color images and semantic images, the joint representation of texture, geometric information and semantic information is achieved, providing support for accurate semantic segmentation of orchards. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the overall structure of the orchard 3D reconstruction and fruit tree semantic segmentation method based on neural radiation field of the present invention;

[0042] Figure 2 It is a flowchart of the preprocessing steps of the method for three-dimensional reconstruction of an orchard and semantic segmentation of fruit trees based on neural radiation fields of the present invention;

[0043] Figure 3 It is a flowchart of the steps of constructing and training the neural radiation field network of the orchard 3D reconstruction and fruit tree semantic segmentation method based on the neural radiation field of the present invention;

[0044] Figure 4 A flowchart of a sampling point coordinate shrinkage module for inputting the orchard three-dimensional reconstruction and fruit tree semantic segmentation method based on neural radiation field of the present invention;

[0045] Figure 5 It is a flowchart of multi-resolution hash coding and compact MLP architecture of the orchard 3D reconstruction and fruit tree semantic segmentation method based on neural radiation field of the present invention;

[0046] Figure 6 It is a flow chart of the transparency estimation network of the orchard 3D reconstruction and fruit tree semantic segmentation method based on neural radiation field of the present invention;

[0047] Figure 7 It is a schematic diagram of the actual operation system of the method for orchard 3D reconstruction and fruit tree semantic segmentation based on neural radiation field of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Point cloud-based 3D reconstruction technology has been widely used in the agricultural field. For example, Yali Li used improved structured light technology (SfM) to reconstruct 3D point clouds of jujube trees, achieving accurate measurement of the number of branches and crown diameter; Jinyong Wu constructed dense point clouds by taking photos of cotton with drones, which enabled dynamic monitoring of cotton canopy parameters, including plant height and leaf area index; Yuchao Li took multi-view image sequences of corn seedlings and reconstructed point clouds based on the SfM algorithm, successfully segmenting corn stems and leaves and measuring related characterization parameters. These studies based on point cloud technology have demonstrated their wide application potential in the agricultural field, but they also have many limitations, especially when dealing with complex structural details of plants, which are often limited by sensor performance and data acquisition conditions.

[0050] The emergence of Neural Radiance Field (NeRF) technology has brought new breakthroughs in high-fidelity 3D reconstruction. NeRF technology generates high-precision 3D models from 2D images through deep learning methods, greatly improving the accuracy and detail of reconstruction. Jing Zhang et al. improved NeRF's scene reconstruction and rendering methods, and achieved realistic rendering effects for scenes of different scales in strawberry planting sheds. Xiajun Zheng proposed the "Tomato-NeRF" technology, which can generate high-resolution 3D models from tomato images taken by mobile phones. However, NeRF technology is currently mainly used in small-scale agricultural scenes. When drones are used for large-scale 3D reconstruction of orchards, NeRF technology still faces technical bottlenecks such as illumination changes, reduced reconstruction quality, slow speed, and high requirements for data collection.

[0051] The present invention provides a technical solution: a method for three-dimensional reconstruction of an orchard and semantic segmentation of fruit trees based on a neural radiation field, comprising the following steps:

[0052] S1. Image acquisition: Use high-resolution drones to take multi-view photos of the orchard to obtain the original image dataset for 3D reconstruction. This image dataset contains rich perspective information and provides basic data for subsequent 3D model construction.

[0053] S2. Preprocessing: Preprocessing the original image dataset, including camera pose estimation and dedistortion, image downsampling, monocular depth estimation and semantic annotation, to generate RGB images, depth maps and semantic maps suitable for neural radiation field network training.

[0054] S3. Neural radiation field network construction and training:

[0055] S31. Based on the original neural radiation field network model, the input sampling point coordinate shrinkage module, multi-resolution hash coding and compact MLP architecture, appearance embedding vector, transparency estimation network and semantic MLP module are integrated to construct an improved neural radiation field network model. The specific optimizations are as follows:

[0056] Input sampling point coordinate shrinkage module: By converting the coordinates of the input scene from Euclidean space to restricted space, the distant points are distributed in inverse proportion to the distance. Combined with the linear parallax sampling strategy, the sampling intervals in the distant area are denser, while the sampling intervals in the close distance are relatively sparse, thereby reducing the computational burden and improving the modeling accuracy of the distant area;

[0057] Multi-resolution hash coding and compact MLP architecture: Multi-resolution hash coding is used to encode the input coordinates at multiple scales to effectively capture the global and local information of the scene. The coarser resolution level is used to obtain the global structure, and the finer resolution level captures the local details, thereby optimizing the reconstruction effect at different scales. The use of a more compact MLP architecture improves training efficiency, making 3D reconstruction of large-scale scenes more efficient;

[0058] Appearance embedding vector: Each image is assigned an appearance embedding vector to encode the appearance information of the image. This low-dimensional vector is automatically learned through the training optimization process, and then participates in the calculation of the volume density and color value of each sampling point during the ray tracing process, thereby ensuring that the visual features such as lighting and color of different images are correctly reproduced;

[0059] Transparency estimation network: Multiple MLP networks are introduced to jointly predict transparency and color, where the smaller MLP network focuses on predicting volume density, while the larger MLP network is responsible for predicting color and volume density. In addition, through online rectification technology, multiple networks are jointly trained to optimize the transparency estimation of the model and improve the accuracy of color and density of scene prediction;

[0060] Semantic MLP module: A dedicated semantic MLP module is added to the NeRF model to predict the semantic label of each point in the scene. This module uses the cross entropy loss function to optimize the supervised learning of the training data to ensure that the semantic label of each point can be accurately classified. This function helps to improve the detail capture ability of 3D reconstruction and provides support for accurate semantic segmentation of the orchard.

[0061] S32. Input the image data set preprocessed in step S2 into the improved neural radiation field network model in step S31 for training to generate three-dimensional reconstruction and semantic segmentation of the orchard scene.

[0062] S4. Loss function design: A composite loss function including reconstruction loss, depth loss, semantic loss, visibility loss, distortion loss and inter-layer loss is used to train the improved neural radiation field network model in step S31 to further optimize the effects of 3D reconstruction and semantic segmentation. Specifically, the composite loss function is:

[0063] Reconstruction loss: NeRF core loss function used to measure the difference between the generated image and the real image, based on color rendering loss;

[0064] Depth loss: Use the depth map of monocular depth estimation as a priori and calculate it using mean square error (MSE);

[0065] Semantic loss: The difference between the predicted semantic label and the true label is measured by cross entropy loss;

[0066] Visibility loss: By penalizing the high density values ​​of invisible areas in the training view, floating objects in the scene are removed and the scene geometry is optimized. Visibility loss is applied in the early stage of training to reduce inference time;

[0067] Distortion loss: penalizes the weight distribution of distant points, ensures that the weights are concentrated at closer locations, and ensures that the density estimation is more accurate and localized;

[0068] Inter-layer loss: supervises the voxel density distribution at different levels to ensure the consistency between the transparency estimation network and the NeRF network prediction results.

[0069] S5, performance evaluation: The performance of the 3D reconstruction and semantic segmentation results in step S32 is evaluated using peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), perceived image similarity (LPIPS), rendering frames per second (FPS), intersection over union (IoU), mean intersection over union (mIoU), single category pixel accuracy and total pixel accuracy indicators, where the 3D reconstruction performance evaluation indicators are specifically:

[0070] Peak signal-to-noise ratio (PSNR): used to measure the similarity between the generated image and the real image. The higher the value, the better the image quality.

[0071] Structural Similarity Index (SSIM): used to evaluate the fidelity of image structural information and reflect the structural similarity between the reconstructed image and the reference image;

[0072] Perceptual Image Similarity (LPIPS): The perceptual distance of the image is calculated through a deep learning network. The smaller the value, the smaller the perceptual difference between the generated image and the real image.

[0073] Rendering frames per second (FPS): used to measure the rendering efficiency of 3D scenes. The higher the value, the stronger the real-time rendering capability of the system.

[0074] The semantic segmentation performance evaluation indicators are as follows:

[0075] Intersection over Union (IoU): Evaluates the degree of overlap between the segmented area predicted by the model and the true area. The higher the value, the better the segmentation effect.

[0076] Mean Intersection over Union (mIoU): The IoU of all categories is averaged to measure the accuracy of the overall semantic segmentation.

[0077] Single-category pixel accuracy: Calculate the correct pixel ratio for each classification to evaluate the accuracy of segmentation.

[0078] Total pixel accuracy: Calculates the proportion of pixels predicted correctly among all pixels, reflecting the overall segmentation performance of the model on the entire image.

[0079] S6. 3D visualization and rendering: Input the improved neural radiation field network model trained in step S4 into the Nerfstudio 3D visualization tool to render the orchard scene and visualize the segmentation results. The rendering resolution, viewing angle and scene scaling of the 3D model can be freely selected.

[0080] The present invention significantly improves the accuracy and efficiency of three-dimensional reconstruction when dealing with large-scale outdoor environments, especially complex lighting conditions and changing terrain. Traditional three-dimensional reconstruction methods, such as point cloud-based technology, are often limited by sensor performance and data acquisition conditions, and it is difficult to achieve ideal results when dealing with complex structural details of plants. The present invention combines a number of advanced technologies, such as optimized drone multi-perspective reconstruction, improved NeRF algorithm, and introduced semantic fields. The comprehensive use of these technologies enables the present invention to overcome the limitations of traditional methods when facing large-scale orchard scenes, and achieve faster, more precise and accurate three-dimensional reconstruction.

[0081] Secondly, the present invention adopts multi-resolution hash coding technology to achieve efficient scene representation. By adjusting the size of the hash table, the dimension of the feature vector and the number of resolution levels, the present invention achieves a good balance between reconstruction quality, memory consumption and performance. This efficient scene representation method not only improves the reconstruction efficiency, but also makes the model more stable and reliable when processing large-scale data.

[0082] The present invention also introduces appearance embedding vectors to cope with changes in lighting conditions in the scene. The appearance embedding vectors are automatically optimized during the training process, allowing the model to learn the unique appearance features of each image. The introduction of this technology enables the present invention to accurately reflect these features when generating new perspectives, thereby improving the visual consistency of the reconstruction. This feature is particularly important for the 3D reconstruction of orchard scenes, because the lighting conditions in orchards are often complex and changeable, and traditional 3D reconstruction methods often have difficulty coping with such changes.

[0083] The present invention also adds a 3D semantic segmentation function. By fusing multi-view color images and semantic images, the present invention realizes the joint representation of texture, geometric information and semantic information. This function not only improves the detail capture capability of 3D reconstruction, but also provides support for the accurate semantic segmentation of the orchard. This enables orchard managers to more intuitively understand the distribution and status of various objects in the orchard, and provides an intelligent solution for fruit tree monitoring, disease detection and orchard management.

[0084] Finally, the present invention further reduces the artifacts inside and outside the scene and optimizes the representation of the overall geometric structure by designing a composite loss function. The composite loss function includes multiple parts such as reconstruction loss, depth loss, semantic loss, visibility loss, distortion loss and inter-layer loss, which work together in the training process of the model, so that the model can more accurately reflect the characteristics of the real scene when generating 3D images.

[0085] In summary, the orchard 3D reconstruction and fruit tree semantic segmentation method based on neural radiation field of the present invention has significant advantages in dealing with large-scale outdoor environments. It not only improves the accuracy and efficiency of 3D reconstruction, but also realizes efficient scene representation and the ability to cope with changes in lighting conditions. At the same time, the added 3D semantic segmentation function and the designed composite loss function further improve the performance and practicality of the model. These beneficial effects make the present invention have broad application prospects in the field of smart agriculture.

[0086] See also Figure 1-6 The present invention also provides a specific embodiment of a method for orchard 3D reconstruction and fruit tree semantic segmentation based on neural radiation field, comprising the following steps:

[0087] S1. Image acquisition: Use drones to take multi-view photos of the orchard to obtain the original image data set for 3D reconstruction, specifically:

[0088] The multi-view RGB image set of the orchard was taken by a drone at an altitude of 55 meters above the ground. The drone flew at a constant speed in a pre-planned trajectory and took pictures at the same time interval, covering an orchard area of ​​2.03 hectares. The resolution of the original image is 5472×3078. The dataset contains 162 vertical images and 86 oblique images. The images from two different angles are used for training and testing sets respectively.

[0089] S2. Preprocessing: Preprocess the original image dataset, including camera pose estimation and dedistortion, image downsampling, monocular depth estimation and semantic annotation, to generate RGB images, depth maps and semantic maps suitable for neural radiation field network training, specifically:

[0090] like Figure 2 As shown in the figure, the preprocessing process includes the following steps: first, input the original image, use the COLMAP tool to perform camera pose estimation and dedistortion processing, generate input data for high-precision 3D reconstruction, use COLMAP to input the original image, after feature detection and extraction, feature matching and geometric verification, structure recovery structure, so as to obtain camera pose estimation, and then further remove image distortion to improve the reconstruction effect, then downsample the image to one-fourth of the original resolution, that is, 1356×763, to reduce the amount of calculation and video memory usage. Next, use the pre-trained Depth Anything model to perform monocular depth estimation on the RGB image to generate the depth map required for model depth supervision. Finally, use the labelme tool to semantically annotate the canopy of the fruit trees in the image and generate a semantic map for semantic segmentation of the scene.

[0091] S3. Neural radiation field network construction and training:

[0092] S31. Based on the original neural radiation field network model, the input sampling point coordinate shrinkage module, multi-resolution hash coding and compact MLP architecture, appearance embedding vector, transparency estimation network and semantic MLP module are integrated to construct an improved neural radiation field network model. The specific optimizations are as follows:

[0093] Input sampling point coordinates to shrink module: Figure 4 As shown, the present invention performs scene shrinkage processing on the input coordinate parameters, specifically adopts the L∞ shrinkage technology to shrink the unbounded space into a cube with a side length of 4. In this process, the samples in the cube with a side length of 2 remain unchanged, while the sampling points outside the cube are compressed into the cube with a side length of 4 to improve space utilization and reconstruction accuracy.

[0094] Multi-resolution hash coding and compact MLP architecture: Figure 5 As shown, in order to improve the reconstruction efficiency, the present invention adopts multi-resolution hash coding technology. First, the adjacent voxels of each resolution level are found according to the input coordinates, and the vertices of these voxels are hashed. Then, the feature vector is extracted from the hash table, and linear interpolation is performed according to the relative position of the input coordinates in space. Finally, the feature vector is combined with parameters such as the viewing direction and input into the neural network for evaluation. By reasonably adjusting the size of the hash table (T), the dimension of the feature vector (F) and the number of resolution levels (L), a balance can be achieved between reconstruction quality, memory consumption and performance. Among them, T of the hash code is set to 19, F is 2, and L is 8192.

[0095] Appearance embedding vector: To cope with changes in lighting conditions in the scene, the model introduces appearance embedding technology. The appearance embedding vector is automatically optimized during the training process. By adjusting the actual image data during training, the difference between the predicted image and the real image is minimized, allowing the model to learn the unique appearance features of each image, accurately reflect these features when generating new perspectives, and improve the visual consistency of reconstruction.

[0096] Transparency estimation network: Figure 6 As shown in the figure, the model adopts an efficient light sampling strategy, using two proposal networks PropNet1 and PropNet2. During the training process, first, the coarse network performs preliminary sampling on each light. The distribution of the sampling points can be uniform or based on some prior distribution. Then, the probability density function sampler generates new sampling points according to the weights of the existing sampling points, and predicts the density of each sampling point. After calculating the weights according to the density values ​​of the initial sampling points, these weights are used to resample the light, focusing on high-density areas to improve sampling efficiency. Finally, the fine network is used to evaluate the resampled points to generate the final color and density estimates, thereby generating the final image.

[0097] Semantic MLP module: The model also integrates semantic fields to output results containing semantic information. By fusing multi-view color images and semantic images, it can achieve the joint representation of texture, geometric information and semantic information. This method can not only use sparse or noisy semantic labels for training, but also effectively reduce the workload of annotation. The present invention roughly annotates the data set manually and adds semantic label information of fruit trees and background. By introducing cross entropy loss in training, the accuracy of semantic classification is ensured, and the performance of 3D reconstruction and semantic segmentation is further improved;

[0098] S32. Input the image data set preprocessed in step S2 into the improved neural radiation field network model in step S31 for training to generate three-dimensional reconstruction and semantic segmentation of the orchard scene.

[0099] S4. Loss function design: A composite loss function including reconstruction loss, depth loss, semantic loss, visibility loss, distortion loss and inter-layer loss is used to train the improved neural radiation field network model in step S31 to further optimize the effects of 3D reconstruction and semantic segmentation. The loss function design is specifically as follows:

[0100] Reconstruction loss:

[0101]

[0102] in:

[0103] is the set of all rays in the training set;

[0104] C(r) is the rendered color;

[0105] is the color of the real image.

[0106] Depth Loss:

[0107]

[0108] in: and D gt The minimum and maximum depth values ​​in ;

[0109] and D pred The minimum and maximum depth values ​​in .

[0110] Semantic loss:

[0111]

[0112] Where: y i,k Represents the true label of the i-th pixel in the k-th category (one-hot encoding); represents the predicted probability of the i-th pixel in the k-th category.

[0113] Visibility loss:

[0114]

[0115] Where: σ(q i ) represents the 3D position x i NeRF density value at ;

[0116] V(x i ) is the visibility function, if the position x i is not visible in the training view, then V(x i )=0. Distortion loss:

[0117]

[0118] Where: N is the number of sampling points along the ray;

[0119] w i and w j are the weights assigned to the i-th and j-th sampling points respectively;

[0120] s i and j is the normalized distance between the ith and jth sampling points along the ray.

[0121] Interlayer loss:

[0122]

[0123] Where: t and w are the distance and weight generated by the NeRF network;

[0124] and are the distances and weights generated by the transmittance estimation network;

[0125] T i is the i-th interval;

[0126] represents all the intervals T i The sum of the weights of the overlapping transmittance estimation networks. The total loss function is:

[0127]

[0128] in:

[0129] α1, α2, α3, α4, α5, and α6 are weight coefficients of each partial loss, and these coefficients can be adjusted according to actual conditions and specific problems. In the model of this embodiment, α1=1, α2=1e-3, α3=1, α4=1e-6, α5=1e-2, and α6=1.

[0130] S5, performance evaluation: The performance of the 3D reconstruction and semantic segmentation results in step S32 is evaluated using peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), perceived image similarity (LPIPS), rendering frames per second (FPS), intersection over union (IoU), mean intersection over union (mIoU), single category pixel accuracy and total pixel accuracy indicators, where the 3D reconstruction performance evaluation indicators are specifically:

[0131] Peak signal-to-noise ratio (PSNR): used to measure the similarity between the generated image and the real image. The higher the value, the better the image quality.

[0132] Structural Similarity Index (SSIM): used to evaluate the fidelity of image structural information and reflect the structural similarity between the reconstructed image and the reference image;

[0133] Perceptual Image Similarity (LPIPS): The perceptual distance of the image is calculated through a deep learning network. The smaller the value, the smaller the perceptual difference between the generated image and the real image.

[0134] Rendering frames per second (FPS): used to measure the rendering efficiency of 3D scenes. The higher the value, the stronger the real-time rendering capability of the system.

[0135] The semantic segmentation performance evaluation indicators are as follows:

[0136] Intersection over Union (IoU): Evaluates the degree of overlap between the segmented area predicted by the model and the true area. The higher the value, the better the segmentation effect.

[0137] Mean Intersection over Union (mIoU): The IoU of all categories is averaged to measure the accuracy of the overall semantic segmentation.

[0138] Single-category pixel accuracy: Calculate the correct pixel ratio for each classification to evaluate the accuracy of segmentation.

[0139] Total pixel accuracy: Calculates the proportion of pixels predicted correctly among all pixels, reflecting the overall segmentation performance of the model on the entire image.

[0140] S6. 3D visualization and rendering: Input the improved neural radiation field network model trained in step S4 into the Nerfstudio 3D visualization tool to render the orchard scene and visualize the segmentation results. The rendering resolution, viewing angle and scene scaling of the 3D model can be freely selected.

[0141] The present invention also provides the following comparative experiments, wherein Table 1 is a comparison of the present invention with other NeRF-type three-dimensional reconstruction models in terms of three-dimensional reconstruction, and Table 2 is a comparison of the present invention with the classic two-dimensional semantic segmentation model DeepLab V3+ in terms of semantic segmentation.

[0142] Table 1: Comparison of the present invention with other NeRF-based 3D reconstruction models in 3D reconstruction

[0143]

[0144] Table 2. Comparison of the semantic segmentation of the present invention with the classic 2D semantic segmentation model DeepLab V3+

[0145]

[0146] According to the comparative experiments in Table 1 and Table 2, the model of the present invention converges quickly within five minutes, and the PSNR reaches 23.1. In addition, it surpasses the current advanced NeRF-type models such as Instant-NGP and Mip-NeRF 360 in terms of 3D reconstruction indicators such as PSNR, SSIM and LPIPS.

[0147] A New Look at Fruit Trees Figure 2 In the 3D semantic segmentation task, the present invention achieved a mean intersection over union (mIoU) of 0.89, which is more than 7% higher than the DeepLab V3+ model. This not only improves the accuracy of semantic segmentation, but also provides strong support for the application of complex agricultural scenes. The method of the present invention has opened up a new path for the application of digital twin technology in agriculture, demonstrated excellent performance in 3D reconstruction quality and computational efficiency, and has broad application prospects.

[0148] See also Figure 7 This embodiment also discloses Figure 7 The figure shows a schematic diagram of an actual operation system of a method for 3D reconstruction of an orchard based on a UAV. The UAV is equipped with a flight module and an image module. The flight module is responsible for sensing the environment, maintaining a stable flight attitude, and designing the flight trajectory of the UAV to achieve efficient flight. The module includes a variety of sensors, a flight control system, and navigation technology. The sensor uses a three-axis gyroscope, and the navigation system uses GPS positioning technology. The flight control system is responsible for interacting with the ground remote control station to control the flight of the UAV. The image module is used to automatically avoid obstacles and obtain video streams during the flight of the UAV. The automatic obstacle avoidance system can effectively reduce the interference of obstacles during flight operations and improve operation efficiency. The UAV ground station is responsible for route planning and controlling the flight altitude and speed. After the UAV shoots the orchard image, the video stream is transmitted to the background server in real time. The server cuts the video stream and pre-processes the image. The processed image sequence is input into the 3D reconstruction model for reconstruction, and the reconstruction results can be browsed and analyzed through the 3D visualization tool of the Web browser.

[0149] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0150] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for 3D reconstruction of orchards and semantic segmentation of fruit trees based on neural radiation fields, characterized by: The following steps are involved: S1. Image acquisition: Use drones to take multi-view photos of the orchard to obtain the original image dataset for 3D reconstruction; S2, preprocessing: preprocessing the original image data set, including camera pose estimation and dedistortion, image downsampling, monocular depth estimation and semantic annotation, to generate image data of RGB images, depth maps and semantic maps suitable for neural radiation field network training; S3. Neural radiation field network construction and training: S31. Based on the original neural radiation field network model, the input sampling point coordinate shrinkage module, multi-resolution hash coding and compact MLP architecture, appearance embedding vector, transparency estimation network and semantic MLP module are integrated to construct an improved neural radiation field network model; S32, inputting the image data set preprocessed in step S2 into the improved neural radiation field network model in step S31 for training, generating a three-dimensional reconstruction and semantic segmentation of the orchard scene; S4. Loss function design: using a composite loss function including reconstruction loss, depth loss, semantic loss, visibility loss, distortion loss and inter-layer loss to train the improved neural radiation field network model in step S31, so as to further optimize the effects of 3D reconstruction and semantic segmentation; S5, performance evaluation: using peak signal-to-noise ratio, structural similarity index, perceived image similarity, rendering frames per second, intersection-over-union ratio, average intersection-over-union ratio, single-category pixel accuracy and total pixel accuracy indicators to perform performance evaluation on the 3D reconstruction and semantic segmentation results in step S32; S6. 3D visualization and rendering: Apply the improved neural radiation field network model trained in step S4 to the 3D visualization tool to achieve rendering of the orchard scene and visualization of the segmentation results.

2. The method for orchard 3D reconstruction and fruit tree semantic segmentation based on neural radiation field according to claim 1, characterized in that: In step S31: The input sampling point coordinate shrinkage module is used to reduce the computational burden and improve the modeling accuracy; The multi-resolution hash coding and compact MLP architecture are used to capture global and local information of the scene; The appearance embedding vector is used to encode the appearance information of the image; The transparency estimation network is used to optimize the transparency estimation of the model; The semantic MLP module is used to predict the semantic label of each point in the scene.

3. The method for orchard 3D reconstruction and fruit tree semantic segmentation based on neural radiation field according to claim 1, characterized in that: The composite loss function in step S4 is specifically: The reconstruction loss is used to measure the difference between the generated image and the real image, based on the color rendering loss; The depth loss uses the depth map of monocular depth estimation as a priori and is calculated using mean square error; The semantic loss measures the difference between the predicted semantic label and the true label through the cross entropy loss; The visibility loss removes floating objects in the scene and optimizes the scene geometry by penalizing high density values ​​in invisible areas in the training view. The distortion loss is used to penalize the weight distribution of distant points, ensuring that the weights are concentrated at closer locations, ensuring that the density estimation is more accurate and localized; The inter-layer loss is used to supervise the voxel density distribution at different levels to ensure the consistency of the prediction results of the transparency estimation network and the NeRF network.

4. The method for orchard 3D reconstruction and fruit tree semantic segmentation based on neural radiation field according to claim 1, characterized in that: The performance evaluation index in step S5 is specifically: The peak signal-to-noise ratio is used to measure the similarity between the generated image and the real image; The structural similarity index is used to evaluate the fidelity of image structural information and reflects the structural similarity between the reconstructed image and the reference image; The perceived image similarity: calculating the perceived distance of the images through a deep learning network; The rendering frames per second is used to measure the rendering efficiency of the three-dimensional scene; Intersection-over-Union: Evaluates the degree of overlap between the segmented area predicted by the model and the true area. Average Intersection over Union (IoU): The IoU of all categories is averaged to measure the accuracy of the overall semantic segmentation. Single category pixel accuracy: calculate the correct pixel ratio for each category and evaluate the accuracy of segmentation; Total pixel accuracy: Calculates the proportion of pixels predicted correctly among all pixels, reflecting the overall segmentation performance of the model on the entire image.

Images