Neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation

Abstract

The invention discloses a neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation. The method comprises the following steps: inputting an RGB image set; calculating a 3D position and an observation direction; a step of initializing NeRF training; based on the 5D coordinate (x, d) of the residual image set, threshold sampling is carried out through a modified NeRF network, the color and opacity of the RGB image are calculated, the color and opacity are integrated into Beta distribution, and rendering uncertainty is calculated; inputting a 3D coordinate x of a residual image set, judging whether a track is a plane or a non-plane through a classifier, and estimating position information by using a Voronoi graph algorithm to obtain position uncertainty; an optimal view selection step: performing normalization and summation on the rendering uncertainty and the position uncertainty, and calculating the hybrid uncertainty of each RGB image; and selecting the image with the highest hybrid uncertainty, adding the image into a training set, and repeating the process until a specific reconstruction effect or a preset image number limit is achieved.

Description

Technical Field

[0001] The technical field involved in this article is the field of computer vision technology for large-scale three-dimensional reconstruction using Neural Radiance Fields (NeRF). Specifically, the present invention particularly relates to an incremental optimal view selection method for Neural Radiance Fields based on hybrid uncertainty estimation. Background Art

[0002] At present, image-based 3D reconstruction technology is mainly divided into two categories: one is the traditional geometry-based method, and the other is the neural network-based method. The traditional method mainly includes two steps: Structure From Motion (SfM) and Multiple View Stereo (MVS). SfM aims to analyze feature points from image sequences and obtain 3D structures and camera trajectories by adjusting parameters. Subsequently, MVS generates cloud points of a single viewpoint by estimating the depth map of each viewpoint. Another method is represented by Neural Radiance Fields (NeRF), which is a major breakthrough in 3D reconstruction technology in recent years. NeRF implicitly expresses 3D scene information by training network parameters and can synthesize images from new perspectives. However, large-scale reconstruction technology at the city level faces two major challenges: first, large-scale reconstruction usually requires processing and storing large amounts of data, which may quickly exhaust the memory resources of a single GPU, resulting in slower processing, insufficient memory, and other performance issues, which is particularly unfriendly to users or researchers with limited video memory resources; second, with the growing demand for real-time or near-real-time applications, such as navigation or disaster management, it has become particularly urgent to study faster and more efficient large-scale 3D reconstruction technology.

[0003] To create more accurate, detailed, and useful large-scale 3D models and address the practical challenges of working with large datasets and limited computing resources, this study proposes an efficient method for fast and robust large-scale 3D reconstruction under limited video memory resources. Summary of the invention

[0004] Based on the problems that the training speed of existing large-scale reconstruction technology is still very long and the video memory requirements are high, the present invention provides a fast and efficient large-scale 3D reconstruction method based on neural radiation field that can effectively process a large amount of data input under limited video memory resources. The method proposed in the present invention mainly includes two parts: a view planning strategy based on uncertainty and a view selection strategy based on information gain.

[0005] The method proposed in the present invention fuses features of different time window sizes, uses multiple prediction heads to output multiple suggested positions at different levels, and takes the point position with the highest confidence as the final point tracking result. The constructed model has no restrictions on the length of the input video sequence and the number of tracking points, and can complete the tracking of all points in all frames in parallel through only one forward propagation process.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] A neural radiance field incremental optimal view selection method based on hybrid uncertainty estimation comprises the following steps:

[0008] Input RGB image set step: input a set of N RGB image data with uniform resolution taken by a drone;

[0009] Calculate 3D position and viewing direction; process each input RGB image, perform pose estimation, restore the structure in the RGB image and output the 3D position x and viewing direction vector d corresponding to each image in the scene; the structure in the RGB image refers to the scene geometry information restored from the RGB image, including but not limited to the position, shape and spatial relationship of objects;

[0010] Initialize the NeRF training step; randomly select a predetermined proportion of images from the RGB image set to perform initialization training of the NeRF model; the predetermined proportion is not less than 15%;

[0011] Calculate the mixed uncertainty step; Based on the 5D coordinates (x, d) of the remaining image set, perform threshold sampling through the modified NeRF network, calculate the color and opacity of the RGB image, and then integrate the color and opacity into the Beta distribution to calculate the rendering uncertainty; Input the 3D coordinate x of the remaining image set, use the classifier to determine whether the trajectory is planar or non-planar, use the Voronoi diagram algorithm to estimate the position information, and obtain the position uncertainty;

[0012] The optimal view step is selected, the rendering uncertainty and position uncertainty are normalized and summed, and the mixed uncertainty of each RGB image is calculated; the image with the highest mixed uncertainty is selected, added to the training set, and this process is repeated until a specific reconstruction effect or a preset image number limit is achieved.

[0013] The training set added is an image dataset used to train a NeRF model. The NeRF model and traditional triangular meshes, point clouds, voxels, etc. are all forms of expression for 3D reconstruction, but triangular meshes, point clouds, voxels, etc. are all explicit expressions, while NeRF is an implicit expression, and the scene information is stored in a neural network. To reconstruct a scene, you need to train a network specifically and use the collected images of the scene for multi-view reconstruction. The images used for reconstruction are the training set. If there is enough video memory, the more training set images there are, the more information they bring, and the better the reconstruction effect.

[0014] As a preferred aspect of the present invention, in the step of calculating mixed uncertainty,

[0015] Based on the 5D coordinates (x, y, z, θ, φ) of the remaining image set, threshold sampling is performed through a modified NeRF network, and the color and opacity of the sampling points on each ray of each RGB image are calculated through the multi-resolution hash information storage of the explicit voxel grid and the implicit neural network. The rendering uncertainty of each image is calculated by integrating it into the Beta distribution; at the same time,

[0016] The 3D coordinates (x, y, z) of the remaining image sets are input, and the classifier is used to determine whether the UAV flight trajectory is planar or non-planar. For planar trajectories, the Voronoi information radiation field is used to perform top-level global planning and bottom-level local planning, thereby compressing and limiting the position information. For non-planar trajectories, the Voronoi clustering algorithm is used to quantify the uncertainty of the position by the distance from the center point.

[0017] Among them, (x, d) and (x, y, z, θ, φ) are different descriptions of 5D coordinates. The x and d in the former represent the position and viewing direction in the three-dimensional space respectively, while the x, y, z in the latter refer to the values ​​of the xyz coordinate system in the three-dimensional space, and θ, φ refer to the horizontal and vertical components of the direction.

[0018] As a preferred aspect of the present invention, the step of calculating mixed uncertainty is specifically performed as follows:

[0019] Through the NeRF model, the color and opacity of each ray are obtained, and the volume density is calculated; the volume density calculation formula is as follows:

[0020]

[0021] Among them, a i represents the volume density of the i-th sampling point on the ray, σ j represents the opacity of the jth sampling point, δ j Represents the distance between the jth sampling point and the previous sampling point.

[0022] The color variance is calculated for each ray using the NeRF model. The color variance is calculated using the following formula:

[0023] β 2 (r(t i ))=-P(α i )log P(α i )

[0024] Among them, β 2 (r(t i )) represents the variance of the i-th sampling point on the ray, α i represents the volume density of the i-th sampling point (Formula 1 above), P(α i ) represents α i The total volume density of the ray ∑α i proportion;

[0025] Construct a Beta distribution based on the estimated mean and variance in, Represents the color of a predicted ray, Represents the mean color of a ray, Represents the mean of the variance of a ray;

[0026] The continuous function is optimized by minimizing the squared reconstruction error between the true RGB image and the rendered pixel color, while calculating the rendering uncertainty of each image; the calculation formula for image rendering uncertainty is as follows:

[0027]

[0028] Represents the rendering uncertainty of the I-th image, N r Represents the total number of rays per image, Represents the squared error between the true RGB image and the rendered pixel color, represents the average variance of ray i;

[0029] Calculate the Hausdorff distance between the two data sets to determine whether the image trajectory is planar or non-planar; where the Hausdorff distance is h(A,B) is the Hasdorff distance between two trajectories A and B, ||a i -b j || is the distance between the i-th point in trajectory A and the j-th point in trajectory B.

[0030] As a preferred aspect of the present invention, the step of calculating the mixed uncertainty is specifically performed as follows: for a planar trajectory, the position uncertainty is calculated using the following formula:

[0031]

[0032] Among them, F p (I) is the plane position uncertainty value of photo I, N v Represents the total number of 3D pose points, Represents the weight value λ of point i and point j at i i The distance value under A i Represents the area of ​​the Voronoi diagram at point i.

[0033] As a preferred aspect of the present invention, for non-planar trajectories, the following formula is used to estimate the position uncertainty:

[0034]

[0035] Among them, F np (I) is the non-planar position uncertainty value of photo I, G i is the probability of selecting image i based on the Voronoi graph polygon, r o Relative local density that represents the density change of the evaluation point before and after selection.

[0036] As a preferred aspect of the present invention, in the step of selecting the optimal view, the rendering uncertainty and the position uncertainty are normalized and summed to form a mixed uncertainty of each image; the image with the highest mixed uncertainty is selected from the candidate set, added to the training set, and this process is repeated until a specific reconstruction effect or a preset image quantity limit is achieved, thereby optimizing the view selection strategy; wherein the calculation formula of the mixed uncertainty is as follows:

[0037]

[0038] in, represents the position uncertainty of image I, represents the rendering uncertainty of image I. The normalized sum of the two is the total uncertainty of image I ψ 2 (I).

[0039] As a preferred aspect of the present invention, it also includes:

[0040] Iterative optimization step; in each iteration, the uncertainty of each unused candidate view is calculated based on the current training set, and the view with the largest information gain is selected and added to the training set to gradually improve the quality of the synthesis of new views.

[0041] As a preferred aspect of the present invention, in the step of inputting RGB image collection, a rotary-wing UAV is first used to fly a specific route to the target area, and the flight route is divided into plane and non-plane; fixed-point shooting is performed during the flight, so as to obtain a set of N aerial survey RGB image data with the same resolution size, which are taken by the UAV in the form of orthogonal or oblique images.

[0042] As a preferred aspect of the present invention, in the calculation of the 3D position and the observation direction, COLMAP software is used to estimate the pose of all input RGB images to obtain the camera shooting pose corresponding to each image; wherein the camera model is set to a pinhole camera model by default and all cameras are set to perform incremental SfM reconstruction with the same internal parameters, thereby obtaining the internal and external parameters of all cameras.

[0043] As a preferred aspect of the present invention, in the initialization NeRF training step, at least 15% of the images are randomly selected from the RGB image set for initialization training of the NeRF model, so that the NeRF model learns the basic structure and appearance features of the scene and stores them in the multi-layer perceptron network.

[0044] Among them, 15% of the images are randomly selected from the training set for initialization training to obtain a problematic scene model, and subsequent iterative training is performed based on the initialization model. 15% of the images are randomly selected from the training set for initialization training to obtain a problematic scene model. If the problem is relatively small, fewer subsequent iterations can achieve better results; otherwise, more iterations are required.

[0045] For example: there are 100 images, 10 images (10%) are selected as the test set to evaluate the quality of the model, and the remaining 90 images are used as the training set, of which 15 images (15%) are used for initialization training sets, and the rest are training candidate sets. At this time, a poor scene model is obtained after initialization training. At this time, the mixed uncertainty of all images in the training candidate set is calculated to obtain the candidate image with the largest amount of information, which is added to the training set (the training set is 16 images at this time, 15 initialization images + 1 newly added candidate image), and iterative training is performed to obtain a new optimized scene model. Repeat the above steps until the set conditions are met, for example, 15 images are selected.

[0046] The specific advantages of the present invention are:

[0047] The present invention introduces a hybrid uncertainty based on position uncertainty and rendering uncertainty to achieve information gain, and at the same time, through the selection of the viewing angle with the maximum gain, it helps to achieve the task of fast large-scale 3D reconstruction under limited video memory resources. Compared with the existing technology, it has significant advantages such as being able to process larger input data, completing large-scale 3D reconstruction faster, and rendering higher image quality under the same conditions.

[0048] This neural radiance field incremental optimal view selection method based on hybrid uncertainty estimation can solve the artifact problem in multiple viewpoints and reduce the computational cost while maintaining high rendering realism. This method pays special attention to how to effectively select the view that can bring the maximum information gain from the candidate views through the incremental optimal view selection strategy under the constraint of limited video memory resources, thereby improving rendering quality and efficiency.

[0049] In addition, the present invention also combines hybrid multi-resolution hash storage of explicit voxel grids and implicit neural networks, Voronoi diagram information gain radiation field and clustering algorithm, threshold sampling and flight classifier to further improve processing efficiency while ensuring the performance of the original NeRF network. This technology can be used as a plug-in tool to assist existing systems in achieving more efficient 3D scene reconstruction and rendering.

[0050] With reference to the following description and accompanying drawings, specific embodiments of the present invention are disclosed in detail, indicating the manner in which the principles of the present invention can be adopted. It should be understood that the embodiments of the present invention are not limited in scope. Within the spirit and scope of the appended claims, the embodiments of the present invention include many changes, modifications and equivalents.

[0051] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0052] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1It is a schematic diagram of the neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation of the present invention.

[0055] Figure 2 It is a flow chart of the neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation of the present invention.

[0056] Figure 3 It is a framework diagram of the construction method of the neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation of the present invention.

[0057] Figure 4 It is a schematic diagram of the Voronoi diagram information gain radiation field used in the planar flight trajectory in the neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation of the present invention.

[0058] Figure 5 It is a schematic diagram of the Voronoi graph clustering algorithm used for non-planar flight trajectories in the neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation of the present invention.

[0059] Figure 6 This is a demonstration of the effectiveness of some results in the incremental optimal view selection method of the neural radiation field based on hybrid uncertainty estimation of the present invention. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0061] like Figures 1 to 6 As shown, one embodiment of the present invention proposes a neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation, which is mainly used to solve the problem of large-scale 3D reconstruction and new perspective rendering output. The algorithm is shown in FIG. Figure 1 shown.

[0062] The algorithm of the present invention mainly includes two parts: calculating rendering uncertainty and calculating position uncertainty and obtaining mixed uncertainty to perform perspective selection planning with maximum information gain. The algorithm flow is as follows: Figure 2 As shown, it mainly includes the following steps:

[0063] S1. First, input a set of RGB real photos. Input a set of high-resolution RGB image data taken by drones.

[0064] Use a rotary wing drone to fly a specific route in a certain area. The flight routes are divided into plane and non-plane. Non-plane is generally flown manually. Plane routes can be set in the app, usually in a tic-tac-toe pattern. During the flight, fixed-point shooting is performed to obtain a set of N aerial survey image data with the same resolution size, which are taken by the drone in the form of orthophoto or oblique projection.

[0065] S2. Use COLMAP to estimate the pose (camera position in space and camera orientation) and reconstruct sparse point clouds for all input images to obtain the camera shooting pose and sparse point cloud information corresponding to each image.

[0066] Use COLMAP to estimate the pose of all input images (the position of the camera in space and the orientation of the camera), and then you can get the camera pose corresponding to each image. The default camera model is set to the pinhole camera model (PINEHOLE) and all cameras are set to use the same internal parameters for incremental SfM reconstruction, thereby obtaining the internal and external parameters of all cameras.

[0067] S3. Initialize NeRF training. Randomly select a certain proportion of images (recommended at least 15%) from the input dataset to perform initialization training of the NeRF model.

[0068] At least 15% of the images are randomly selected from the input dataset for initialization training of the NeRF model to ensure that the model can learn the basic structure and appearance characteristics of the scene.

[0069] S4. Calculate hybrid uncertainty. Based on the 5D coordinates (x, d) of the remaining image set, threshold sampling is performed through the modified NeRF network to calculate the color and opacity of the image.

[0070] The NeRF network is the part of the NeRF model responsible for learning and encoding scene information. The NeRF model relies on the NeRF network to learn the three-dimensional representation of the scene, and other components (such as volume rendering equations, etc.) are also required to achieve new perspective synthesis and three-dimensional reconstruction.

[0071] The complete image set is divided into a test set, a training set, and a training candidate set. The remaining image set refers to the training candidate set. The test set does not participate in the training process and is only used to evaluate the training results. For example, there are 100 images, and 10 (10%) images are selected as the test set to evaluate the quality of the model, and the remaining 90 images are used as the training candidate set. First, 15 (15%) images are selected for initialization training set training, and the images with the largest amount of information are selected from the remaining training candidate set and added to the training set.

[0072] Then, the color and opacity are integrated into the Beta distribution to calculate the rendering uncertainty. For the position uncertainty, the 3D coordinate x of the remaining image set is input, the classifier is used to determine whether the trajectory is planar or non-planar, and then the Voronoi diagram algorithm is used to estimate the position information to obtain the position uncertainty. The specific execution algorithm of this step is as follows:

[0073] S41. The color and opacity of each ray are calculated through the MLP of the NeRF model. The volume density calculated later takes into account the distance and opacity between adjacent samples. The volume density calculation formula is as follows:

[0074]

[0075] Among them, a i represents the volume density of the i-th sampling point on the ray, σ j represents the opacity of the jth sampling point, δ j Represents the distance between the jth sampling point and the previous sampling point.

[0076] S42. Calculate the color variance for each light ray using the NeRF model. The calculation formula for the color variance is as follows:

[0077] β 2 (r(t i ))=-P(α i )log P(α i )

[0078] Among them, β 2 (r(t i )) represents the variance of the i-th sampling point on the ray, α i represents the volume density of the i-th sampling point (Formula 1 above), P(α i ) represents α i The total volume density of the ray ∑α i proportion.

[0079] S43. Since the volume density at a specific location is only affected by its own 3D coordinates and not by the viewing direction, the distributions at different locations are independent of each other. At the same time, volume rendering can be approximated as a linear combination of sampling points along the light, so a Beta distribution can be constructed based on the estimated mean and variance.

[0080] in, Represents the predicted (calculated according to the constructed model) color of a ray, Represents the mean color of a ray, Represents the mean of the variance of a ray and constructs a beta distribution.

[0081] S44, optimize the continuous function by minimizing the squared reconstruction error between the true RGB image and the rendered pixel color, and calculate the rendering uncertainty of each image. The calculation formula for image rendering uncertainty is as follows:

[0082]

[0083] Represents the rendering uncertainty of the I-th image, N r Represents the total number of rays per image, Represents the squared error between the true RGB image and the rendered pixel color, represents the average variance of ray i.

[0084] S45. Calculate the Hausdorff distance between two data sets to determine whether the image trajectory is planar or non-planar.

[0085] Among them, h(A,B) is the Hasdorff distance between two trajectories A and B, ||a i -b j || is the distance between the i-th point in trajectory A and the j-th point in trajectory B.

[0086] S46. For planar trajectories, the position uncertainty is calculated using the Voronoi diagram, taking into account the importance of each position point and the area of ​​the planar Voronoi diagram. The position uncertainty is calculated using the following formula:

[0087]

[0088] Among them, F p (I) is the plane position uncertainty value of photo I, N v Represents the total number of 3D pose points, Represents the weight value λ of point i and point j at i i The distance value under A i Represents the area of ​​the Voronoi diagram at point i.

[0089] For non-planar trajectories, the Voronoi diagram clustering algorithm is used to estimate the position uncertainty, taking into account the importance of three-dimensional space and local correlation density. At this time, the position uncertainty is calculated using the following formula:

[0090]

[0091] Among them, F np (I) is the non-planar position uncertainty value of photo I, G i is the probability of selecting image i based on the Voronoi graph polygon, r iRelative local density that represents the density change of the evaluation point before and after selection.

[0092] S5. Select the optimal view. Normalize and sum the rendering uncertainty and position uncertainty to calculate the mixed uncertainty of each image. Select the image with the highest mixed uncertainty, add it to the training set, and repeat this process until a specific reconstruction effect or a preset image number limit is achieved. The calculation formula for mixed uncertainty is as follows:

[0093]

[0094] in, represents the position uncertainty of image I, represents the rendering uncertainty of image I. The normalized sum of the two is the total uncertainty of image I ψ 2 (I).

[0095] S6, iterative optimization. In each iteration, the uncertainty of each unused candidate view is calculated based on the current training set, and the view with the largest information gain is selected and added to the training set to gradually improve the quality of new view synthesis.

[0096] It should be noted that NeRF introduces a new scene representation and view synthesis method that allows highly realistic 3D scenes to be synthesized from 2D images. The training process involves capturing a set of 2D images of the scene from different viewpoints. For each pixel in the image, NeRF calculates the corresponding light in 3D space. NeRF estimates the scene color and opacity at each point along these rays, then compares the predicted appearance color values ​​with the observed appearance color values ​​and adjusts the network parameters to minimize the difference. After training, NeRF is able to synthesize new scene views by projecting rays from a virtual camera position and accumulating color and opacity along the rays using the learned neural radiance field. This process allows photos to be generated from previously unseen viewpoints, providing realistic scene rendering images. Instant-NGP is a highly engineering-worthy neural radiance field model proposed by NVIDIA that is currently the fastest. This work proposes a multi-resolution hash encoding to accelerate model training, and the multi-resolution hash table enhancement of trainable feature vectors can further reduce the model size, and the entire system is implemented using fully fused CUDA cores, which maximizes parallelism.

[0097] In this embodiment, each iterative training is based on a three-dimensional reconstruction method of a neural radiation field, taking into account perspective planning of position uncertainty and rendering uncertainty, wherein the specific model is Instant-NGP.

[0098] S6, iterative optimization. In each iteration, the uncertainty of each unused candidate view is calculated based on the current training set, and the view with the largest information gain is selected and added to the training set to gradually improve the quality of new view synthesis.

[0099] Depend on Figure 6 It can be seen that some results of the fast large-scale three-dimensional reconstruction method based on neural radiation field of the present invention selected six sets of public large scene data sets and compared them with the previous baselines, all of which achieved better reconstruction effects.

[0100] In the present application, multiple elements, compositions, parts or steps can be provided by a single integrated element, composition, part or step. Alternatively, a single integrated element, composition, part or step can be divided into separate multiple elements, compositions, parts or steps. The disclosure "one" or "an" used to describe an element, composition, part or step does not mean to exclude other elements, compositions, parts or steps.

[0101] It should be understood that the above description is for illustration and not for limitation. Many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of the present teachings should not be determined with reference to the above description, but should be determined with reference to the appended claims and the full scope of equivalents possessed by such claims. For the purpose of comprehensiveness, all articles and references, including disclosures of patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the preceding claims is not intended to be a waiver of the subject matter, nor should it be considered that the inventors did not consider the subject matter to be part of the disclosed inventive subject matter.

Claims

1. A neural radiance field incremental optimal view selection method based on hybrid uncertainty estimation, characterized in that: The following steps are involved: Input RGB image set step: input a set of N RGB image data with uniform resolution taken by a drone; Calculate 3D position and viewing direction; Process each input RGB image, perform pose estimation, restore the structure in the RGB image and output the 3D position x and viewing direction vector d corresponding to each image in the scene; Initialize the NeRF training step; randomly select a predetermined proportion of images from the RGB image set to perform initialization training of the NeRF model; the predetermined proportion is not less than 15%; Calculate the mixed uncertainty step; Based on the 5D coordinates (x, d) of the remaining image set, perform threshold sampling through the modified NeRF network, calculate the color and opacity of the RGB image, and then integrate the color and opacity into the Beta distribution to calculate the rendering uncertainty; Input the 3D coordinate x of the remaining image set, use the classifier to determine whether the trajectory is planar or non-planar, use the Voronoi diagram algorithm to estimate the position information, and obtain the position uncertainty; The optimal view step is selected, the rendering uncertainty and position uncertainty are normalized and summed, and the mixed uncertainty of each RGB image is calculated; the image with the highest mixed uncertainty is selected, added to the training set, and this process is repeated until a specific reconstruction effect or a preset image number limit is achieved.

2. The neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation according to claim 1, characterized in that: In the step of calculating mixed uncertainty, Based on the 5D coordinates (x, y, z, θ, φ) of the remaining image set, threshold sampling is performed through a modified NeRF network, and the color and opacity of the sampling points on each ray of each RGB image are calculated through the multi-resolution hash information storage of the explicit voxel grid and the implicit neural network. The rendering uncertainty of each image is calculated by integrating it into the Beta distribution; at the same time, The 3D coordinates (x, y, z) of the remaining image sets are input, and the classifier is used to determine whether the UAV flight trajectory is planar or non-planar. For planar trajectories, the Voronoi information radiation field is used to perform top-level global planning and bottom-level local planning, thereby compressing and limiting the position information. For non-planar trajectories, the Voronoi clustering algorithm is used to quantify the uncertainty of the position by the distance from the center point.

3. The neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation according to claim 2, characterized in that: The step of calculating the mixed uncertainty is specifically performed as follows: Through the NeRF model, the color and opacity of each ray are obtained, and the volume density is calculated; the volume density calculation formula is as follows: Among them, a i represents the volume density of the i-th sampling point on the ray, σ j represents the opacity of the jth sampling point, δ j Represents the distance between the jth sampling point and the previous sampling point. The color variance is calculated for each ray using the NeRF model. The color variance is calculated using the following formula: b 2 (r(t i ))=-P(a i )log P(a i ) Among them, β 2 (r(t i )) represents the variance of the i-th sampling point on the ray, α i represents the volume density of the i-th sampling point (Formula 1 above), P(α i ) represents α i The total volume density of the ray ∑α i proportion; Construct a Beta distribution based on the estimated mean and variance in, Represents the color of a predicted ray, Represents the mean color of a ray, Represents the mean of the variance of a ray; The continuous function is optimized by minimizing the squared reconstruction error between the true RGB image and the rendered pixel color, while calculating the rendering uncertainty of each image; the calculation formula for image rendering uncertainty is as follows: Represents the rendering uncertainty of the I-th image, N r Represents the total number of rays per image, Represents the squared error between the true RGB image and the rendered pixel color, represents the average variance of ray i; Calculate the Hausdorff distance between the two data sets to determine whether the image trajectory is planar or non-planar; where the Hausdorff distance is h(A,B) is the Hasdorff distance between two trajectories A and B, ||a i -b j || is the distance between the i-th point in trajectory A and the j-th point in trajectory B.

4. The neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation according to claim 3, characterized in that: The step of calculating the mixed uncertainty is specifically performed as follows: for a planar trajectory, the position uncertainty is calculated using the following formula: Among them, F p (I) is the plane position uncertainty value of photo I, N v Represents the total number of 3D pose points, Represents the weight value λ of point i and point j at i i The distance value under A i Represents the area of ​​the Voronoi diagram at point i.

5. The neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation according to claim 3, characterized in that: For non-planar trajectories, the following formula is used to estimate the position uncertainty: Among them, F np (I) is the non-planar position uncertainty value of photo I, G i is the probability of selecting image i based on the Voronoi graph polygon, r i Relative local density that represents the density change of the evaluation point before and after selection.

6. The neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation according to claim 1, characterized in that: In the step of selecting the optimal view, the rendering uncertainty and the position uncertainty are normalized and summed to form the mixed uncertainty of each image; the image with the highest mixed uncertainty is selected from the candidate set, added to the training set, and this process is repeated until a specific reconstruction effect or a preset image quantity limit is achieved, thereby optimizing the view selection strategy; wherein the calculation formula of the mixed uncertainty is as follows: in, represents the position uncertainty of image I, represents the rendering uncertainty of image I. The normalized sum of the two is the total uncertainty of image I ψ 2 (I).

7. The neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation according to claim 1, characterized in that: Also includes: Iterative optimization steps; In each iteration, the uncertainty of each unused candidate view is calculated based on the current training set, and the view with the largest information gain is selected and added to the training set to gradually improve the quality of synthesis from new perspectives.

8. The neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation according to claim 1, characterized in that: In the step of inputting the RGB image set, a rotary-wing UAV is first used to fly a specific route to the target area, and the flight route is divided into plane and non-plane; fixed-point shooting is performed during the flight, so as to obtain a set of N aerial survey RGB image data with the same resolution size, which are taken by the UAV in the form of orthogonal or oblique images.

9. The neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation according to claim 1, characterized in that: In the calculation of the 3D position and observation direction, COLMAP software is used to estimate the pose of all input RGB images to obtain the camera shooting pose corresponding to each image; wherein, the camera model is set to a pinhole camera model by default and all cameras are set to perform incremental SfM reconstruction with the same internal parameters, thereby obtaining the internal and external parameters of all cameras.

10. The neural radiation field incremental optimal view selection method based on hybrid uncertainty estimation according to claim 1, characterized in that: In the initialization NeRF training step, at least 15% of the images are randomly selected from the RGB image set for initialization training of the NeRF model, so that the NeRF model can learn the basic structure and appearance features of the scene and store them in the multi-layer perceptron network.

Images