Single repositioning method for large view variable light field scene based on neural radiance field supervision

Abstract

The application discloses a single relocation method for a large-view variable light scene based on neural radiation field supervision, and comprises the following steps: acquiring a large-view multi-view image under a luminosity change condition by using a camera; estimating an initial pose of the camera by using an SFM algorithm; training a neural radiation field model for the large-view variable light scene by taking the original image and the camera pose as inputs; training a pose estimation model by using the original image and a CNN, inputting an estimated pose into the neural radiation field model to query a corresponding rendering image, and optimizing the pose estimation model by constraining the difference between the rendering image and the original image; and obtaining a relative pose by using the pose estimation model, and obtaining an absolute pose by combining an absolute pose acquisition strategy and a coordinate system conversion scheme. After the training is completed, high-precision positioning can be realized by using only a single image, the reasoning speed is fast, and the storage space required by the model is small.

Description

Technical Field

[0001] This invention relates to the field of large field-of-view variable lighting scenes, and in particular to a single relocalization method for large field-of-view variable lighting scenes based on neural radiation field supervision. Background Technology

[0002] In recent years, with the rapid development of drone technology, drones, due to their small size, high maneuverability, and ease of operation, have been increasingly applied to various scenarios such as firefighting, earthquake rescue, and bridge analysis. Precise positioning is fundamental for drones to achieve obstacle avoidance, path planning, and exploration functions; therefore, all applications place high demands on the control and positioning accuracy of drones. Currently, methods based on externally assisted positioning are widely used, such as the Global Positioning System (GNSS) and optical motion capture using high-resolution cameras. However, GPS fails in densely built-up areas and indoor environments, and optical systems have limited adaptability, leading to the emergence of multi-sensor fusion navigation solutions. The combination of inertial measurement units (IMUs) and Kalman filtering is widely used, but the lack of global information in IMUs can lead to drift errors, affecting the accuracy of long-term positioning. Therefore, vision-based navigation and positioning technology shows greater application potential in drone positioning.

[0003] Vision-based localization tasks are generally divided into real-time mapping localization and offline mapping localization. Real-time mapping localization methods, such as visual SLAM (Simultaneous Localization and Mapping), have advantages such as high real-time performance and low cost. They can accurately acquire camera poses and build maps of the surrounding environment both indoors and outdoors, and are therefore widely used in various localization and mapping tasks. However, these methods require strict conditions to achieve good results, including rich texture details, appropriate lighting intensity, and sufficient overlap between adjacent frames. Therefore, these solutions perform poorly in scenes with changing lighting. In addition, visual SLAM is usually computationally complex, requiring powerful computing platforms, is sensitive to dynamic objects, and accumulates errors and consumes a lot of storage space over time and as the trajectory expands, making it difficult to deploy on mobile devices. At the same time, these solutions mainly support online localization and cannot effectively complete relocalization tasks, thus prompting the development of offline mapping localization technologies.

[0004] To address the pose estimation problem for UAVs (cameras), an offline mapping and localization-based approach first establishes the correspondence between 2D image points and 3D spatial points. Then, it applies the Perspective-n-Point (PnP) algorithm combined with RANSAC for outlier filtering to improve robustness. Traditional pose estimation methods typically use the Structure of Motion (SFM) algorithm to associate each 3D point cloud with a feature operator, extract feature descriptors from the query image, match them with the 3D point cloud, and use the PnP algorithm to solve for the final pose. In recent years, deep learning-based pose estimation algorithms have received widespread attention, exhibiting faster inference speeds and simpler model structures. Some algorithms utilize geometry-based local learning methods to regress scene coordinates, calculating the camera pose by regressing the scene coordinates corresponding to feature points in the query image using a neural network. While these methods offer relatively high accuracy, they also suffer from long training times and slow execution speeds. Furthermore, due to the pre-fixed 3D scene structure and the ease with which the scene coordinate regression network can confuse local scenes, they exhibit poor robustness under varying light levels or in large scenes, and consume significant storage resources.

[0005] To balance storage space and inference time, single-shot relocalization methods based on convolutional neural networks (CNNs) typically train for specific scenes and input a single image during the inference phase to predict the camera's pose relative to that scene through a pose estimation model. These methods are popular on mobile devices due to their fast and reliable inference capabilities, low memory footprint, simple structure, and high efficiency, making them highly portable. Previous research, such as PoseNet, PoseNet2, and Pose-GNN, has focused on improving model accuracy and generalization, or adding geometric constraints through reprojection errors. However, these models often suffer from low accuracy because they largely ignore scene geometry during learning. One possible solution is to introduce high-precision maps to improve localization accuracy, but this approach requires significant storage space for map retrieval and has slow query speeds, lacking portability. Real-time map loading faces challenges such as excessive computation and signal instability, often requiring additional sensors, which contradicts some of the original design principles. Furthermore, the above-mentioned schemes rely too heavily on visual images, resulting in poor adaptability to changes in lighting conditions and poor relocation performance in complex environments. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a single relocalization method for large field-of-view changing scenes based on neural radiation field supervision.

[0007] The technical solution to achieve the objective of this invention is: a single relocalization method for a large field-of-view changing light scene based on neural radiation field supervision, comprising:

[0008] Step 1: Use a camera to acquire wide field-of-view, multi-angle images under varying light conditions;

[0009] Step 2: Estimate the initial pose of the camera using the SFM algorithm;

[0010] Step 3: Use the original image and camera pose together as input to train the neural radiation field model for large field-of-view variable lighting scenes;

[0011] Step 4: Use the original image to train a pose estimation model using CNN, and input the estimated pose into the neural radiation field model obtained in Step 3 to query the corresponding rendered image;

[0012] Step 5: Based on the differences between the rendered images from Steps 3 and 4 and the original images, constrain and optimize the pose estimation model in Step 4 so that it can estimate the camera pose at any position in the entire scene.

[0013] Step 6: Obtain the relative pose using the pose estimation model, and obtain the absolute pose by combining the absolute pose acquisition strategy and coordinate system transformation scheme.

[0014] Furthermore, multi-view images of wide-field-of-view variable-light scenes are captured using cameras or drones. The collected image data is then used to estimate the pose using the SFM algorithm. First, a corresponding point search is performed to find overlapping scenes. Then, the projections of the same points are marked and geometrically verified image pairs are output for feature extraction and feature matching. Finally, images of the same scene parts are found based on the feature set, and a similarity metric is established to obtain the accurate camera pose.

[0015] A large field of view refers to a scene where the distance between the foreground and background exceeds a threshold, resulting in a blurred background if the foreground is focused on during scene modeling, and severe artifacts in the foreground if the background is focused on. Variation in exposure refers to significant differences in exposure caused by different shooting angles.

[0016] Furthermore, step 3 specifically involves: simultaneously feeding the original images obtained in steps 1 and 2, along with the corresponding camera poses, into a pre-designed multilayer perceptron to train a neural radiation field model for a large field-of-view variable lighting scene; during training, dense sampling is performed on the foreground and sparse sampling on the background; and dense sampling is performed on high-frequency information to compensate for the impact of illumination on reconstruction and rendering quality.

[0017] Furthermore, sampling points are obtained according to formula (1), where s' is the sampling point in uniform sampling, n is the nearest point in the scene, and f is the farthest point in the scene:

[0018] (1)

[0019] According to formula (1), s is used as a new sampling point to achieve sampling of dense foreground and sparse background;

[0020] A scene-invariant feature attention module is introduced to intensively sample high-frequency information in the scene, converting a set of input images into a spectrum image, and then using the network to adaptively generate the boundary frequency between high and low frequencies. and In order to adapt to different scenarios; under the sampling method of dense foreground and sparse background, an adaptive sampling weight is assigned to the ray r(s)=o+sd, as shown in formula (2), so as to realize the sampling method of high frequency dense and low frequency sparse.

[0021] w(s) = (2)

[0022] in, This is the low-frequency limit frequency. θ is the high-frequency limit frequency, o is the source point of the ray, and d is the direction of the ray. , , For hyperparameters and > > , , These are the weighting coefficients. To accumulate transparency, For opacity, This is the sampling step size;

[0023] Based on the neural radiation field model, the composite color and depth values ​​are calculated cumulatively using formulas (3) and (4):

[0024] (3)

[0025] (4)

[0026] Where C(s) is the cumulative color, Let c(s) be the cumulative color at the previous sampling point, c(s) be the color value at sampling point s, and D(s) be the cumulative depth. This represents the cumulative depth at the previous sampling point. The sampling step size, The opacity is at sampling point s.

[0027] Furthermore, step 4 specifically involves: using the original image as input, training a pose estimation model using a CNN to obtain the corresponding estimated pose. Since the neural radiation field model has already been trained in step 3, the estimated pose is used to query the corresponding viewpoint rendering image and serve as an important basis for further constraining the pose estimation model.

[0028] Furthermore, step 5 specifically involves: rendering the image based on the neural radiation field model, estimating the pose query, and constraining the rendered image and input image to each other. At this point, each process is differentiable, and a loss function is constructed to further optimize the pose estimation model.

[0029] Furthermore, step 6 specifically involves obtaining the camera pose relative to the scene using a single image. Since the neural radiation field model also has depth estimation capabilities, the absolute camera pose is obtained by combining depth information and coordinate system transformation.

[0030] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0031] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the above-described method.

[0032] Compared with the prior art, the significant advantages of the present invention are:

[0033] (1) Reconstruct large scenes with clear foreground and background, and adapt to changes in light intensity to ensure high-quality reconstruction of large scenes;

[0034] (2) It has photometric generalization during the positioning process, realizing high-precision positioning function across weather conditions;

[0035] (3) After training, high-precision localization can be achieved using only a single image, and the inference speed is fast with a small storage space required by the model. Attached Figure Description

[0036] Figure 1 This is a diagram of a single relocalization system for a large scene with photometric variations supervised by neural radiation field (NeRF).

[0037] Figure 2 This is a schematic diagram of the foreground and background sampling strategies for a scene-invariant feature-based attention-based foreground-background discriminative sampling strategy.

[0038] Figure 3 This is a schematic diagram comparing the effects of lighting on high and low frequency components of a scene.

[0039] Figure 4 This is a schematic diagram of a scene-invariant feature sampling strategy for foreground-background discriminative sampling based on scene-invariant feature attention.

[0040] Figure 5 This is a single relocalization framework for large-scale photometric variation scenes based on NeRF supervision.

[0041] Figure 6Qualitative results for NeRF reconstruction and new view rendering of large-scale outdoor scenes.

[0042] Figure 7 Synthesize new view results from images with varying lighting.

[0043] Figure 8 This provides a qualitative result for a single location test conducted by the UAV.

[0044] Figure 9 This provides qualitative results from a comparative experiment of CNN-based direct regression pose localization schemes.

[0045] Figure 10 The rendering results are for NeRFmm and Nope-NeRF. Detailed Implementation

[0046] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0047] Neural Radiation Field (NeRF) technology has made groundbreaking progress in New View Synthesis (NVS) and 3D reconstruction, laying the foundation for applications such as city mapping and autonomous navigation. It has also spurred a series of studies based on the original NeRF, including accelerated rendering and large-scene rendering. NeRF uses bounding boxes to enclose the scene to be rendered, defines a ray passing through a pixel along the camera center, calculates the 3D position of sampling points along the ray and encodes the position, and finally optimizes by minimizing photometric error, synthesizing a new view through voxel rendering. Subsequent research has optimized NeRF by improving the network structure and adjusting the bounding box size, making it more widely applicable. Inspired by NeRF, this invention combines NeRF technology, which has high-quality 3D reconstruction and new view synthesis, with single-relocalization based on CNN, thereby introducing scene geometric information into the localization model and using photometric loss to further constrain the pose inference model, thus significantly improving the accuracy of pose estimation. Specifically, a NeRF model of a large scene with photometric variations is trained and used as a supervision signal to obtain a high-precision pose inference model. In practical localization inference, the long NeRF training can be omitted. A high-precision pose can be obtained within 0.2 seconds using a pose estimation model based on a single query image, thereby achieving the localization task of the UAV.

[0048] This invention utilizes a foreground-background discriminative sampling strategy to train a high-precision NeRF for large scenes. Subsequently, this NeRF is used to incorporate scene geometric information as supervised training for a pose estimation model. This allows the model to maintain lightweight and fast inference while possessing a certain degree of illumination generalization, achieving centimeter-level positioning accuracy during inference. This scheme can achieve high-precision positioning in large scenes and compensate for illumination variations caused by drone flight angles, weather conditions, etc., and is easier to deploy on mobile devices in terms of computational complexity and storage space. The improvements of this invention are as follows:

[0049] 1) This invention proposes a large-scale scene NeRF modeling method based on a foreground-background discriminative sampling strategy with scene invariant feature attention. This scheme controls the amount of computation in the large-scale scene reconstruction process by sampling the dense foreground and sparse background. It can reconstruct a high-precision large-scale scene NeRF while ensuring reasonable computing resources. At the same time, this strategy makes full use of the invariant features in the scene and can cope with the differences caused by changes in lighting, weather, etc., and has a certain lighting generalization.

[0050] 2) This invention designs a single-shot relocalization framework based on NeRF supervision. This is a passive localization framework that, while utilizing CNN for localization, uses scene-invariant features and scene geometric information captured during the NeRF sampling process as supervision, and further constrains the pose estimation model using photometric loss. This model exhibits illumination generalization during localization and can handle large-scale scene localization problems with varying photometric values.

[0051] 3) The method of this invention does not rely on NeRF, which has a long training time and high computational cost, during inference. It only requires a single query image as input and can obtain centimeter-level positioning accuracy within 0.2 seconds through the pose estimation model. Furthermore, the absolute pose of the camera can be obtained based on the camera's pose relative to the scene. In addition, the lightweight model only occupies megabytes of storage space, making it convenient for deployment on mobile devices.

[0052] In UAV localization tasks, a proper balance between storage space, inference time, and localization accuracy is crucial. A significant weakness in any one of these areas can lead to localization failure or even irreparable damage to the UAV. Current solutions often focus on one aspect at the expense of the other, sacrificing quality in the latter. This invention, however, utilizes Neural Radiation Field (NeRF) technology to incorporate scene geometry and photometric generalization, building upon single-pass relocalization based on Convolutional Neural Networks (CNNs). This improves accuracy while maintaining a fast and lightweight inference process.

[0053] This invention proposes a single-stage relocalization task and training framework for large-scale scenes with photometric variations supervised by Neural Radiation Field (NeRF). During NeRF training, the method synthesizes a high-precision NeRF for large scenes while maintaining reasonable computational load. It also avoids illumination variations caused by weather, drone flight angles, etc., thereby enhancing illumination generalization and robustness. The subsequently designed framework uses NeRF as supervision to train the pose estimation model. During the inference phase, the lightweight pose estimation model can achieve centimeter-level localization accuracy within 0.2 seconds. The overall system is as follows: Figure 1 As shown. Specifically, the method of this invention consists of three parts. In the first stage, a NeRF model for large scenes with varying photometric values ​​is modeled using a foreground-background discriminative sampling strategy based on scene-invariant feature attention, reconstructing a high-precision large-scene NeRF while ensuring reasonable computational cost. In the second stage, a pose estimation model is trained based on a NeRF-supervised single-relocalization framework. This NeRF model incorporates scene geometric information and utilizes its captured invariant features to bring a certain degree of illumination generalization to the pose estimation model, thereby optimizing the pose estimation network and obtaining a high-precision pose estimation model with cross-environment generalization and robustness for this scene. Finally, in the inference stage, only a single query image needs to be input to perform a single relocalization within 0.2 seconds using this lightweight pose estimation model, obtaining centimeter-level high-precision localization results for the UAV (camera) relative to the scene, and the model only occupies tens of megabytes of storage space.

[0054] The method of the present invention will be described in detail below.

[0055] Neural Radiation Field (NeRF) uses a bounding box to surround the objects to be rendered and represents the scene as a continuous particle volume field along the ray. The desired color can be represented as Where o is the ray source point, t is the distance, d is the direction, T(t) is the cumulative transparency, σ is the volume density, and c is the color. During training, the loss function is minimized. To synthesize a high-quality new view, in which Original color, The desired color.

[0056] For small scenes, NeRF achieves extremely high synthesis quality. However, choosing the size of the bounding box in scenes with a clear distinction between foreground and background is often a contradictory problem. If the size is too small, only foreground information is included, resulting in reconstructions with only local details and overall blurriness; if the size is too large, the entire scene is included, leading to poor overall detail in the reconstruction. NeRF++ proposes inverted spherical parameterization to parameterize the unbounded background into bounded parameters, utilizing two multilayer perceptrons (MLPs) to optimize the foreground within the sphere and the background outside the sphere respectively, and dividing the rendering equation into two parts: one inside the sphere and one outside the sphere. Specifically... Figure 2 As shown. For points inside the sphere, we can calculate using the Pythagorean theorem. ,get Where H is the center of the sphere, A is the intersection of the ray and the sphere, and B is the intersection of the perpendicular line drawn from the center of the sphere to the ray and the ray itself. Let t be the ray direction vector. B Let t' be the distance from point B to the origin of the ray, and t' be the distance from a point inside the sphere to the origin of the ray.

[0057] For points outside the sphere It can be parameterized as a quaternion, that is ,in ,and , where are the parameterized coordinates. for Rotate ω to obtain, The direction in which the ray rotates inward toward the sphere. The value of t can be calculated using Rodrigues' rotation formula, and then an improved rendering formula is applied:

[0058]

[0059] A new color is synthesized, where s is the sampling point.

[0060] For large-scale outdoor scenes, one approach is to sample the foreground and background separately and model them using two MLPs to avoid overall blurring and loss of detail. However, NeRF's sampling strategy and rendering mechanism consume a lot of computational resources. Simply sampling the foreground and background separately in the same way as NeRF++ would occupy a lot of video memory, which is impractical. For the foreground part of a scene, it usually needs to have sufficient detailed features, while the background needs to have a clear outline. Therefore, this invention chooses a sampling method with dense foreground and sparse background to achieve high-quality large-scale outdoor scene reconstruction and rendering while ensuring a reasonable amount of computation. The sampling points are obtained according to formula (1), where s' is the sampling point in uniform sampling, n is the nearest point in the scene, and f is the farthest point in the scene:

[0061] (1)

[0062] According to formula (1), using s as the new sampling point allows for dense sampling of the foreground and sparse sampling of the background, thereby significantly reducing the computational load in the NeRF model construction process. Figure 2 As shown in the sampling points, the foreground is sampled densely and the background is sampled sparsely. The different colors are to reflect the effect of light penetration. There is no difference between gray and black sampling points.

[0063] The environmental and lighting changes caused by factors such as drone flight angle, time, and weather conditions further increase the requirements for model accuracy. Furthermore, for offline localization tasks, after training the pose estimation model, it is also necessary to achieve accurate localization under various conditions; therefore, the model's generalization ability to various environments is crucial. The NeRF-based scheme uses photometric loss as a constraint. Experiments show that high-frequency components in the scene are less affected by lighting, while low-frequency components are significantly affected, such as… Figure 3 As shown. Therefore, based on the above foreground-background distinguishing sampling, this invention introduces a scene-invariant feature attention module to perform dense sampling of high-frequency information in the scene, such as... Figure 4 As shown, this is to compensate for the impact of lighting on reconstruction and rendering quality and positioning tasks.

[0064] A set of input images is converted into a spectral image, and a network is used to adaptively generate high-frequency and low-frequency decompositions. and To adapt to different scenarios. Under the foreground-background discriminative sampling method, adaptive sampling weights are assigned along the ray r(s)=o+sd, as shown in formula (2), where, , , For hyperparameters and > > , , These are the weighting coefficients. To accumulate transparency, For opacity, This represents the sampling step size. This enables a high-frequency, dense, and low-frequency, sparse sampling method, giving the method greater generalizability to environmental and lighting conditions.

[0065] w(s) = (2)

[0066] in, This is the low-frequency limit frequency. This is the high-frequency limit frequency.

[0067] The above scheme eliminates low-frequency information redundancy in the image while making better use of invariant information in the scene, thus allowing the reconstruction and rendering process to better generalize to environment and lighting. Based on the NeRF model, the synthesized color and depth values ​​are cumulatively calculated using formulas (3) and (4):

[0068] (3)

[0069] (4)

[0070] Where C(s) is the cumulative color, Let c(s) be the cumulative color at the previous sampling point, c(s) be the color value at sampling point s, and D(s) be the cumulative depth. This represents the cumulative depth at the previous sampling point. This is the sampling step size.

[0071] By employing a foreground-background discriminative sampling strategy based on scene-invariant feature attention, this invention enables reasonable control of computational resources when reconstructing large-scale outdoor scenes using NeRF, allowing for the reconstruction of high-quality scenes with feasible parameter quantities and rendering of high-quality new views, while ensuring generalization of lighting.

[0072] The method of this invention mainly consists of three parts: large-scale scene NeRF reconstruction, training and optimization of the pose estimation model, and single-relocalization pose inference. The overall framework is as follows: Figure 5 As shown.

[0073] First, using NeRF++ as a baseline, a foreground-background discriminative sampling strategy based on scene-invariant feature attention is introduced. This efficiently solves the problem of large-scale outdoor scene reconstruction and rendering with limited computational resources. The NeRF model has illumination generalization properties, effectively handling changes in lighting and weather. The pose estimation model is similar to PoseNet, using a convolutional neural network (CNN) to extract image features and predict the six-DOF camera pose. Compared with the true pose Construct loss function based on the difference between them As one of the supervisory measures, as shown in formula (5):

[0074] (5)

[0075] The pose of the query image I is initially estimated using a pose estimation model. Rendered images are obtained using a neural radiation field model. The above processes are all differentiable; use formula (6) to construct a loss function for the rendered image and the original image. Further monitoring of the network:

[0076] (6)

[0077] Training for large-scale outdoor scenes is expensive under the premise of single relocalization because it requires separate training for the foreground and background. Although not distinguishing between foreground and background can significantly reduce the amount of computation, it will lead to serious artifacts. Therefore, this invention uses a foreground-background discriminative sampling strategy based on scene-invariant feature attention, and this scheme also has a certain degree of illumination generalization. The NeRF model N(P) is defined as Equation (7):

[0078] (7)

[0079] Where F θ Let S(·) be the functional form of NeRF, representing a foreground-background discriminative sampling strategy based on scene-invariant feature attention, to obtain the 5D input of NeRF. (x,y,z) represents the position. For direction.

[0080] During training, rendered images are used. With input image I gt A loss function is constructed to optimize the NeRF model, where N is the NeRF module and P is the corresponding pose, thereby obtaining a high-quality large-scale outdoor scene NeRF reconstruction and new view synthesis task with illumination generalization. Once the NeRF model's accuracy reaches the required level, the NeRF portion is frozen for subsequent training.

[0081] The pose estimation model of this invention is similar to the scheme of single direct regression of pose based on CNN, using CNN to predict the pose of the UAV (camera). Specifically, a pose estimation model is first pre-trained using CNN, and the predicted pose matrix is ​​output using a fully connected layer. It includes rotation matrices. Translation matrix , It is a special European style group. For a Lie algebra of a special Euclidean group, the method of this invention only predicts the camera pose, i.e., the extrinsic parameter matrix, while the intrinsic parameter matrix K is represented by the focal length f and the center point of the camera model according to the pinhole camera model. ,and , , where W and H are the width and height of the image, respectively. Existing methods typically predict quaternions, while the method of this invention directly predicts the pose matrix. This is because the NeRF input is the pose matrix, and it avoids the problem of balancing rotation and translation terms. The NeRF module is used to adjust and optimize the pre-trained model, thereby improving the accuracy of the pose estimation model. Formula (5) is used as a supervised optimization of the pose estimation model, including pre-training and overall framework training.

[0082] First, input image I and use pose estimation model N to obtain the estimated pose. ,Will Input into the NeRF model to obtain By constructing a joint loss using formulas (5) and (6) to jointly supervise the overall framework, an accurate pose estimation model is obtained, as shown in formula (8), where... and Let N be the weight value for each loss term, and N be the pose estimation model.

[0083] (8)

[0084] After training the pose estimation model, only a single query image needs to be input to obtain a positioning result with centimeter-level accuracy within 0.2 seconds. The positioning result is the pose result of the UAV (camera) relative to the scene.

[0085] The camera pose relative to the scene can be obtained using a single query image [R] sc |t sc ], R sc Let t be the rotation matrix of the camera relative to the scene. sc Let be the translation matrix of the camera relative to the scene. Meanwhile, the neural radiation field model can provide the depth information D corresponding to the original image. Therefore, formula (9) is used to calculate the 3D points in the camera coordinate system:

[0086] (9)

[0087] in, and Centered on the camera and Focal length; For 3D points in the camera coordinate system for coordinates For pixels Depth information.

[0088] Then, by transforming the pose in the camera coordinate system to the world coordinate system, we can obtain its corresponding absolute pose, as shown in formula (10):

[0089] (10)

[0090] Among them, P w T is a three-dimensional world coordinate point. wc For the absolute pose of the camera, The rotation matrix in the world coordinate system. This is the translation matrix in the world coordinate system.

[0091] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0092] Example

[0093] This embodiment uses data collected by UAVs in large-scale outdoor scenes as the dataset. A single relocalization framework for large-scale scenes with photometric variations based on NeRF supervision is constructed using PyTorch. For the NeRF part, the initial pose is obtained using COLMAP, which belongs to SE(3). In addition, this embodiment adopts a foreground-background discriminative sampling strategy based on scene-invariant feature attention for sampling, sampling 2048 rays in each iteration, and iterating 200,000 times with a learning rate of 5e-4. For the pose estimation model, ResNet34 is used as the backbone to regress the pose. In the experiment, the learning rate is set to 1e-4, and the learning rate is decayed every 50 epochs with a decay factor of 0.95 and a batch size of 4. In addition, to save training time and computing resources, this embodiment also adopts an early stopping strategy during the pre-training process and sets the patience value to 150. For the overall framework, the formula (8) is set as follows: and The values ​​are 0.3 and 0.7 respectively. The reason for this is... Setting a larger weighting is to improve the model's generalization ability under different lighting conditions. The learning rate is set to 1e-5, and the same early stopping strategy described above is employed. Training in this embodiment was performed entirely on an NVIDIA 3090 graphics card.

[0094] NeRF++ has been shown to produce artifacts in scenes with large depth of field, and in large-scale outdoor scene datasets, it even fails to synthesize new views. This invention, however, introduces a foreground-background discriminative sampling strategy based on scene-invariant feature attention, building upon NeRF++. This reconstructs high-quality NeRF for large-scale outdoor scenes while maintaining reasonable computational cost, and simultaneously renders high-quality new views. Experimental qualitative results are as follows: Figure 6 As shown in Table 1, the quantitative results are as follows.

[0095] The experimental results show that the sampling strategy proposed in this invention can handle large-scale outdoor scenes, converges more easily during training, and can reconstruct a high-quality NeRF model and render a new view with excellent results.

[0096] Table 1 Quantitative Results of NeRF Reconstruction and New View Rendering for Large-Scale Outdoor Scenes

[0097]

[0098] Furthermore, the method of this invention can also address the strong lighting changes in images acquired due to different angles during the flight of the UAV. This approach can synthesize high-quality new views under different exposures, such as... Figure 7 As shown, this demonstrates that the sampling strategy proposed in this invention can save computational resources during training by reducing the number of sampling points and multilayer perceptrons (MLPs), while achieving a NeRF model that responds to illumination changes with performance comparable to NeRF_W, thus demonstrating better practicality for NeRF reconstruction of large-scale outdoor scenes.

[0099] While direct pose regression (CPR) methods, which rely solely on CNNs for single-pass relocalization, offer advantages such as high portability, single-image input, fast inference speed, and low storage requirements, they often suffer from low pose estimation accuracy due to the lack of scene geometry information and generally lack generalization ability regarding illumination. Therefore, this invention proposes a single-pass relocalization method for large scenes with varying light intensity based on NeRF supervision. This invention first uses ResNet34 as the backbone to pre-train a pose estimation model, then incorporates scene geometry information through the NeRF component. Furthermore, rendering new views further supervises the pose estimation model, resulting in higher accuracy. In addition, the model better addresses illumination variations caused by weather conditions and different angles during drone flight. Figure 8 As shown, (a) represents the localization result of the pre-trained model, (b) represents the localization result of the present invention under varying illumination, and (c) represents the localization result of the present invention under ideal conditions. Experimental results demonstrate that the method of the present invention achieves higher localization accuracy compared to the pre-trained model, proving that the method of the present invention significantly improves localization accuracy compared to simply relying on CNNs by incorporating scene geometric information. Furthermore, the method of the present invention also exhibits certain generalization ability under varying illumination conditions.

[0100] This embodiment compares various localization schemes, including neural network-based direct pose regression, scene coordinate-based pose regression, and NeRF-based localization schemes, to demonstrate the advantages of the method of the present invention.

[0101] Compared to various schemes that rely on neural networks for single-shot localization to directly regress pose, these schemes all take a single frame as input. The method of this invention, by introducing an additional large-scale outdoor scene NeRF as supervision to supplement scene geometric information, significantly outperforms other schemes in performance. Furthermore, the method of this invention also significantly outperforms other schemes in terms of illumination generalization performance. Experimental qualitative results are as follows: Figure 9 As shown in Table 2, the quantitative results are as follows: the first row represents scenario_1, the second row represents scenario_2, and the third row represents scenario_3.

[0102] Table 2. Quantitative results of the comparative experiment on pose estimation schemes based on neural networks for single-shot localization and direct regression.

[0103]

[0104] Based on the above results, it can be seen that the single relocalization method for large scenes with photometric variations based on NeRF supervision proposed in this invention utilizes NeRF to introduce scene geometric information and uses it as additional supervision, achieving the highest accuracy in both the estimated rotation and translation matrices. Furthermore, the foreground-background discriminative sampling strategy based on scene-invariant feature attention proposed in this invention can better reconstruct the NeRF model of large outdoor scenes while ensuring reasonable computational load. It also utilizes high-frequency information in the scene to effectively cope with illumination changes, and further constrains the pose estimation model using a NeRF model with a certain degree of illumination generalization, thereby bringing illumination generalization gain to the pose inference stage.

[0105] Scene coordinate regression-based camera pose relocalization methods typically offer high accuracy, but they suffer from slow training speeds and high computational resource consumption. This embodiment selects several scene coordinate regression-based pose relocalization methods, including DSAC++, HSC-Net, VS-Net, and SANet. Quantitative results are shown in Table 3.

[0106] Table 3. Quantitative Results of Pose Localization Based on Scene Coordinate Regression

[0107]

[0108] Based on the above results, it can be seen that the method of this invention relies on pixel alignment between the query image and the rendered image, thus the translation matrix is ​​relatively accurate. In contrast, scene coordinate regression-based camera pose relocalization methods typically construct detailed scene coordinates rather than simply relying on pixel alignment between the query and rendered images, resulting in higher overall accuracy, especially in the rotation matrix. However, these methods require significant training time and computational resources to construct scene coordinates and 3D point clouds. Furthermore, these methods lack generalization and robustness to lighting conditions after the scene is fixed, while the single-shot relocalization framework for large-scale photometric-varying scenes based on NeRF supervision proposed in this invention exhibits better generalization to lighting conditions. In summary, although the method of this invention has only a slight advantage in accuracy compared to scene coordinate regression-based pose relocalization schemes, it has significant advantages in training cost, computational resources, and lighting generalization.

[0109] NeRF-based localization schemes include joint optimization of 3D scene and camera pose methods (represented by NeRFmm and Nope-NeRF) and inverted NeRF schemes (represented by iNeRF and PNeRFP). Specifically, joint optimization schemes do not acquire the initial camera pose but estimate it through the network. Inverted NeRF schemes render the view using a pre-constructed NeRF model; when the query view and the rendered view are aligned, their corresponding poses are the estimated poses. However, these two types of schemes typically require continuous iterative optimization, which is time-consuming (usually tens of hours) and consumes a large amount of computational resources, making them almost impossible to implement in large-scale outdoor scenes. The method of this invention does not rely on iterative optimization and can quickly obtain high-precision localization results using only a single image, making it an efficient localization model. For fairness, the parameters of the two schemes are kept consistent with the NeRF model of this invention. The results of the joint optimization scheme are shown in Table 4, and the results of the inverted NeRF scheme are shown in Table 5.

[0110] Table 4. Quantitative results of comparative experiments on localization schemes based on joint optimization of 3D scene and camera pose.

[0111]

[0112] Table 5. Quantitative results of comparative experiments on localization schemes based on inverted NeRF.

[0113]

[0114] Experimental results show that the camera pose estimation accuracy of the localization scheme based on joint optimization of 3D scene and camera pose is far lower than that of the scheme proposed in this invention. NeRFmm is almost unable to synthesize new views, and the new views synthesized by Nope-NeRF also suffer from severe artifacts. Both of these methods are almost incapable of reconstructing high-quality NeRF models for large-scale outdoor scenes. Figure 10 As shown, jointly optimizing the scene and pose to estimate the pose can lead to significant errors. The accuracy of the NeRF component in the method of this invention results in high precision in the pose inference process. However, localization schemes based on inverted NeRF require substantial computational resources and are almost unsuitable for large-scale outdoor scenes. The rendered new view is completely obscured by artifacts, and it may even be impossible to reconstruct the NeRF model, thus making localization nearly impossible. The method of this invention, however, does not rely on the computationally expensive NeRF component during the inference phase. It can quickly infer a high-precision pose using only a single input image and the pose estimation model, thus avoiding the aforementioned problems.

[0115] This invention proposes a single-shot UAV relocalization framework based on NeRF for large-scale cross-environment scenarios. This framework achieves relocalization tasks through absolute pose regression and outperforms previous direct pose regression methods based on CNN for single-shot localization, demonstrating satisfactory cross-environment generalization. The method of this invention achieves high-precision localization results with illumination generalization while performing fast inference. Furthermore, the method of this invention only requires coordinate transformation to obtain the absolute pose from information such as GPS, making it easier to implement downstream tasks.

Claims

1. A single-shot relocalization method for a large field-of-view scene with varying brightness based on neural radiation field supervision, characterized in that, include: Step 1: Use a camera to acquire wide-field-of-view, multi-view images under varying light conditions, and use these as the original images; Step 2: Estimate the initial pose of the camera using the SFM algorithm; Step 3: Simultaneously feed the original images obtained in Step 1 and Step 2, along with the corresponding initial camera poses, into a pre-designed multilayer perceptron to train the neural radiation field model for the large field-of-view variable light scene; during training, the foreground is densely sampled and the background is sparsely sampled. Dense sampling of high-frequency information compensates for the impact of lighting on reconstruction and rendering quality; Step 4: Use the original image to train a pose estimation model using CNN, and input the estimated pose into the neural radiation field model obtained in Step 3 to query the corresponding rendered image; Step 5: Based on the differences between the rendered images from Steps 3 and 4 and the original images, constrain and optimize the pose estimation model in Step 4 so that it can estimate the camera pose at any position in the entire scene. Step 6: Obtain the relative pose using the pose estimation model, and obtain the absolute pose by combining the absolute pose acquisition strategy and coordinate system transformation scheme.

2. The single relocalization method for a large field-of-view changing scene based on neural radiation field supervision according to claim 1, characterized in that, Multi-view images of wide-field-of-view variable-light scenes are captured using cameras or drones. The collected image data is then used to estimate the pose using the SFM algorithm. First, a corresponding point search is performed to find overlapping scenes. Then, the projections of the same points are marked and geometrically verified image pairs are output for feature extraction and feature matching. Finally, images of the same scene parts are found based on the feature set, and a similarity metric is established to obtain the accurate camera pose.

3. The single relocalization method for a large field-of-view changing light scene based on neural radiation field supervision according to claim 1, characterized in that, The sampling points are obtained according to formula (1), where s' is the sampling point in uniform sampling, n is the nearest point in the scene, and f is the farthest point in the scene: （1） According to formula (1), s is used as a new sampling point to achieve sampling of dense foreground and sparse background; A scene-invariant feature attention module is introduced to intensively sample high-frequency information in the scene, converting a set of input images into a spectrum image, and then using the network to adaptively generate the boundary frequency between high and low frequencies. and In order to adapt to different scenarios; under the sampling method of dense foreground and sparse background, an adaptive sampling weight is assigned to the ray r(s)=o+sd, as shown in formula (2), so as to realize the sampling method of high frequency dense and low frequency sparse. w(s)= (2) in, This is the low-frequency limit frequency. θ is the high-frequency limit frequency, o is the ray source point, and d is the ray direction; , , For hyperparameters and > > , , These are the weighting coefficients. To accumulate transparency, For opacity, This is the sampling step size; Based on the neural radiation field model, the composite color and depth values ​​are calculated cumulatively using formulas (3) and (4): （3） （4） Where C(s) is the cumulative color, Let c(s) be the cumulative color at the previous sampling point, c(s) be the color value at sampling point s, and D(s) be the cumulative depth. This represents the cumulative depth at the previous sampling point. The sampling step size, The opacity is at sampling point s.

4. The single relocalization method for a large field-of-view changing scene based on neural radiation field supervision according to claim 1, characterized in that, Step 4 specifically involves: The original image is used as input, and a pose estimation model is trained using CNN to obtain the corresponding estimated pose. Since the neural radiation field model has been trained in step 3, the estimated pose is used to query the corresponding viewpoint to render the image.

5. The single relocalization method for a large field-of-view changing scene based on neural radiation field supervision according to claim 4, characterized in that, Step 5 specifically involves: The image is rendered based on the neural radiation field model. The rendered image after the pose estimation query and the input image are mutually constrained. At this time, each process is differentiable. The loss function is constructed in this way to optimize the pose estimation model.

6. The single relocalization method for a large field-of-view changing scene based on neural radiation field supervision according to claim 5, characterized in that, Convolutional Neural Networks (CNNs) are used to extract image features to predict six-DOF camera pose. The estimated pose is then used to... Compared with the true pose Construct loss function based on the difference between them As one of the supervisory measures, as shown in formula (5): （5） The pose of the query image I is initially estimated using a pose estimation model. Rendered images are obtained using a neural radiation field model. ; Using formula (6), a loss function is constructed for the rendered image and the original image. Further monitoring of the network: （6）。 7. The single relocalization method for a large field-of-view changing scene based on neural radiation field supervision according to claim 1, characterized in that, Step 6 specifically involves: Obtain the camera pose relative to the scene using a single query image. R sc Let t be the rotation matrix of the camera relative to the scene. sc The neural radiation field model provides the depth information D corresponding to the original image, which is the translation matrix of the camera relative to the scene. The three-dimensional points in the camera coordinate system are calculated using formula (7): （7） in, and Centered on the camera and Focal length; A 3D point in the camera coordinate system. for coordinates For pixels Depth information; Then, by transforming the pose in the camera coordinate system to the world coordinate system, we can obtain its corresponding absolute pose, as shown in formula (8): （8） Among them, P w T is a three-dimensional point in the world coordinate system. wc For the absolute pose of the camera, The rotation matrix in the world coordinate system. This is the translation matrix in the world coordinate system.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

Images