Method and device for constructing urban airspace map based on image acquisition equipment

Abstract

The application provides a kind of based on the construction method and device of city airspace map of image acquisition equipment, comprising: the image of the city airspace to be constructed is collected by image acquisition equipment and the pose corresponding to the image, obtain image set, according to the image set obtain neural radiation field, using the preset optimization strategy is adjusted to the neural radiation field, obtain the preliminary pose of the image;The image set and the preliminary pose are input to the common view area search module constructed in advance, image similarity search is carried out, obtain image pair, according to the image pair calculate corresponding the relative transformation of the neural radiation field, obtain the rough relative pose of the image pair;According to the image pair and the two neural radiation fields corresponding thereto, the rough relative pose is fine-tuned, to obtain accurate relative pose;According to the accurate relative pose, all the neural radiation fields are fused to obtain the target map.The pose estimation, registration and fusion result obtained by the application are high in accuracy.

Description

Technical Field

[0001] This invention relates to the field of positioning and navigation technology, and in particular to a method and apparatus for constructing urban airspace maps based on image acquisition devices. Background Technology

[0002] The construction of low-storage, high-precision maps and the implementation of corresponding positioning schemes are the technological foundation for UAVs to effectively and safely conduct flights in complex urban low-altitude airspace. However, precisely because of the complexity of this special environment, the development of UAVs faces a series of severe challenges, including positioning difficulties, insufficient accuracy, high computing power requirements, and low efficiency. Against this backdrop, Neural Radiance Fields (NeRF) have been widely used in UAV mapping and navigation due to their ability to achieve highly realistic rendering while maintaining a compact model structure. The use of distributed NeRF to model a wide range of urban environments has attracted considerable attention.

[0003] However, existing methods suffer from the core problem of aliasing caused by differences in rendering resolution and suboptimal pose accuracy, resulting in poor robustness of the final pose estimation and problems such as occlusion artifacts during the fusion stage. Summary of the Invention

[0004] This invention provides a method and apparatus for constructing urban airspace maps based on image acquisition devices, which solves the defects of poor robustness of pose estimation and occlusion artifacts in the fusion stage in the prior art, and achieves more robust and accurate pose estimation, registration and fusion.

[0005] This invention provides a method for constructing an urban airspace map based on an image acquisition device, comprising:

[0006] Images of the urban airspace to be constructed and their corresponding poses are acquired using image acquisition devices to obtain an image set. A neural radiation field is obtained based on the image set, and the neural radiation field is bundled and adjusted using a preset optimization strategy to obtain the preliminary pose of the image. The number of image acquisition devices is not less than two.

[0007] The image set and the preliminary pose are input into a pre-built common-view region retrieval module to perform image similarity search, obtain image pairs, and calculate the relative transformation of the corresponding neural radiation field based on the image pairs to obtain the coarse relative pose of the image pairs.

[0008] Based on the image pair and its corresponding two neural radiation fields, the coarse relative pose is finely adjusted to obtain the accurate relative pose;

[0009] Based on the accurate relative pose, all the neural radiation fields are fused to obtain the target map.

[0010] According to the present invention, a method for constructing an urban airspace map based on an image acquisition device, wherein the step of inputting the image set and the preliminary pose into a pre-constructed common-view region retrieval module to perform image similarity search and obtain image pairs specifically includes:

[0011] Extract feature vectors from the image set, and establish an index based on the feature vectors;

[0012] Based on the index, a preset search algorithm is used to retrieve the most similar set of images, resulting in a first image and a second image, thus obtaining the image pair.

[0013] According to the present invention, a method for constructing an urban airspace map based on an image acquisition device, wherein calculating the relative transformation of the corresponding neural radiation field based on the image pair to obtain the coarse relative pose of the image pair specifically includes:

[0014] Extract the preliminary poses of the first image and the second image from the image pair to obtain the first preliminary pose and the second preliminary pose;

[0015] Starting with the second preliminary pose as the initial estimation of the relative pose, based on the pre-built frame-to-model estimation model, the pose of the current frame is optimized by minimizing the difference between the rendered image of the second neural radiation field corresponding to the second image and the first image, thus obtaining the coarse relative pose of the image pair.

[0016] According to the present invention, a method for constructing an urban airspace map based on an image acquisition device, wherein the step of fine-tuning the coarse relative pose based on the image pair and its corresponding two neural radiation fields to obtain the accurate relative pose specifically includes:

[0017] Based on the first neural radiation field corresponding to the first image in the image pair, the pose of the target image acquisition device is queried to obtain the first target pose; based on the second neural radiation field corresponding to the second image in the image pair, the pose of the target image acquisition device is queried to obtain the second target pose.

[0018] A composite image is obtained by rendering based on the first target pose, the second target pose, and the coarse relative pose;

[0019] Based on the frame-to-model estimation model, the accurate relative pose is obtained from the synthesized image.

[0020] According to the present invention, a method for constructing an urban airspace map based on an image acquisition device, wherein fusing all neural radiation fields according to the accurate relative pose to obtain a target map specifically includes:

[0021] The weights for each of the neural radiation fields are determined using inverse distance weighting;

[0022] The target map is obtained by fusing the neural radiation field based on the weights and the accurate relative pose.

[0023] According to the present invention, a method for constructing an urban airspace map based on an image acquisition device, wherein obtaining a neural radiation field from the image set specifically includes:

[0024] Camera parameters are calculated using the images in the image set and the corresponding poses;

[0025] Generate sampling rays based on the camera parameters;

[0026] Based on the sampled rays, image rendering is performed using a multilayer perceptron to obtain the neural radiation field corresponding to the image set.

[0027] The present invention also provides an apparatus for constructing urban airspace maps based on image acquisition devices, comprising:

[0028] A preliminary pose unit is used to acquire images of the urban airspace to be constructed and the poses corresponding to the images through an image acquisition device to obtain an image set, obtain a neural radiation field based on the image set, and perform bundled adjustment on the neural radiation field using a preset optimization strategy to obtain the preliminary pose of the image; wherein, the number of the image acquisition devices is not less than two.

[0029] The coarse relative pose unit is used to input the image set and the preliminary pose into a pre-constructed common-view region retrieval module, perform image similarity search, obtain image pairs, calculate the relative transformation of the corresponding neural radiation field based on the image pairs, and obtain the coarse relative pose of the image pairs.

[0030] The accurate relative pose unit is used to fine-tune the coarse relative pose based on the image pair and its two corresponding neural radiation fields to obtain the accurate relative pose.

[0031] The fusion unit is used to fuse all the neural radiation fields according to the accurate relative pose to obtain the target map.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for constructing an urban airspace map based on an image acquisition device as described above.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing an urban airspace map based on an image acquisition device as described above.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for constructing an urban airspace map based on an image acquisition device as described above.

[0035] The present invention provides a method and apparatus for constructing urban airspace maps based on image acquisition devices. The method involves acquiring images of the urban airspace to be constructed and their corresponding poses using image acquisition devices to obtain an image set. A neural radiation field is obtained from the image set, and a preset optimization strategy is used to bundle and adjust the neural radiation field to obtain a preliminary pose of the images. The number of image acquisition devices is not less than two. The image set and the preliminary pose are input into a pre-constructed common-view region retrieval module for image similarity search to obtain image pairs. The relative transformation of the corresponding neural radiation field is calculated based on the image pairs to obtain a coarse relative pose of the image pairs. The coarse relative pose is fine-tuned based on the image pairs and their corresponding two neural radiation fields to obtain an accurate relative pose. All neural radiation fields are fused based on the accurate relative pose to obtain a target map. This invention acquires an image set of the urban airspace to be constructed using an image acquisition device, trains a neural radiation field based on the image set, and then optimizes the neural radiation field in three progressive stages, including bundle adjustment of individual neural radiation fields, frame-to-model optimization, and model-to-model optimization. Finally, the optimized scene representation and pose are fused. The pose estimation, registration, and fusion results obtained through the above steps have high accuracy. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is one of the flowcharts illustrating the method for constructing an urban airspace map based on an image acquisition device provided by the present invention;

[0038] Figure 2 This is the second flowchart illustrating the method for constructing an urban airspace map based on an image acquisition device provided by the present invention.

[0039] Figure 3 This is a comparison of registration results from an embodiment of the urban airspace map construction method based on image acquisition equipment provided by the present invention;

[0040] Figure 4This is a comparison image of the fusion results of an embodiment of the method for constructing urban airspace maps based on image acquisition devices provided by the present invention;

[0041] Figure 5 This is a schematic diagram comparing the method for constructing urban airspace maps based on image acquisition equipment provided by this invention with other different types of truncated dynamic low-pass filters.

[0042] Figure 6 This is a schematic diagram of the structure of the urban airspace map construction device based on image acquisition equipment provided by the present invention;

[0043] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0044] Figure label:

[0045] 610: Preliminary pose unit; 620: Coarse relative pose unit; 630: Accurate relative pose unit; 640: Fusion unit. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] The following is combined with Figures 1-5 This invention describes a method for constructing urban airspace maps based on image acquisition devices. Figure 1 This is one of the flowcharts illustrating the method for constructing an urban airspace map based on an image acquisition device provided by the present invention, such as... Figure 1 As shown, the method includes:

[0048] Step 110: Acquire images of the urban airspace to be constructed and the corresponding poses of the images using image acquisition devices to obtain an image set. Obtain a neural radiation field based on the image set. Use a preset optimization strategy to bundle and adjust the neural radiation field to obtain the preliminary pose of the image. The number of image acquisition devices is not less than two.

[0049] It is important to emphasize that the image acquisition device used in this invention is capable of acquiring images of the airspace map. In some embodiments, the image acquisition device includes a drone. This invention leverages the high maneuverability of drones to address the bottleneck technical problems existing in the efficient flight of current drones in densely built-up urban environments.

[0050] It should be emphasized that the present invention uses an image acquisition device to acquire images of the urban airspace to be constructed and the pose of the image acquisition device at the time of image acquisition, as the basis for constructing a map of the urban airspace to be constructed.

[0051] In some embodiments, the pose of the image acquisition device during image acquisition is the pose calibrated by COLMAP.

[0052] After obtaining the image of the urban airspace to be constructed and its corresponding pose, the images acquired by each image acquisition device form an image set, and the neural radiation field is obtained from the image set. It should be noted that the image acquisition devices involved in this invention are at least two. It is understood that the neural radiation field synthesizes highly realistic scene rendering by encoding the volume density and color of the scene within the weight range of a coordinate-based multilayer perceptron (MLP). Furthermore, this invention does not limit the method of obtaining the neural radiation field from the image set; the method of constructing the neural radiation field can be selected according to actual needs. In some embodiments, obtaining the neural radiation field from the image set specifically includes:

[0053] Camera parameters are calculated using the images in the image set and the corresponding poses;

[0054] Generate sampling rays based on the camera parameters;

[0055] Based on the sampled rays, image rendering is performed using a multilayer perceptron to obtain the neural radiation field corresponding to the image set.

[0056] Specifically, during the training process, camera parameters are incorporated into the training of the neural radiation field model to jointly optimize the neural radiation field using camera parameters and the 3D scene.

[0057] It is understood that incorporating camera parameters into the training process of the neural radiation field model is an important improvement direction of this invention. Studies have shown that the model trained in this way outperforms the model without synchronous joint optimization in terms of image quality, and achieves better results in terms of average PSNR (Peak Signal-to-Noise Ratio), average SSIM (Structural Similarity), and average LPIPS (Learned Perceptual Image Patch Similarity).

[0058] In the specific implementation process, firstly, an image set {I} and a pose {p} calibrated by COLMAP are given. The image set includes N images [I] i Then, the camera parameters were optimized.

[0059] In one specific embodiment, camera parameters are first defined: two trainable parameters are created using the nn.param function, representing the camera rotation [φ] and translation [t] for the N images. These parameters are variables learned by the model during training and are used to estimate the camera pose.

[0060] Next, the Mip-NeRF-360 model is defined: a Mip-NeRF-360 model is created using Mip-NeRF-360_Module(). This model consists of multiple layers of neural networks and is used to learn the complex mapping relationship between images and camera parameters.

[0061]

[0062]

[0063] F θ =Mip-NeRF-360_Module()

[0064] After obtaining the camera parameters, sampled rays are generated from the camera parameters.

[0065] Specifically, when generating a sampling ray, a ray emanating from the camera position can be constructed using camera parameters and pixel positions. This process includes the following steps:

[0066] 1) Convert pixel coordinates: Convert pixel coordinates to normalized coordinates, usually within the range [0,1].

[0067] 2) Generate ray direction: Calculate the ray direction using the camera's intrinsic and extrinsic parameters, as well as normalized pixel coordinates. This can be achieved through back projection.

[0068] It is important to note that the origin of the ray also needs to be determined, that is, the position of the camera is used as the origin of the ray.

[0069] Ultimately, a parameterized ray can be obtained, that is, the ray parameters can be confirmed: its origin and direction are determined by the camera parameters and pixel positions.

[0070]

[0071] The representation of the view cone only requires introducing the radius of the cone at the ray position. The sampled ray is passed to a multilayer perceptron, and finally the pixel value {Cj} corresponding to the ray is generated using the volume rendering formula, and combined into the rendered image to obtain the neural radiation field.

[0072]

[0073] In some embodiments, Mip-NeRF 360 cleverly addresses aliasing issues when rendering at different resolutions, as a compact representation of the scene, and the ambiguity involved in reconstructing 3D content from 2D images in large-scale, borderless environments. This embodiment is trained directly based on the model parameters of Mip-NeRF 360. It is understood that Mip-NeRF-360 does not incorporate camera pose into the parameters to be optimized during training, causing it to rely on an image pose dataset built using SfM technology. However, these techniques still cannot achieve completely accurate pose calibration.

[0074] It is understood that during training, the error function includes photometric loss, the correctness of scene reconstruction geometry, and parameters measuring the performance of a small-scale network. During training, the photometric loss needs to participate in the camera pose update and is the sole loss term in the pose optimization phase of this paper; the correctness of the scene reconstruction geometry also needs to participate in the camera pose update; the parameters measuring the performance of the small-scale network are only used for model training and do not participate in the camera pose update.

[0075] To obtain an accurate target map, this invention mainly comprises two tasks: neural radiation field registration and neural radiation field mixing. In the neural radiation field registration stage, we employed a three-stage pose optimization to achieve robust registration results.

[0076] In the first stage, a neural radiation field obtained by bundle adjustment training is introduced to achieve joint optimization of scene representation and pose of the input image.

[0077] This embodiment performs joint optimization of camera pose during training to improve its consistency with the scene and reduce rendering errors and subsequent localization errors based on it. The motivation for joint optimization of NeRF (Neural Radiance Fields) is to improve the quality of novel view synthesis and scene reconstruction. By optimizing the camera pose and the NeRF model, the resulting synthesized image is more accurate and realistic. The importance of joint optimization lies in its ability to enable end-to-end NeRF model training. These advantages have significant implications for applications such as virtual reality, augmented reality, and computer graphics. For localization tasks, calibrated camera pose also means that the results obtained when optimizing the pose using the model in reverse are more accurate.

[0078] After obtaining the neural radiation field, the neural radiation field is bundled and adjusted using a preset optimization strategy to obtain the preliminary pose of the image.

[0079] Specifically, since errors in camera extrinsics are unavoidable and can affect the quality of scene representation, this invention optimizes both scene representation and camera extrinsics by minimizing a weighted combination of distortion loss, aberration loss, and proposal loss.

[0080]

[0081] Where, Θ * T represents the parameters of the neural radiation field network. * Indicates camera external parameters. Indicates distortion loss. Indicates distortion loss. This indicates a loss incurred by the proposal.

[0082] During training, a coarse-to-fine optimization approach is applied, primarily focusing on positional encoding. This embodiment differs from constructing positional encoding (PE) features from a tiny point in space; instead, it constructs an integrated positional encoding (IPE) representation of the volume covered by each cone truncated cone. However, similar to PE, the initial optimization phase using IPE can also lead to the model getting stuck in local optima due to excessive high-frequency influences on the gradient. Therefore, a coarse-to-fine optimization strategy is introduced by adding a weight layer ω(α), where the weight ω of the k-th frequency component is:

[0083]

[0084] α∈[0,L] is related to training progress, when ω k When (α) is set to 0, gradient contributions from the k-th and higher frequency components are effectively neutralized. Starting from α = 0, the encoding is gradually activated until full positional encoding is achieved at α = l. This approach helps our model to initially align with a smoother signal, paving the way for subsequent capture of high-resolution scene representations.

[0085] Step 120: Input the image set and the preliminary pose into the pre-constructed common-view region retrieval module to perform image similarity search, obtain image pairs, calculate the relative transformation of the corresponding neural radiation field based on the image pairs, and obtain the coarse relative pose of the image pairs.

[0086] After obtaining the initial pose, in the second stage of optimization, the common-view region retrieval module is used to search for similar images. Similar image pairs are identified in the image set of each image acquisition device, and the image pairs composed of the most similar images are used for further pose optimization.

[0087] In some embodiments, inputting the image set and the preliminary pose into a pre-built common-view region retrieval module to perform image similarity search and obtain image pairs specifically includes:

[0088] Extract feature vectors from the image set, and establish an index based on the feature vectors;

[0089] Based on the index, a preset search algorithm is used to retrieve the most similar set of images, resulting in a first image and a second image, thus obtaining the image pair.

[0090] Specifically, a common-view region retrieval module is constructed using deep learning and efficient similarity search methods. Feature extraction is performed through deep learning to extract feature vectors for each image in the image set. In some embodiments, given the efficiency of the pre-trained ResNet-50 model in distinguishing complex feature levels from images, the pre-trained ResNet-50 model is adopted in the common-view region retrieval module. By adjusting the ResNet-50 architecture and omitting its final fully connected layer, we seek a high-dimensional representation of the image that concisely summarizes its core visual attributes.

[0091] After obtaining the feature vectors, these vectors are used to build an index in an approximate nearest neighbor search library. In some implementations, the FAISS library is chosen as the approximate nearest neighbor search library. It is understood that the FAISS library is an open-source clustering and similarity search library from the Facebook AI team, providing efficient similarity search and clustering for dense vectors, supporting searches of billions of vectors, and is currently the most mature approximate nearest neighbor search library.

[0092] Subsequently, based on the constructed index, a preset search algorithm is used for retrieval, thereby finding the most similar first and second images from multiple image sets to form image pairs. It is important to emphasize that this invention does not limit the number of image pairs; all photos in an image set can form image pairs with the most similar photos from other image sets. Furthermore, in some embodiments, in terms of similarity search, the extracted feature vector of a given query image is used to explore the index, and the most similar image in the image set is identified based on the Euclidean distance between the feature vectors.

[0093] In one specific embodiment, the preset search algorithm adopts the Hierarchical Navigable Small World (HNSW) search algorithm, which maintains a harmonious balance between speed, accuracy, memory efficiency and scalability, and is suitable for retrieving high-dimensional vectors.

[0094] After obtaining the image pairs, the common viewing regions among multiple image sets are found. Then, the relative transformation between the corresponding neural radiation fields is calculated to obtain a coarse relative pose. Understandably, since different image acquisition devices define their own local coordinate systems, a transformation matrix from local to global is needed to achieve seamless blending between models. The purpose of calculating the coarse relative pose is to obtain this local-to-global transformation matrix.

[0095] In some embodiments, calculating the relative transformation of the corresponding neural radiation field based on the image pair to obtain the coarse relative pose of the image pair specifically includes:

[0096] Extract the preliminary poses of the first image and the second image from the image pair to obtain the first preliminary pose and the second preliminary pose;

[0097] Starting with the second preliminary pose as the initial estimation of the relative pose, based on the pre-built frame-to-model estimation model, the pose of the current frame is optimized by minimizing the difference between the rendered image of the second neural radiation field corresponding to the second image and the first image, thus obtaining the coarse relative pose of the image pair.

[0098] Specifically, given a specific set of image pairs (I i ,I j Extract the relevant pose T i and T j It needs to be emphasized that T at this time... i and T j This is the initial pose after the first stage of optimization. For ease of understanding, it needs to be explained that I... i Corresponding to the first image, I j Corresponding to the second image, T i Corresponding to the first initial pose, T j Corresponding to the second initial pose; NeRF Corresponding to the first neural radiation field, NeRF This corresponds to the second neural radiation field. The terms "first" and "second" are used only to distinguish different images, preliminary poses, and neural radiation fields, and do not imply any order or other meanings. Naturally, images, preliminary poses, and neural radiation fields with the same number correspond to the same image set. Furthermore, here (I i ,T i ) and (I j ,T j These correspond to the camera model that acts as a coordinate transformation bridge and acquires the image, that is, the corresponding image acquisition device.

[0099] For ease of explanation, the first image and the first preliminary pose (I) are used. i ,T i The corresponding camera C) k For example, a pre-built frame-to-model estimation model is used to locate the camera in NeRF. That is, in the second neural radiation field, the first image I i Acting as an observer, second preliminary pose T j Acting as relative pose T j *The initial guess is used to obtain a rough relative pose in order to achieve a robust and accurate pose.

[0100] In some embodiments, the pre-built frame-to-model estimation model iMNeRF is constructed by combining an inverted Mip-NeRF360 with a truncated dynamic low-pass filter. Specifically, the inverted Mip-NeRF360 includes a truncated dynamic low-pass filter, meaning the pose is first passed through the truncated dynamic low-pass filter before being input into the Mip-NeRF360 to render the image, thereby optimizing the pose through photometric errors.

[0101] It should be noted that since Mip-NeRF 360 primarily addresses the anti-aliasing problem in unbounded scenes, this invention mainly uses it as an example for illustration. However, this does not constitute a limitation of the invention; other neural radiation field models can also be used to implement the technical solutions of this invention. When implementing with other neural radiation field models, the design of the truncated low-pass filter in the frame-to-model estimation model will change accordingly.

[0102] Understandably, the goal of the frame-to-model estimation model is to minimize the observation of the first image I. i With the second neural radiation field NeRF Rendered image To optimize T based on the photometric loss between j In other words, from the second initial pose T j As T j * Starting with the initial estimation, based on a NeRF model with fixed parameters. By minimizing the rendered image with I i The differences between the frames are used to optimize the pose of the current frame. Therefore, the second-stage optimization can also be called frame-to-model optimization, or Frame2Model optimization. In this stage, a coarse-to-fine strategy is used to ensure that local optima are avoided, and the results of the second-stage registration are obtained.

[0103] Furthermore, in some embodiments, to make the optimization smoother and converge faster, optimization is performed on the tangent plane. The optimization steps can be described as follows:

[0104]

[0105] in, Describe the Lie algebra of the three-dimensional Euclidean motion group.

[0106] What is known is that The elements of ξ can typically be represented as a six-dimensional vector ξ, which includes rotational components along three rotation axes and three translational components. This representation can usually be written as:

[0107]

[0108] Here, ω is a three-dimensional rotation vector representing the direction and angle of the rotation axis, while v is a three-dimensional translation vector.

[0109] This lowercase It is typically used to describe minute changes in the motion of rigid bodies.

[0110] Because the neural radiation field trained by the method provided by this invention has an excellent ability to render at different resolutions without producing image aliasing, it can still generate high-quality images even when the viewing angle from the image acquisition device changes significantly, thereby expanding the potential application areas of this invention.

[0111] The Integrated Position Encoding (IPE) used in this invention improves the accuracy of high-frequency position encoding while preserving high-frequency information of the scene. This leads to two key design decisions in this invention: First, a coarse-to-fine strategy helps avoid local optima. A truncate dynamic low-pass filter (TDLF) is incorporated into the IPE to avoid invalid outputs at low-frequency thresholds. Second, no gradual modification is necessary for high-frequency encoding; only some high frequencies need to be suppressed in the initial stage. After reaching a set iteration threshold, all high-frequency information can be released. Figure 5 As shown, Figure 5 (a) A truncated dynamic low-pass filter for beam-adjusted Mip-NeRF360. Figure 5 (b) is the truncated dynamic low-pass filter of LATITUDE. Figure 5 (c) is a truncated dynamic low-pass filter according to an embodiment of the present invention. The present invention only needs to suppress some high frequencies in the initial stage, without the need to progressively adjust the high-frequency coding as in the filters in LATITUDE.

[0112] weight ω k for:

[0113]

[0114] Here, α∈[0,L] is changed when the threshold τ=0.8 is reached, and the empirical value of α0 is 0.6.

[0115] Furthermore, in determining T j * Then, the transformation matrix T can be determined. ij .

[0116] Among them, T ji =T j * (T i )-1 Similarly, in NeRF Perform the same steps in the middle to locate (I) j ,T j The camera can also get T. ij =T i * (T j ) -1 .

[0117] Then, from the new input perspective, the camera pose T i q That is, starting with other highly similar image pairs, the previous pose optimization is repeatedly performed. For each iteration, |T| is calculated. ji T ij -E|, where E is the identity matrix. The transformation T that minimizes this metric is... ji It can be used as a rough estimate of the relative pose between different neural radiation fields, i.e., a rough relative pose.

[0118] Step 130: Fine-tune the coarse relative pose based on the image pair and its corresponding two neural radiation fields to obtain the accurate relative pose.

[0119] After obtaining the coarse relative pose, the third stage of optimization is carried out. The coarse relative pose between the distributed neural radiation fields obtained from the registration results of the second stage can be used as a priori to transform the sampling points of different neural radiation fields to a unified coordinate system.

[0120] In some embodiments, the step of fine-tuning the coarse relative pose based on the image pair and its corresponding two neural radiation fields to obtain the accurate relative pose specifically includes:

[0121] Based on the first neural radiation field corresponding to the first image in the image pair, the pose of the target image acquisition device is queried to obtain the first target pose; based on the second neural radiation field corresponding to the second image in the image pair, the pose of the target image acquisition device is queried to obtain the second target pose.

[0122] A composite image is obtained by rendering based on the first target pose, the second target pose, and the coarse relative pose;

[0123] Based on the frame-to-model estimation model, the accurate relative pose is obtained from the synthesized image.

[0124] Specifically, after obtaining the coarse relative pose, the sampling points of different neural radiation fields are then converted into a unified global coordinate system. At this point, a more precise transformation matrix is ​​still needed when rendering given the camera pose. Therefore, a third stage of final optimization is required before blending.

[0125] Given the first neural radiation field NeRF The trained model takes the input query as the sampled camera pose. After applying this model, a set of synthetic images can be generated. Next, the first target pose Multiply by the result calculated in the previous step The second target pose can then be obtained. For the second target pose The rendered image. Then, the frame-to-model estimation model is used again, based on minimizing the synthesized image. and The error was optimized to obtain an accurate relative pose.

[0126] After completing the third stage of optimization, the neural radiation field registration task is finished, and the next step is the neural radiation field mixing task. It is understood that obtaining the final [data / data / etc.]... Afterward, the rendered images will be closely aligned, providing a foundation for facilitating the blending of large-scale scenes.

[0127] In one specific embodiment, Figure 3 A comparison chart of registration results performed according to the three-stage optimization method provided in this invention is given, as follows: Figure 3 As shown, this embodiment compares the registration results of two existing methods and the present invention based on the differences in error scales between real-world and simulator data. The first existing method is LATITUDE, which is currently the state-of-the-art (SOTA) method for PE-based NeRF registration. The second existing method is iNGP, a NeRF registration method based on hash coding. The method provided by the present invention is called iMNeRF. Results show that LATITUDE's performance significantly decreases during image acquisition device rise and fall. This is because LATITUDE's network architecture, derived from Mega-NeRF, cannot handle aliasing issues at different resolutions, leading to performance degradation. For iNGP, due to its explicit coding method, the overall image quality is very good. However, some floating artifacts affect the localization results. The present invention, on the other hand, demonstrates superior performance overall, especially in real-world data scenarios.

[0128] Step 140: Based on the accurate relative pose, fuse all the neural radiation fields to obtain the target map.

[0129] Once the pose optimization of all three stages is completed, the registration task is finished. Next, the neural radiation field will be mixed to obtain the target map.

[0130] In some embodiments, fusing all the neural radiation fields according to the accurate relative pose to obtain the target map specifically includes:

[0131] The weights for each of the neural radiation fields are determined using inverse distance weighting;

[0132] The target map is obtained by fusing the neural radiation field based on the weights and the accurate relative pose.

[0133] Specifically, when using multiple neural radiation fields to render an image, inverse distance weighting (IDW) can be used to determine the contribution of each neural radiation field. For example... Figure 2 As shown on the far right, from top to bottom, the first two images are rendered by two distributed NeRFs with the same pose, and the bottom image is the result after blending.

[0134] It is understood that IDW considers the back projection of the pixels to be rendered onto the rays for merging. In some embodiments, when the neural radiation field is Mip-NeRF 360, since Mip-NeRF 360 is based on Gaussian sampling, the sampling results are interpreted as the midpoints of the intervals, and then they are merged.

[0135] Furthermore, from NeRF and NeRF The merged sample composed of the synthetic image z obtained from the output. (After k is the point index on the merge ray), the distance from the merged sample point to each NeRF origin can be obtained. Further, the distance is calculated based on the weight ω of the NeRF points according to the distance ratio. k By filtering the contributions point by point, we obtain the contribution at that point:

[0136]

[0137] Where, d i,k Indicates the sampling point to NeRF The distance, d j,k Indicates the sampling point to NeRF The distance.

[0138] The neural radiation fields are fused according to their respective weights to obtain the target map.

[0139] The method for constructing urban airspace maps based on image acquisition devices provided by this invention includes three pose optimization stages. In the first stage, a preliminary pose of the image is obtained by bundling and adjusting the neural radiation fields using a coarse-to-fine strategy. In the second stage, a robust and accurate pose is achieved through a common-view region retrieval module, referred to as Frame2Model optimization. Based on this, a coarse transformation between neural radiation fields in different coordinate systems is obtained. In the third stage, the transformation between neural radiation fields is fine-tuned through Model2Model pose optimization. After obtaining accurate transformation parameters, the distributed neural radiation fields are fused, demonstrating optimal performance in both real-world and simulation data, and possessing significant practical value.

[0140] In one embodiment, such as Figure 4 As shown, this embodiment benchmarks NeRFacto based on ngp (NeRFacto also runs in a joint optimization mode for scene representation and pose), as follows. Figure 4 As shown, the visualization results demonstrate the fusion of images synthesized from the new perspective. NeR-Fuser, using NeRFacto to train a distributed NeRF, exhibits inaccuracies in Model-2Model registration. Therefore, the images synthesized from the new viewpoint cannot be seamlessly aligned within a unified coordinate system. Consequently, the final fusion result is not optimal. In contrast, the framework provided by this invention benefits from the superior registration results after three-stage optimization, achieving better fusion results. It can be observed that the final fused image effectively integrates the advantages of the two distributed images and compensates for their shortcomings, thereby improving performance.

[0141] The present invention provides a method for constructing an urban airspace map based on an image acquisition device. The method involves acquiring images of the urban airspace to be constructed and their corresponding poses using the image acquisition device to obtain an image set. A neural radiation field is obtained from the image set, and a preset optimization strategy is used to bundle and adjust the neural radiation field to obtain a preliminary pose of the images. The number of image acquisition devices is not less than two. The image set and the preliminary pose are input into a pre-constructed common-view region retrieval module for image similarity search to obtain image pairs. The relative transformation of the corresponding neural radiation field is calculated based on the image pairs to obtain a coarse relative pose of the image pairs. The coarse relative pose is fine-tuned based on the image pairs and their corresponding two neural radiation fields to obtain an accurate relative pose. All neural radiation fields are fused based on the accurate relative pose to obtain the target map. This invention acquires an image set of the urban airspace to be constructed using an image acquisition device, trains a neural radiation field based on the image set, and then optimizes the neural radiation field in three progressive stages, including bundle adjustment of individual neural radiation fields, frame-to-model optimization, and model-to-model optimization. Finally, the optimized scene representation and pose are fused. The pose estimation, registration, and fusion results obtained through the above steps have high accuracy.

[0142] The following describes the urban airspace map construction apparatus based on the image acquisition device provided by the present invention. The urban airspace map construction apparatus based on the image acquisition device described below and the urban airspace map construction method based on the image acquisition device described above can be referred to in correspondence. Figure 6 This is a schematic diagram of the structure of the urban airspace map construction device based on image acquisition equipment provided by the present invention, as shown below. Figure 6 As shown, the device includes:

[0143] The preliminary pose unit 610 is used to acquire images of the urban airspace to be constructed and the poses corresponding to the images through an image acquisition device to obtain an image set, obtain a neural radiation field based on the image set, and perform bundle adjustment on the neural radiation field using a preset optimization strategy to obtain the preliminary pose of the image; wherein, the number of the image acquisition devices is not less than two.

[0144] The coarse relative pose unit 620 is used to input the image set and the preliminary pose into a pre-constructed common-view region retrieval module, perform image similarity search, obtain image pairs, calculate the relative transformation of the corresponding neural radiation field based on the image pairs, and obtain the coarse relative pose of the image pairs.

[0145] The accurate relative pose unit 630 is used to fine-tune the coarse relative pose according to the image pair and its two corresponding neural radiation fields to obtain the accurate relative pose.

[0146] The fusion unit 640 is used to fuse all the neural radiation fields according to the accurate relative pose to obtain a target map.

[0147] According to the present invention, an apparatus for constructing an urban airspace map based on an image acquisition device, wherein the step of inputting the image set and the preliminary pose into a pre-constructed common-view region retrieval module to perform image similarity search and obtain image pairs specifically includes:

[0148] Extract feature vectors from the image set, and establish an index based on the feature vectors;

[0149] Based on the index, a preset search algorithm is used to retrieve the most similar set of images, resulting in a first image and a second image, thus obtaining the image pair.

[0150] According to the present invention, an apparatus for constructing an urban airspace map based on an image acquisition device, wherein calculating the relative transformation of the corresponding neural radiation field based on the image pair to obtain the coarse relative pose of the image pair specifically includes:

[0151] Extract the preliminary poses of the first image and the second image from the image pair to obtain the first preliminary pose and the second preliminary pose;

[0152] Starting with the second preliminary pose as the initial estimation of the relative pose, based on the pre-built frame-to-model estimation model, the pose of the current frame is optimized by minimizing the difference between the rendered image of the second neural radiation field corresponding to the second image and the first image, thus obtaining the coarse relative pose of the image pair.

[0153] According to the present invention, an apparatus for constructing an urban airspace map based on an image acquisition device, wherein the step of fine-tuning the coarse relative pose based on the image pair and its corresponding two neural radiation fields to obtain an accurate relative pose specifically includes:

[0154] Based on the first neural radiation field corresponding to the first image in the image pair, the pose of the target image acquisition device is queried to obtain the first target pose; based on the second neural radiation field corresponding to the second image in the image pair, the pose of the target image acquisition device is queried to obtain the second target pose.

[0155] A composite image is obtained by rendering based on the first target pose, the second target pose, and the coarse relative pose;

[0156] Based on the frame-to-model estimation model, the accurate relative pose is obtained from the synthesized image.

[0157] According to the present invention, an apparatus for constructing an urban airspace map based on an image acquisition device, wherein fusing all the neural radiation fields according to the accurate relative pose to obtain a target map specifically includes:

[0158] The weights for each of the neural radiation fields are determined using inverse distance weighting;

[0159] The target map is obtained by fusing the neural radiation field based on the weights and the accurate relative pose.

[0160] According to the present invention, an apparatus for constructing an urban airspace map based on an image acquisition device, wherein obtaining a neural radiation field from the image set specifically includes:

[0161] Camera parameters are calculated using the images in the image set and the corresponding poses;

[0162] Generate sampling rays based on the camera parameters;

[0163] Based on the sampled rays, image rendering is performed using a multilayer perceptron to obtain the neural radiation field corresponding to the image set.

[0164] The present invention provides an apparatus for constructing urban airspace maps based on image acquisition devices. The apparatus acquires images of the urban airspace to be constructed and their corresponding poses using image acquisition devices to obtain an image set. A neural radiation field is obtained from the image set, and a preset optimization strategy is used to bundle and adjust the neural radiation field to obtain a preliminary pose of the images. The number of image acquisition devices is not less than two. The image set and the preliminary pose are input into a pre-constructed common-view region retrieval module for image similarity search to obtain image pairs. The relative transformation of the corresponding neural radiation field is calculated based on the image pairs to obtain a coarse relative pose of the image pairs. The coarse relative pose is fine-tuned based on the image pairs and their corresponding two neural radiation fields to obtain an accurate relative pose. All neural radiation fields are fused based on the accurate relative pose to obtain a target map. This invention acquires an image set of the urban airspace to be constructed using an image acquisition device, trains a neural radiation field based on the image set, and then optimizes the neural radiation field in three progressive stages, including bundle adjustment of individual neural radiation fields, frame-to-model optimization, and model-to-model optimization. Finally, the optimized scene representation and pose are fused. The pose estimation, registration, and fusion results obtained through the above steps have high accuracy.

[0165] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communications bus 740. The processor 710 can call logic instructions in the memory 730 to execute a method for constructing an urban airspace map based on an image acquisition device. This method includes: acquiring images of the urban airspace to be constructed and their corresponding poses using the image acquisition device to obtain an image set; obtaining a neural radiation field based on the image set; and using a preset optimization strategy to bundle and adjust the neural radiation field to obtain a preliminary pose of the images; wherein the number of image acquisition devices is not less than two; inputting the image set and the preliminary pose to a pre-constructed common-view region retrieval module to perform image similarity search to obtain image pairs; calculating the relative transformation of the corresponding neural radiation field based on the image pairs to obtain a coarse relative pose of the image pairs; fine-tuning the coarse relative pose based on the image pairs and their corresponding two neural radiation fields to obtain an accurate relative pose; and fusing all neural radiation fields based on the accurate relative pose to obtain a target map.

[0166] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the urban airspace map construction method based on image acquisition devices provided by the above methods. The method includes: acquiring images of the urban airspace to be constructed and the poses corresponding to the images through image acquisition devices to obtain an image set; obtaining a neural radiation field based on the image set; using a preset optimization strategy to bundle and adjust the neural radiation field to obtain a preliminary pose of the images; wherein the number of image acquisition devices is not less than two; inputting the image set and the preliminary poses into a pre-constructed common-view region retrieval module to perform image similarity search to obtain image pairs; calculating the relative transformation of the corresponding neural radiation fields based on the image pairs to obtain a coarse relative pose of the image pairs; fine-tuning the coarse relative pose based on the image pairs and their corresponding two neural radiation fields to obtain an accurate relative pose; and fusing all the neural radiation fields based on the accurate relative poses to obtain a target map.

[0168] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for constructing an urban airspace map based on an image acquisition device, as provided in the above methods. This method includes: acquiring images of the urban airspace to be constructed and their corresponding poses using an image acquisition device to obtain an image set; obtaining a neural radiation field based on the image set; and adjusting the neural radiation field using a preset optimization strategy to obtain a preliminary pose of the images. The number of image acquisition devices is not less than two. The image set and the preliminary pose are input into a pre-constructed common-view region retrieval module for image similarity search to obtain image pairs. The relative transformation of the corresponding neural radiation field is calculated based on the image pairs to obtain a coarse relative pose of the image pairs. The coarse relative pose is fine-tuned based on the image pairs and their corresponding two neural radiation fields to obtain an accurate relative pose. All neural radiation fields are fused based on the accurate relative pose to obtain a target map.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an urban airspace map based on an image acquisition device, characterized in that, include: Images of the urban airspace to be constructed and their corresponding poses are acquired using image acquisition devices to obtain an image set. A neural radiation field is then obtained based on the image set. A preset optimization strategy is used to bundle and adjust the neural radiation field to obtain the initial pose of the images. The number of image acquisition devices is not less than two. During the process of obtaining the neural radiation field from the image set, camera parameters are added to the training process of the neural radiation field model to jointly optimize the neural radiation field using camera parameters and the 3D scene. The image set and the preliminary pose are input into a pre-built common-view region retrieval module to perform image similarity search, obtain image pairs, and calculate the relative transformation of the corresponding neural radiation field based on the image pairs to obtain the coarse relative pose of the image pairs. Based on the image pair and its corresponding two neural radiation fields, the coarse relative pose is finely adjusted to obtain the accurate relative pose; Based on the accurate relative pose, all the neural radiation fields are fused to obtain the target map; The bundle adjustment optimizes scene representation and camera extrinsics by minimizing a weighted combination of distortion loss, aberration loss, and proposal loss. The preset optimization strategy is a coarse-to-fine optimization strategy, which gradually activates high-frequency position encoding in the neural radiation field model by truncating the dynamic low-pass filter.

2. The method for constructing an urban airspace map based on an image acquisition device according to claim 1, characterized in that, The step of inputting the image set and the preliminary pose into a pre-constructed common-view region retrieval module to perform image similarity search and obtain image pairs specifically includes: Extract feature vectors from the image set, and establish an index based on the feature vectors; Based on the index, a preset search algorithm is used to retrieve the most similar set of images, resulting in a first image and a second image, thus obtaining the image pair.

3. The method for constructing an urban airspace map based on an image acquisition device according to claim 2, characterized in that, The step of calculating the relative transformation of the corresponding neural radiation field based on the image pair to obtain the coarse relative pose of the image pair specifically includes: Extract the preliminary poses of the first image and the second image from the image pair to obtain the first preliminary pose and the second preliminary pose; Starting with the second preliminary pose as the initial estimation of the relative pose, based on the pre-built frame-to-model estimation model, the pose of the current frame is optimized by minimizing the difference between the rendered image of the second neural radiation field corresponding to the second image and the first image, thus obtaining the coarse relative pose of the image pair.

4. The method for constructing an urban airspace map based on an image acquisition device according to claim 3, characterized in that, The step of fine-tuning the coarse relative pose based on the image pair and its corresponding two neural radiation fields to obtain the accurate relative pose specifically includes: Based on the first neural radiation field corresponding to the first image in the image pair, the pose of the target image acquisition device is queried to obtain the first target pose; based on the second neural radiation field corresponding to the second image in the image pair, the pose of the target image acquisition device is queried to obtain the second target pose. A composite image is obtained by rendering based on the first target pose, the second target pose, and the coarse relative pose; Based on the frame-to-model estimation model, the accurate relative pose is obtained from the synthesized image.

5. The method for constructing an urban airspace map based on an image acquisition device according to claim 1, characterized in that, The step of fusing all the neural radiation fields based on the accurate relative pose to obtain the target map specifically includes: The weights for each of the neural radiation fields are determined using inverse distance weighting; The target map is obtained by fusing the neural radiation field based on the weights and the accurate relative pose.

6. The method for constructing an urban airspace map based on an image acquisition device according to claim 1, characterized in that, The step of obtaining the neural radiation field based on the image set specifically includes: Camera parameters are calculated using the images in the image set and the corresponding poses; Generate sampling rays based on the camera parameters; Based on the sampled rays, image rendering is performed using a multilayer perceptron to obtain the neural radiation field corresponding to the image set.

7. A device for constructing urban airspace maps based on image acquisition equipment, characterized in that, include: The preliminary pose unit is used to acquire images of the urban airspace to be constructed and the corresponding poses of the images through image acquisition devices to obtain an image set. Based on the image set, a neural radiation field is obtained, and the neural radiation field is bundled and adjusted using a preset optimization strategy to obtain the preliminary pose of the image. The number of image acquisition devices is not less than two. During the process of obtaining the neural radiation field based on the image set, camera parameters are added to the training process of the neural radiation field model to jointly optimize the neural radiation field with camera parameters and the 3D scene. The coarse relative pose unit is used to input the image set and the preliminary pose into a pre-constructed common-view region retrieval module, perform image similarity search, obtain image pairs, calculate the relative transformation of the corresponding neural radiation field based on the image pairs, and obtain the coarse relative pose of the image pairs. The accurate relative pose unit is used to fine-tune the coarse relative pose based on the image pair and its two corresponding neural radiation fields to obtain the accurate relative pose. A fusion unit is used to fuse all the neural radiation fields according to the accurate relative pose to obtain a target map; The bundle adjustment optimizes scene representation and camera extrinsics by minimizing a weighted combination of distortion loss, aberration loss, and proposal loss. The preset optimization strategy is a coarse-to-fine optimization strategy, which gradually activates high-frequency position encoding in the neural radiation field model by truncating the dynamic low-pass filter.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for constructing an urban airspace map based on an image acquisition device as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for constructing an urban airspace map based on an image acquisition device as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for constructing an urban airspace map based on an image acquisition device as described in any one of claims 1 to 6.

Images