Method and system for optimizing a neural radiance field with few views based on an object pose probe

Abstract

The application discloses a few-view neural radiance field optimization method and system based on an object pose probe, the neural radiance field problem without pose prior is extended to scene reconstruction, an object in the scene is selected as a pose probe, a template prior is provided for the shape of the selected object, and a morphing network is learned to adjust the shape of the object, so that the camera pose and scene expression accuracy are improved at the same time, and an incremental camera pose training strategy is further proposed to make up for the defects of the initial camera pose of the existing method. The application aims at the few-view input scene, realizes the synthesis of a realistic new view rendering image by using the neural radiance field without the camera pose, solves the problems that the current pose optimization neural radiance field needs the initial pose, the pure implicit network training speed is slow and difficult to initialize, and the method is insufficiently constrained to the camera pose.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and graphics, specifically relating to a method and system for optimizing neural radiation fields with few viewing angles based on object pose probes. Background Technology

[0002] In the fields of computer vision and graphics, Neural Radiation Fields (NeRFs) mark a significant milestone, providing unprecedented capabilities for realistic scene rendering from images captured from multiple viewpoints. The accuracy of this new perspective rendering relies heavily on the precision of the input camera pose and the number of input images, limiting its application in real-world scenarios. In existing research, the camera pose of the input view is primarily recovered using COLMAP. However, when the input views are limited and sparse, COLMAP performance degrades significantly due to the large viewpoint baseline and difficulties in feature matching. To date, pose-independent NeRF techniques remain a major challenge in the field. Simultaneously solving the problem of 3D scene representation (reconstruction) and localization (registration) of a given camera frame from RGB images is a long-standing chicken-and-egg problem in computer vision—reconstructing the 3D structure requires observations with known camera poses, while localizing the camera requires reliable correspondences from the reconstruction.

[0003] There are currently some related technologies to solve the optimization problem of pose-independent neural radiation fields, such as: Method [1] (see literature: Lin CH, Ma WC, Torralba A, et al. Barf: Bundle-adjusting neuralradiance fields[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2021:5741-5751.) proposes a coarse-to-fine position encoding strategy, which makes joint optimization of pose and scene easier. This strategy effectively balances the difficulty and accuracy of optimization by using a lower frequency position encoding in the early stage of the optimization process and gradually transitioning to a higher frequency position encoding. This method not only simplifies the complexity of simultaneously and accurately estimating scene geometry and camera position, but also improves the stability and accuracy of the reconstruction and registration process. However, this method can only solve forward-facing shooting scenes or requires a better initial pose as input, which leads to greater limitations in practical applications. Method [2] (see reference: Jeong Y, Ahn S, Choy C, et al. Self-calibrating neural radiance fields[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2021:5846-5854.) further optimizes camera distortion and proposes a geometric loss function to normalize the learned parameters. This means that the method not only focuses on the optimization of camera pose and scene content, but also considers the distortion effect that may be introduced by the camera lens. By introducing geometric loss to adjust and normalize these effects, it ensures that the reconstructed scene is as close to the real world as possible. However, this method requires dense camera viewpoints as input and cannot attempt to generate high-quality new viewpoints in scenes with few viewpoints.Method [3] (see reference: Truong P, Rakotosaona MJ, Manhardt F, et al. Sparf: Neural radiance fields from sparse and noisy poses[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2023:4190-4200.) proposes a solution to the challenge of neural radiance fields given only a wide baseline input image with a sparse viewpoint. It proposes to utilize the geometric consistency loss of dense matching and the depth consistency constraint to learn high-precision camera pose and high-quality scene representation at the same time. However, this method also requires a camera pose close to the real camera distribution as initialization, which greatly limits its application in real-world scenarios. This shows that although the method has made progress in dealing with scenes with wide baselines and sparse views, its high dependence on the initial camera pose has become an obstacle to its generalization to real-world applications. This challenge highlights the importance of accurately obtaining the camera pose when applying neural radiance field technology in complex real-world environments. Another class of methods [4] (see reference: Bian W, Wang Z, Li K, et al. Nope-nerf: Optimizing neural radiance field with nopose prior [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 4160-4169.) considers using monocular depth estimation as auxiliary information for learning scene representation without relying on pose prior. By aligning the predicted depth in 3D space, camera pose registration and scene geometry perception are achieved. However, this method usually requires dense image sequences and is difficult to handle wide baseline input images. This means that although this method can learn the 3D representation of the scene without requiring accurate camera pose, its ability to handle small amounts of input data or large camera baseline transformations is limited.

[0004] In 2021, Zhang et al. (see: Zhang, Jason, et al. "Ners: Neural reflectance surfaces for sparse-view 3D reconstruction in the wild." Advances in Neural Information Processing Systems 34(2021):29835-29847.) proposed a sparse-view reconstruction method based on neural reflectance surfaces. This method proposes using a template object as the initial geometric representation, then incorporating a deformation field network to learn the object's accurate geometric representation. Given the initial camera pose, it optimizes the camera pose based on rendering loss while adjusting the object geometry, achieving high-quality object-based reconstruction. However, this technique only provides high-quality object reconstruction and cannot address the technical problem of synthesizing realistic new-view renderings using neural reflectance fields without camera pose. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a method and system for optimizing neural radiation fields with few viewpoints based on object pose probes, in response to the above-mentioned problems in the prior art. This invention aims to achieve the synthesis of realistic new viewpoint renderings using neural radiation fields without camera pose, and solves the problems of the need for initial pose in current pose optimization neural radiation fields, the slow training speed and difficulty in initialization of pure implicit networks, and the insufficient constraints on camera pose of the method.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A method for optimizing the neural radiation field with few viewing angles based on an object pose probe includes the following steps:

[0008] S101, for scenarios with few viewpoints, select an object in the scene as an attitude probe;

[0009] S102, add an image frame, determine whether the newly added image frame is the first image frame. If the newly added image frame is the first image frame, combine the pose probe and the mask image to obtain the initial camera pose of the newly added image frame. Otherwise, combine the optimized camera pose of the previous image frame and use the PnP algorithm to calculate and obtain the initial camera pose of the newly added image frame.

[0010] S103: Initialize the object's template field using a pose probe, and optimize the object's geometry using a deformation network. Based on the optimized object geometry, the initial pose of the newly added image camera, and the input mask image, optimize the object's NeRF parameters using consistency-constrained image rendering loss to obtain accurate scene modeling and optimized camera pose. Based on the newly added image and optimized camera pose, optimize the scene's NeRF parameters using consistency-constrained image rendering loss to obtain optimized camera pose and a synthesized scene with a new perspective.

[0011] S104, determine whether to continue adding images incrementally. If it is still necessary to continue adding images incrementally, then jump to step S102; otherwise, output the final synthesized scene with the new perspective.

[0012] Optionally, step S102, which combines the pose probe and the mask image to obtain the initial camera pose of the newly added image frame, includes: using a preset geometry as the initial geometry of the pose probe, sampling multiple projection views around the pose probe of the scene to obtain a mask image, and selecting the projection view corresponding to the mask image that best matches the preset reference mask as the initial camera pose of the first frame image.

[0013] Optionally, the preset geometry is a cube shape.

[0014] Optionally, in step S102, the calculation of the initial camera pose for the newly added image frame using the PnP algorithm, based on the optimized camera pose from the previous image frame, includes: for each newly added image frame I... i+1 Calculate the frame I of the previous image i The 2D correspondence, the previous image frame I i The matching pixels in the image intersect the projected rays on the object to obtain the corresponding 3D points, using the previous image frame I. i Optimized camera pose P i Perform precise surface positioning to form the newly added image frame I i+1 The 2D-3D correspondence between the object and the perspective n-point PnP of RANSAC is then used to calculate the newly added image frame I. i+1 The initial pose is used to obtain the newly added image frame I. i+1 The initial camera pose.

[0015] Optionally, in step S103, the template field of the object is initialized using a pose probe, and the optimized geometry of the object is obtained by optimizing the template field of the object through a deformation network. This includes: acquiring multiple rays starting from pixel p based on the current camera pose, using a deformation network D to predict a displacement vector v and a scalar correction value Δs for each pixel p, determining the new position p′ corresponding to each pixel p based on the displacement vector v, and interpolating to generate the final signed distance function SDF of pixel p, finally obtaining the optimized geometry of the object composed of the final signed distance functions SDF of each pixel p.

[0016] Optionally, the function expression for the final signed distance function SDF of the interpolated pixel p is:

[0017] SDF(p)=T(P+D V (p))+D Δs (p) = interp(p + v, V) (sdf) )+Δs,

[0018] In the above formula, SDF(p) is the final signed distance function SDF of pixel p, T(P+D) V (p) represents the pixel P+D in the template field T of the object. V The value of (p), D V (p) and D Δs (p) represents the deformation adjacency and correction amount, respectively, interp represents interpolation, and V (sdf) This represents the voxel mesh of the template field.

[0019] Optionally, the functional expression for the consistency-constrained image rendering loss in step S103 is:

[0020] The functional expression for the consistency-constrained image rendering loss in step S103 is:

[0021]

[0022]

[0023]

[0024] In the above formula, To constrain image rendering loss for consistency, Let V be the image pose, V be the set of image pixel matching pairs, and w be the image pose. x Let λ and λ1 be the confidence scores for matching pixels x and y, respectively, and D(x,y) be the weighting coefficients. D(x,y) represents the geometric projection distance consistency constraint. For surface sampling point constraints, γ xThis indicates whether pixel x is occluded when viewed from another perspective; 0 indicates occlusion, and 1 indicates non-occlusion. x A multi-layer feature measure representing pixel x;

[0025]

[0026] In the above formula, M represents the number of feature layers, cosine similarity is the cosine similarity, and F represents the image features extracted from different layers. Represents surface point S x The k-th layer image features projected onto image j, F i,k (x) represents the k-th layer image feature of pixel x in image i; and we have:

[0027]

[0028]

[0029] In the above formula, sdf represents the mapping of the final symbolic distance function SDF. and These represent the sampling points closest to the object surface in the rays emitted from pixels x and y, respectively. ρ is the Huber loss function, and π is the projection function. x ,S y ) is a surface point located using ray projection for a pair of matching pixels (x, y) of a given image pixel matching pair. and Image I i and I j The camera pose, and the function expression for the mapping of the final signed distance function SDF, is as follows:

[0030] sdf(p)=β(1 / (1+e SDF(p) -0.5),

[0031] In the above equation, β and γ are trainable parameters, and e is a natural constant.

[0032] Furthermore, the present invention also provides a few-view neural radiation field optimization system based on object pose probe, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the few-view neural radiation field optimization method based on object pose probe.

[0033] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program / instructions that are programmed or configured to execute the object pose probe-based few-view neural radiation field optimization method via a processor.

[0034] Furthermore, the present invention also provides a computer program product, including a computer program / instructions, which are programmed or configured to execute the aforementioned object pose probe-based few-view neural radiation field optimization method via a processor.

[0035] Compared with existing technologies, this invention has the following main advantages: This invention extends the pose-prior-free neural radiation field problem to scene reconstruction. It selects a single object in the scene as a pose probe, provides a template prior for the shape of this selected object, and learns a deformation network to adjust the object's shape, thereby simultaneously improving the accuracy of camera pose and scene representation. Furthermore, this invention improves upon the pose-optimized neural radiation field method by proposing an incremental camera pose training strategy. This achieves a neural radiation field optimization method independent of any camera pose, overcoming the shortcomings of existing methods regarding initial camera pose. It enables the synthesis of realistic new perspective renderings using neural radiation fields without camera pose, solving the problems of current pose-optimized neural radiation field methods requiring initial pose, the slow training speed and difficulty in initializing pure implicit networks, and insufficient constraints on camera pose. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram illustrating the principle of obtaining the initial camera pose in an embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram illustrating the principle of optimizing the geometry of an object in an embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram illustrating the principle of consistency constraints in an embodiment of the present invention.

[0040] Figure 5 This is the camera pose registration effect in an embodiment of the present invention.

[0041] Figure 6 This is a new perspective generation effect produced in the embodiments of the present invention.

[0042] Figure 7 This is the effect of generating a new perspective on the DTU dataset in an embodiment of the present invention.

[0043] Figure 8 This is a graph showing the rotation and translation error changes of the incremental optimization strategy in this embodiment. Detailed Implementation

[0044] like Figure 1 As shown, the few-view neural radiation field optimization method based on object pose probe in this embodiment includes the following steps:

[0045] S101, for scenarios with few viewpoints, select an object in the scene as an attitude probe;

[0046] S102, add an image frame, determine whether the newly added image frame is the first image frame. If the newly added image frame is the first image frame, combine the pose probe and the mask image to obtain the initial camera pose of the newly added image frame. Otherwise, combine the optimized camera pose of the previous image frame and use the PnP algorithm to calculate and obtain the initial camera pose of the newly added image frame.

[0047] S103: Initialize the object's template field using a pose probe, and optimize the object's geometry using a deformation network. Based on the optimized object geometry, the initial pose of the newly added image camera, and the input mask image, optimize the object's NeRF parameters using consistency-constrained image rendering loss to obtain accurate scene modeling and optimized camera pose. Based on the newly added image and optimized camera pose, optimize the scene's NeRF parameters using consistency-constrained image rendering loss to obtain optimized camera pose and a synthesized scene with a new perspective.

[0048] S104, determine whether to continue adding images incrementally. If it is still necessary to continue adding images incrementally, then jump to step S102; otherwise, output the final synthesized scene with the new perspective.

[0049] like Figure 2 As shown, in step S102 of this embodiment, obtaining the initial camera pose of the newly added image frame by combining the pose probe and the mask image includes: using a preset geometry as the initial geometry of the pose probe, sampling multiple projection views around the pose probe in the scene to obtain a mask image, and selecting the projection view corresponding to the mask image that best matches the preset reference mask as the initial camera pose of the first frame image. In this embodiment, the preset geometry is a cube shape.

[0050] In this embodiment, step S102, which combines the optimized camera pose from the previous image frame with the PnP algorithm to calculate and obtain the initial camera pose of the newly added image frame, includes: for each newly added image frame I... i+1 Calculate the frame I of the previous image i The 2D correspondence, the previous image frame I i The matching pixels in the image intersect the projected rays on the object to obtain the corresponding 3D points, using the previous image frame I. i Optimized camera pose P i Perform precise surface positioning to form the newly added image frame I i+1The 2D-3D correspondence between the object and the perspective n-point PnP of RANSAC is then used to calculate the newly added image frame I. i+1 The initial pose is used to obtain the newly added image frame I. i+1 The initial camera pose in the image. Specifically, for each newly added image frame I... i+1 Calculate the frame I of the previous image i The 2D correspondence is a well-known method. For example, in this embodiment, the well-known SuperPoint method and SuperGlue method are specifically used for each newly added image frame I. i+1 Calculate the frame I of the previous image i For details on the 2D correspondence, the SuperPoint method can be found in the following literature: DeTone, Daniel, Tomasz Malisiewicz, and Andrew Rabinovich. "Superpoint: Self-supervised interest point detection and description." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018. The SuperGlue method can be found in the following literature: Sarlin, Paul-Edouard, et al. "Superglue: Learning feature matching with graph neural networks." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020.

[0051] like Figure 3 As shown, in step S103 of this embodiment, initializing the template field of the object using a pose probe and optimizing the template field of the object to obtain the optimized geometry of the object through a deformation network includes: acquiring multiple rays starting from pixel p based on the current camera pose, and using a deformation network D( Figure 5The implicit deformation network (D) is described in the following literature: Deng Y, Yang J, Tong X. Deformed implicit field: Modeling 3d shapes with learned dense correspondence[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2021:10286-10296.) predicts a displacement vector v and a scalar correction value Δs for each pixel p. Based on the displacement vector v, the new position p′ corresponding to each pixel p is determined, and the final signed distance function SDF of pixel p is generated by interpolation. Finally, the optimized geometry of the object is obtained by the final signed distance functions SDF of each pixel p. The deformation network D predicts a displacement vector v and a scalar correction value Δs for each pixel p, which can be expressed as:

[0052]

[0053] The displacement vector v, also known as the deformation vector, determines the new position of each point. The final signed distance function (SDF) value for any point is determined by interpolation at its new position in the deformed position within the template field T, and further refined by a correction scalar. In this embodiment, the function expression for the final signed distance function SDF of the interpolated pixel p is:

[0054] SDF(p)=T(P+D V (p))+D Δs (p) = interp(p + v, V) (sdf) )+Δs,

[0055] In the above formula, SDF(p) is the final signed distance function SDF of pixel p, T(P+D) V (p) represents the pixel P+D in the template field T of the object. V The value of (p), D V (p) and D Δs (p) represents the deformation adjacency and correction amount, respectively, interp represents interpolation, and V (sdf)The voxel mesh represents the template field. The predicted SDF value in this hybrid representation is used to estimate volumetric opacity, but directly using the SDF value defined in the equation above for volumetric rendering is imperfect because the scale of the value is manually predefined. Therefore, this invention proposes a mapping function with two learnable parameters that scales the original SDF value to a scene-customized scale. The final SDF value of point p is represented as sdf(p), and its function expression is as follows:

[0056] sdf(p)=β(1 / (1+e SDF(p) -0.5),

[0057] In the above equation, β and γ are trainable parameters, and e is a natural constant. To ensure that β and γ are always positive to preserve the original SDF sign, the Softplus activation function is applied to them.

[0058] Having clearly defined the scene representation of the pose probe, we will now introduce how to determine the points in 3D space corresponding to image pixels through ray casting. The calculation of surface points is fundamental to the initial pose calculation in PnP and subsequent geometric consistency calculations. According to the definition of SDF, surface points are defined as points set {p|sdf(p)=0}. Thanks to shallow deformable networks and explicit SDF voxel meshes, ray casting can efficiently estimate the surface points corresponding to image pixels. For a given camera pose... Given the pixel coordinates x and x, we can determine the ray emitted from that pixel in the world coordinate system. Let o be the camera optical center, and v be the direction vector of the ray emanating from the optical center. This can be determined by sampling N depth values ​​t on the ray. i To express 3D sampling point p i =o+t i v. To obtain a surface point that is aware of occlusion, then find the minimum index i such that t i The sign of the SDF value at point t and subsequent points t i+1 The smallest index i* is calculated as follows, depending on the sign of the two indices:

[0059] i * =argmin{i|sdf(p i )·sdf(p i+1 )<0},

[0060] In the above formula, argmin means the minimum value of the variable when the expression satisfies the condition. For simplicity, we will use i to represent i from now on. * Next, linear interpolation is used to calculate the depth of the emitted ray and the object's surface, thus obtaining the coordinates of their intersection point in 3D space, as calculated below:

[0061]

[0062] This embodiment also addresses the insufficient constraints on camera pose imposed by neural radiance fields in pose optimization by proposing multi-view geometric consistency and multi-layer feature consistency constraints. For geometric consistency constraints, inspired by existing methods (see: Truong P, Rakotosaona MJ, Manhardt F, et al. Sparf: Neural radiance fields from sparse and noisy poses [C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2023:4190-4200.), which utilize projection errors to constrain the consistency of scene geometry and camera pose, this embodiment employs a more direct multi-view projection distance to constrain camera pose. Specifically, given a pair of images (I... i ,I j And a pair of matching pixels (x, y), this embodiment uses ray projection to locate surface points (S) and a pair of matching pixels (x, y). x ,S y 3D surface points are projected back into the image coordinate system to minimize the distance between corresponding points. π is defined as the projection function, and ρ as the Huber loss function. and Let i and j represent the camera poses of images i and j respectively. The geometric projection distance between pixel pairs (x, y) is obtained as follows:

[0063]

[0064] In the above formula, D(x,y) is the geometric projection distance consistency constraint, ρ is the Huber loss function, π is the projection function, and (S x ,S y Given a pair of images (I i ,I j A pair of matching pixels (x, y) are used to locate surface points using ray projection. and Image I i and I jThe camera pose is determined. Furthermore, based on the prior knowledge that rays emitted from feature points should intersect with the object, a loss function is designed to correct the camera pose by minimizing the SDF (Surface Deformation Factor) of specific points. The core idea of ​​this method is that if the camera pose is accurate, the positions of feature points mapped to 3D space through the camera model should fall on the surface of the object; that is, the SDF values ​​of these points should be close to zero. By optimizing this loss function, the camera pose can be adjusted to ensure that feature points are correctly mapped to the object surface, thereby improving the accuracy of reconstruction and rendering. The loss calculation method is as follows:

[0065]

[0066] In the above formula, SDF represents the final symbolic distance function SDF. and These represent the sampling points closest to the object's surface in the light rays emitted from pixels x and y, respectively. Define w. x Let λ be the confidence score for matching pixels x and y, and λ be the weight ratio. Therefore, the functional expression for the consistency-constrained image rendering loss in step S103 of this embodiment is:

[0067]

[0068]

[0069]

[0070] In the above formula, To constrain image rendering loss for consistency, Let V be the image pose, V be the set of image pixel matching pairs, and w be the image pose. x Let λ and λ1 be the confidence scores for matching pixels x and y, respectively, and D(x,y) be the weighting coefficients. D(x,y) represents the geometric projection distance consistency constraint. For surface sampling point constraints, γ x This indicates whether pixel x is occluded when viewed from another perspective; 0 indicates occlusion, and 1 indicates non-occlusion. x A multi-layer feature measure representing pixel x;

[0071]

[0072] In the above formula, M represents the number of feature layers, cos is the cosine similarity, and F represents the image features extracted by different layers of the pre-trained network VGG (other feature extraction networks can also be used as needed). Represents surface point S x The k-th layer image features projected onto image j, F i,k (x) represents the k-th layer image feature of pixel x in image i; and we have:

[0073]

[0074]

[0075] In the above formula, sdf represents the mapping of the final symbolic distance function SDF. and These represent the sampling points closest to the object surface in the rays emitted from pixels x and y, respectively. ρ is the Huber loss function, and π is the projection function. x ,S y ) is a surface point located using ray projection for a pair of matching pixels (x, y) of a given image pixel matching pair. and Image I i and I j The camera pose is determined. Geometric consistency facilitates rapid convergence in camera pose optimization, while incorrect matching can generate misleading supervision signals, causing the optimization process to get stuck in local optima. Inspired by the bundle adjustment technique in Structure from Motion, this embodiment introduces a multi-layer feature metric consistency constraint. This constraint Aiming to minimize the feature differences of aligned pixels through dot product similarity, this constraint utilizes feature cosine similarity to measure the consistency of current object geometry and camera pose. This constraint considers more image pixels than just key points in geometric consistency. Furthermore, unlike photometric error, which is sensitive to initialization and adds non-convexity, this embodiment's feature-based consistency loss results in a smoother optimization process. Figure 4 As shown, multiple scene images and their corresponding initial poses are input, and the scene is modeled using Neural Radiation Field (NeRF). By using the consistency constraint of the image rendering loss in the above steps, the geometric representation of the scene and the camera pose can be learned better.

[0076] While training the pose probe neural radiance field, this embodiment trains a scene neural radiance field in parallel, learning the entire scene representation while simultaneously fine-tuning the camera pose. To verify the effectiveness of the module proposed in this embodiment, the scene branch employs a baseline neural radiance field model with coarse-to-fine position encoding (see reference: Lin C H, Ma WC, Torralba A, et al. Barf: Bundle-adjusting neural radiance fields[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2021:5741-5751.). Optimizing the radiance field from a sparse view presents a significant challenge. Similar to the pose probe neural radiance branch, this branch also uses projection distance loss as an additional constraint in the scene branch. Considering the addition of depth smoothing priors to enhance the geometric perception of the scene, this embodiment introduces a total depth variation loss on image patches, encouraging that the depths of adjacent pixels of rays emitted from acquired image patches be as similar as possible. During the training process in this embodiment, image rendering loss is added to the loss function for both the pose probe branch and the scene branch. The entire process is optimized from coarse to fine. Considering that too few images in the early training stages may be insufficient to learn an effective deformable network, this embodiment fixes the parameters of the deformable network during image addition. Once all images are added to the training loop, the parameters of the entire network, along with the camera pose, are optimized simultaneously. The final camera pose optimization result is as follows: Figure 5 As shown, Figure 5This demonstrates the alignment effect between the camera pose and the actual camera pose after optimization using the method of this embodiment and other methods. Red represents the actual camera pose distribution, and blue represents the camera pose distribution after algorithm optimization. For details on BARF, please refer to: Lin CH, Ma WC, Torralba A, et al. Barf: Bundle-adjusting neural radiance fields [C] / / Proceedings of the IEEE / CVF International Conference on ComputerVision. 2021:5741-5751.; for Nope-NeRF, please refer to: Bian W, Wang Z, Li K, et al. Nope-nerf: Optimizing neural radiance field with no pose prior [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023:4160-4169.; for SCNeRF, please refer to: Jeong Y, Ahn S, Choy C, et al. Self-calibrating neural radiance. fields[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2021:5846-5854.; For details on SPARF, see the following literature: Truong P, Rakotosaona MJ, Manhardt F, et al. Sparf: Neural radiance fields from sparse and noisy poses[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:4190-4200.; Ours represents the results obtained by the method in this embodiment. The obtained new perspective rendering effect of neural radiation fields is as follows. Figure 6 and Figure 7 As shown, GT represents the real image and Deform represents the deformation. Figure 6 The first row is the RGB image rendered from the new perspective, the second row is the depth image rendered from the new perspective, and the last column shows the geometric changes of the pose probe. Figure 7The first row is the RGB image rendered from a new perspective, and the second row is the depth image rendered from a new perspective. Figure 8 The diagram shows the rotation and translation error variations of the incremental optimization strategy in this embodiment. (a) represents the three-view rotation error, (b) the six-view rotation error, (c) the nine-view rotation error, (d) the three-view translation error, (e) the six-view translation error, and (f) the nine-view translation error. The blue curve (in this embodiment) represents the case including incremental attitude optimization, while the red curve indicates that incremental attitude optimization is disabled. It is evident that the incremental optimization strategy employed in this embodiment can effectively reduce rotation and translation errors.

[0077] In summary, this embodiment addresses the challenge of input images with only a few views (as low as 3) and utilizes neural radiation fields to synthesize realistic new perspective renderings without camera pose information. This embodiment proposes a novel idea: using common objects in the scene as pose probes to facilitate this challenging task. The entire process is self-supervised. This invention allows users to capture images without worrying about the initial pose, simply by selecting an object in the scene as a pose probe and using a cuboid as its initial geometry. Its core focus is on exploring two main benefits of pose probes: (1) This embodiment employs an incremental camera pose optimization strategy to address the challenge of missing initial poses. Views are progressively added from 2D correspondences and perspective n-points (PnP) to the initial poses of previous views. (2) Strong pose constraints are provided, and exploring geometric consistency helps to obtain better poses, leading to better new perspective rendering quality. This embodiment proposes using DeformNet to describe the geometry of the pose probe in detail and using the consistency of multi-view geometry for self-supervised training. The first contribution of this embodiment is the proposal to use common objects as pose probes and combine them with incremental pose optimization, thereby overcoming the limitation of not requiring approximate poses as input in the neural radiation field. In this method, new images are added to the training loop at fixed training intervals. Given an input image and its corresponding mask image (the mask corresponds to the image region of the pose probe), this embodiment designates the first image as reference image I. Multiple projection views are sampled around the pose probe to obtain the mask image. The projection view corresponding to the mask image that best matches the reference mask is selected as the initial pose for the first frame. For each newly added frame I... i+1 In this embodiment, SuperPoint and SuperGlue are first used to calculate the difference between the image I and the previous image. i The 2D correspondence. Image I i The matching pixels in the image project light rays intersect with the object to obtain the corresponding 3D points, and the optimized pose P is used to obtain the corresponding 3D points. iPrecise surface localization is performed. The process of calculating intersections will be explained later in this embodiment. Therefore, this embodiment establishes a 2D-3D correspondence with newly added images and objects, allowing the calculation of image I using RANSAC's perspective n-points (PnP). i+1 The initial pose is then calculated. The image pose and radiation field are then added for optimization. This embodiment uses different scene representations for the pose probe and the entire scene. For the entire scene, this embodiment uses the same pure implicit multilayer perceptron representation as the previous method. For the pose probe, this embodiment proposes an explicit-implicit combined signed distance function (SDF) representation. In the design of the hybrid explicit and implicit signed distance function (SDF) network, the explicit field T is a non-learned SDF-based voxel mesh initialized through a template object, while the implicit deformation field D is implemented through a multilayer perceptron, aiming to predict a deformation field and a correction field on top of the explicit field T. This embodiment initializes the voxel mesh with a template of similar shape, and uses a cube shape as a template sufficient to learn the geometry and appearance of the object. The SDF value in the voxel network is obtained by calculating the nearest distance from each center point in the voxel to the surface and determining whether the point is inside or outside the object. The template field T provides a strong foundation before obtaining the detailed SDF, significantly reducing the search space of the optimization process.

[0078] Furthermore, this embodiment also provides a few-view neural radiation field optimization system based on object pose probe, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the few-view neural radiation field optimization method based on object pose probe.

[0079] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program / instruction that is programmed or configured to execute the object pose probe-based few-view neural radiation field optimization method via a processor.

[0080] Furthermore, this embodiment also provides a computer program product, including a computer program / instruction, which is programmed or configured to execute the object pose probe-based few-view neural radiation field optimization method via a processor.

[0081] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for optimizing the neural radiation field with few viewing angles based on an object pose probe, characterized in that, Includes the following steps: S101, for scenarios with few viewpoints, select an object in the scene as a pose probe; S102, add an image frame, determine whether the newly added image frame is the first image frame. If the newly added image frame is the first image frame, sample multiple projection views around the pose probe of the scene to obtain a mask image. Combine the pose probe and the mask image to obtain the initial camera pose of the newly added image frame. Otherwise, combine the optimized camera pose of the previous image frame with the PnP algorithm to calculate and obtain the initial camera pose of the newly added image frame. S103, Initialize the template field of the object using the pose probe, and obtain the optimized geometry of the object by optimizing the template field of the object through the deformation network. Based on the optimized object geometry, the initial pose of the newly added camera image, and the input mask image, the NeRF parameters of the object are optimized using consistency-constrained image rendering loss to obtain accurate scene modeling and optimized camera pose; based on the newly added image and optimized camera pose, the NeRF parameters of the scene are optimized using consistency-constrained image rendering loss to obtain optimized camera pose and a synthetic scene with a new perspective. S104, determine whether to continue adding images incrementally. If it is still necessary to continue adding images incrementally, then jump to step S102; otherwise, output the final synthesized scene with the new perspective.

2. The method for optimizing the few-view neural radiation field based on an object pose probe according to claim 1, characterized in that, Step S102, which combines the pose probe and the mask image to obtain the initial camera pose of the newly added image frame, includes: using a preset geometry as the initial geometry of the pose probe, sampling multiple projection views around the pose probe of the scene to obtain the mask image, and selecting the projection view corresponding to the mask image that best matches the preset reference mask as the initial camera pose of the first frame image.

3. The method for optimizing the few-view neural radiation field based on an object pose probe according to claim 2, characterized in that, The preset geometry is a cube.

4. The method for optimizing the few-view neural radiation field based on an object pose probe according to claim 1, characterized in that, In step S102, the initial camera pose of the newly added image frame is calculated using the PnP algorithm based on the optimized camera pose of the previous image frame. This includes: for each newly added image frame... Calculate the previous image frame 2D correspondence, the previous image frame The matching pixels in the image are projected onto the object to obtain the corresponding 3D points, using the previous image frame. Optimized camera pose Perform precise surface positioning to form newly added image frames. The 2D-3D correspondence between the object and the viewpoint is then used to calculate the newly added image frames using RANSAC's perspective n-point PnP. The initial pose is used to obtain the newly added image frame. The initial camera pose.

5. The method for optimizing the few-view neural radiation field based on an object pose probe according to claim 4, characterized in that, Step S103 initializes the object's template field using a pose probe, and optimizes the object's template field through a deformation network to obtain the optimized object's geometry, including: based on the current camera pose, from pixel points... Multiple rays were collected at the start, using a deformation network. For each pixel Predict a displacement vector v and a scalar correction value. s, at pixel Based on the displacement vector Determine each pixel Corresponding new position And interpolate to generate pixels. The final symbolic distance function SDF is obtained from the total number of pixels. The final symbolic distance function SDF is used to construct the optimized geometry of the object.

6. The method for optimizing the few-view neural radiation field based on an object pose probe according to claim 5, characterized in that, The interpolation generates pixels. The final symbolic distance function SDF is expressed as follows: ， In the above formula, For pixels The final symbolic distance function SDF, Template field of the object pixels in The value, and These represent adjacent deformation and correction amount, respectively. Indicates interpolation. This represents the voxel mesh of the template field.

7. The method for optimizing the few-view neural radiation field based on an object pose probe according to claim 6, characterized in that, The functional expression for the consistency-constrained image rendering loss in step S103 is: ， ， ， In the above formula, To constrain image rendering loss for consistency, For geometric consistency, For multi-layer feature measurement consistency constraints, For image pose, For the set of image pixel matching pairs, For pixels and Match confidence, and This is the weighting ratio coefficient. For geometric projection distance consistency constraints, For surface sampling point constraints, Represents pixels Observe from another perspective whether there is occlusion; if there is occlusion, the value is 0, otherwise it is 1. Represents pixels Multi-layer feature measurement; ， In the above formula, Represents the feature layer number, and cos is the cosine similarity. This represents the image features extracted from different layers. Represents surface points Projected on the image The first Layer image features, Indicates pixel x in the image The first Layer image features; and have: ， ， In the above formula, This represents the mapping of the final symbolic distance function SDF. and Representing from pixels and The sampling point closest to the object's surface in the emitted light. For Huber loss function, For projection function, For a given image pixel pair, a pair of matching pixels Surface points are located using light projection. and Images and The camera pose, and the function expression for the mapping of the final signed distance function SDF, is as follows: ， In the above formula, and These are trainable parameters. It is a natural constant.

8. A few-view neural radiation field optimization system based on object pose probe, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the few-view neural radiation field optimization method based on object pose probe as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program / instructions, characterized in that, The computer program / instructions are programmed or configured to execute, via a processor, the method for optimizing the few-view neural radiation field based on an object pose probe as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, The computer program / instructions are programmed or configured to execute, via a processor, the method for optimizing the few-view neural radiation field based on an object pose probe as described in any one of claims 1 to 7.

Images