A method for joint positioning and modeling of spatial non-cooperative maneuvering targets based on NeRF

Abstract

This invention discloses a method for joint localization and modeling of non-cooperative maneuvering targets in space based on NeRF. Multiple consecutive RGB-D images are acquired by a monocular depth camera mounted on a capture satellite during the approach to the target satellite. A neural network implicitly represents the geometric structure and appearance color features of the target satellite. After rendering, the RGB-D values ​​of the sampled pixels are obtained. A cost function is constructed using the deviation from the ground truth to optimize the parameters of the neural network, achieving a 3D reconstruction of the target satellite's structure and appearance, while simultaneously estimating the pose information of the capture satellite relative to the target satellite. This method can reasonably predict the observed parts, avoid holes, fully utilize the information in NeRF to simultaneously complete 3D reconstruction and pose estimation, and has strong noise resistance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for joint localization and modeling of non-cooperative maneuvering targets in space based on NeRF. Background Technology

[0002] With increasing competition for space resources, spacecraft are more vulnerable to threats from anomalous encounters with non-cooperative maneuvering targets while in orbit. Developing space situational awareness capabilities can significantly improve the survivability of spacecraft in orbit and lay the foundation for further offensive and defensive maneuvers. Joint localization and modeling (JLM) technology provides 3D and pose information of targets, assisting spacecraft in accurately perceiving and assessing threats and opportunities in the space environment. It is a crucial component of space situational awareness systems. Furthermore, the real-time and accuracy requirements in space offensive and defensive maneuvers place higher demands on target 3D reconstruction and pose estimation.

[0003] Traditional 3D reconstruction of spatial targets primarily employs a combination of Structure from Motion (SFM) and Multiple View Stereo (MVS) algorithms. SFM calculates camera intrinsic parameters, pose, and sparse point cloud information from unlabeled images, while MVS refines this information to generate a denser 3D reconstruction. However, during the capture of spatial targets, the window for close-up shooting is short, resulting in fewer images. Furthermore, the target's pose remains relatively stable throughout the process, making it difficult to obtain comprehensive viewpoint images. This presents challenges for the traditional geometry-based SFM+MVS method.

[0004] The implicit neural representation uses a neural network to map the position (xyz) of a point in the target scene and the direction (θ φ) from the camera's optical center to that point to a volume density ρ and a view-dependent RGB color. The system needs to calculate the input (xyz θ φ) of the NeRF mesh based on the camera's pose information relative to the target. However, due to the target's movement, the pose relative to the target star cannot be directly obtained through onboard sensors such as IMU or GPS. Although the required pose information can be calculated using the SFM+MVS framework, this method clearly does not fully utilize the information in the network, and the computation process is relatively complex. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a method for joint localization and modeling of non-cooperative maneuvering targets in space based on Neural Radiance Fields (NeRF). By acquiring multiple consecutive RGB-D images during the approach to the target star using a monocular depth camera mounted on a capture satellite, a neural network is used to implicitly represent the geometric structure and appearance color features of the target star. After rendering, the RGB-D values ​​of the sampled pixels are obtained. A cost function is constructed using the deviation from the ground truth to optimize the parameters of the neural network, achieving a 3D reconstruction of the target star's structure and appearance, while simultaneously estimating the pose information of the capture satellite relative to the target star. This method can reasonably predict the observed parts, avoid gaps, fully utilize the information in the NeRF to simultaneously complete 3D reconstruction and pose estimation, and has strong noise resistance.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] Step 1: Acquire multiple consecutive RGB-D images during the approach to the target star using a monocular depth camera mounted on the capture star;

[0008] Step 2: In the NeRF network, a neural network is used to characterize the geometric structure and appearance color features of the target star. After rendering, the RGB-D values ​​of each pixel are obtained. The cost function is constructed using the deviation from the true value to optimize the parameters of the neural network, thereby realizing the three-dimensional reconstruction of the structure and appearance of the target star, and simultaneously estimating the pose information of the capture star relative to the target star.

[0009] Step 2-1: Set up a fixed number of frames as a group. Estimate the pose of the capture star relative to the target star in each frame. The network parameters used are the parameters updated after training in the previous group. After completing the relative pose estimation of the first frame of the current group, start to optimize the network parameters using the first frame. The pose information used is the one estimated from the first frame of the current group.

[0010] Step 2-2: The NeRF network is a mapping of the coordinates (xyz) and viewing direction (θ φ) of a point in space to color and occupancy probability (rgb α);

[0011] Position 3D coordinates and viewing direction First, use a mapping function to raise the dimension to a higher dimension. Spatial coordinates are then fed into a neural network to obtain the color and occupancy probability α of a point in three-dimensional space. i ;

[0012] Steps 2-3: Collect M three-dimensional spatial sample points by emitting a ray from the optical center of the capture star camera towards the two-dimensional pixel sampling point. The depth value of each three-dimensional spatial sample point is d. iFor i = 1, 2, ..., M, query the NeRF network to obtain the corresponding color and occupancy probability α. i =F Θ (p i The color obtained by rendering is... for:

[0013]

[0014] depth for:

[0015]

[0016] Where ω i For p i The termination probability of a point is determined by the occupancy probability of that point. Calculation yielded:

[0017]

[0018] The depth variance calculated along the ray is:

[0019]

[0020] Steps 2-4: Loss Function;

[0021] The photometric loss function L used to optimize network parameters and pose estimation p L1 loss between rendered color and actual color:

[0022]

[0023] Geometric loss function L g The L1 loss between the rendered depth and the actual depth is normalized using depth variance:

[0024]

[0025] The network parameters are then optimized using a weighted sum of photometric and geometric losses:

[0026]

[0027] The optimization function used in the relative pose estimation process is:

[0028]

[0029] Preferably, the NeRF network structure F Θ From a multilayer perceptron And the position encoder γ is composed of:

[0030]

[0031] Among them, multilayer perceptron It contains 4 hidden layers, each with 256 hidden units, and the activation function is ReLU; the position encoder is a sine / cosine periodic function encoder.

[0032]

[0033] Where L = 8.

[0034] The beneficial effects of this invention are as follows:

[0035] 1. It can make reasonable predictions about the locations that can be observed, thus avoiding voids;

[0036] 2. It can fully utilize the information in NeRF to simultaneously complete 3D reconstruction and pose estimation;

[0037] 3. It has strong noise resistance;

[0038] 4. It can perform joint positioning and modeling tasks for space-flying cooperative maneuvering targets in real time. Attached Figure Description

[0039] Figure 1 This is a flowchart of the present invention;

[0040] Figure 2 This is a schematic diagram illustrating parallel pose estimation and network parameter optimization in an embodiment of the present invention.

[0041] Figure 3 The embodiments of the present invention include a NeRF network model of a position encoder and a neural network;

[0042] Figure 4 This is a NeRF rendering diagram of an embodiment of the present invention.

[0043] Figure 5 This is the pose estimation result of the capture star in the target star coordinate system according to an embodiment of the present invention.

[0044] Figure 6 (a) is a depth map actually obtained by capturing the satellite in an embodiment of the present invention. Figure 6 (b) is a depth map obtained by rendering based on the output of a neural network according to an embodiment of the present invention. Figure 6 (c) is the difference between the actual depth map and the rendered depth map in the embodiment of the present invention.

[0045] Figure 7 (a) is the RGB image of the captured star obtained in an embodiment of the present invention. Figure 7 (b) is an RGB image obtained by rendering based on the output of a neural network according to an embodiment of the present invention. Figure 7(c) is the difference between the actual RGB image and the rendered RGB image in the embodiment of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] This invention provides a method for joint localization and modeling of non-cooperative maneuvering targets in space based on NeRF, employing the following technical solution:

[0048] By acquiring multiple consecutive RGB-D images of the target star using a monocular depth camera mounted on the capture star during the approach process, a neural network is used in the NeRF network to characterize the geometric structure and appearance color features of the target star. After rendering, the RGB-D values ​​of each pixel are obtained. The parameters of the neural network are optimized by constructing a cost function based on the deviation between the target star and the ground truth value, thereby realizing the three-dimensional reconstruction of the structure and appearance of the target star and simultaneously estimating the pose information of the capture star relative to the target star.

[0049] To improve the real-time performance of the computation, the 3D reconstruction and pose estimation involved in this invention run on different threads, with the two threads sharing network parameters and pose estimation results. The specific process is as follows: a fixed number of frames are grouped together, and the pose of the capturing star relative to the target star is estimated for each frame, using the network parameters updated from the previous group's training. After completing the relative pose estimation for the first frame of the current group, the network parameters are optimized using the first frame, with the pose information obtained from the estimation of the first frame of the current group.

[0050] This invention considers the influence of illumination direction on the color of a target star. The NeRF network described is a mapping from the coordinates (xy z) and viewing direction (θ φ) of a point in space to color and occupancy probability (rgb α). Position encoding enables the network to learn high-frequency geometric structure information, specifically the three-dimensional coordinates of the position and the viewing direction. First, use a mapping function to raise the dimension to a higher dimension. Spatial coordinates are then fed into a neural network to obtain the color and occupancy probability α of a point in three-dimensional space. i .

[0051] To obtain the color and depth of the image at the sampling point, M three-dimensional spatial sample points are sampled by a ray emitted from the optical center of the capturing star camera towards the two-dimensional pixel sampling point. The depth value of each three-dimensional spatial sample point is d. i For each i = 1, 2, ..., M, the corresponding color and occupancy probability α can be obtained by querying the NeRF network. i =F Θ (p i The color obtained by rendering is... for:

[0052]

[0053] depth for:

[0054]

[0055] Where ω i For p i The termination probability of a point can be determined by the occupancy probability α of that point. pi Calculation yielded:

[0056]

[0057] The depth variance calculated along the ray is:

[0058]

[0059] The photometric loss function L used to optimize network parameters and pose estimation p L1 loss between rendered color and actual color:

[0060]

[0061] Geometric loss function L g The L1 loss between the rendered depth and the actual depth is normalized using depth variance:

[0062]

[0063] The network parameters are then optimized using a weighted sum of photometric and geometric losses:

[0064]

[0065] The optimization function used in the relative pose estimation process is:

[0066]

[0067] Example:

[0068] like Figure 1 As shown, the invention relates to a NeRF-based method for joint localization and modeling of non-cooperative maneuvering targets in space, which uses a neural network to characterize the geometric structure and appearance color features of the target star. The neural network can be a multi-layer perceptron (MLP).

[0069] As an optional embodiment, the parallel computing of the entire implementation process is as follows: Figure 2As shown, in order to achieve parallel computation for joint localization and modeling of non-cooperative maneuvering targets in space, five frames of images are set as a group, and the pose of the capturing star relative to the target star is estimated for each frame. The MLP parameters used are the parameters updated after training in the previous group. The MLP parameters are optimized only for the first frame of each group, and the pose information used is the pose estimated from the first frame of the current group.

[0070] As an optional embodiment, the NeRF network structure F Θ like Figure 3 As shown, it mainly consists of a standard multilayer perceptron. And the position encoder γ is composed of:

[0071]

[0072] in It can optionally contain 4 hidden layers, each with 256 hidden units, and the activation function is ReLU. The position encoder can be a sine / cosine periodic function encoder.

[0073] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp))

[0074] As an optional embodiment, L = 8.

[0075] As an optional embodiment, a schematic diagram of the color and depth of the target star rendered by the NeRF network is shown below. Figure 4 As shown. Based on the estimated camera pose and camera intrinsic parameters, 100 sample points can be collected along the ray emitted from the optical center of the capturing star camera towards the pixel. The depth value of each sample point is d. i (i = 1, 2, ..., 100), the corresponding color and density α can be obtained by querying in the MLP network. i =F Θ (p i The color obtained by rendering is... for:

[0076]

[0077] depth for:

[0078]

[0079] Where the termination probability ω i satisfy:

[0080]

[0081] The depth variance calculated along the ray is:

[0082]

[0083] During the optimization of the MLP network parameters, 10,000 two-dimensional pixels were randomly sampled at the plane of the star-capturing camera, and the photometric loss function L... p It can be calculated that:

[0084]

[0085] Geometric loss function L g for:

[0086]

[0087] Then, the MLP parameters are optimized using a weighted sum of photometric and geometric losses. As an optional implementation, λ is chosen. p =0.2:

[0088]

[0089] The optimization function used in the relative pose estimation process is:

[0090]

[0091] As an optional embodiment, λ is taken. pt =0.25.

[0092] As an optional embodiment, both the MLP parameters and relative pose estimation are optimized using the Adam algorithm, with optimization rates set to 0.001 and 0.05 respectively. The 3D reconstruction results are as follows: Figure 6 (a)~ Figure 6 (c) and Figure 7 (a)~ Figure 7 As shown in (c), the pose estimation results are as follows: Figure 5 As shown.

Claims

1. A method for joint localization and modeling of non-cooperative maneuvering targets in space using NeRF, characterized in that, Includes the following steps: Step 1: Acquire multiple consecutive RGB-D images during the approach to the target star using a monocular depth camera mounted on the capture star; Step 2: In the NeRF network, a neural network is used to characterize the geometric structure and appearance color features of the target star. After rendering, the RGB-D values ​​of each pixel are obtained. The cost function is constructed using the deviation from the true value to optimize the parameters of the neural network, thereby realizing the three-dimensional reconstruction of the structure and appearance of the target star, and simultaneously estimating the pose information of the capture star relative to the target star. Step 2-1: Set up a fixed number of frames as a group. Estimate the pose of the capture star relative to the target star in each frame. The network parameters used are the parameters updated after training in the previous group. After completing the relative pose estimation of the first frame of the current group, start to optimize the network parameters using the first frame. The pose information used is the one estimated from the first frame of the current group. Step 2-2: The NeRF network is a mapping of the coordinates (xyz) and viewing direction (θ φ) of a point in space to color and occupancy probability (rgb α); Position 3D coordinates and viewing direction First, use a mapping function to raise the dimension to a higher dimension. Spatial coordinates are then fed into a neural network to obtain the color and occupancy probability α of a point in three-dimensional space. i ; Steps 2-3: Collect M three-dimensional spatial sample points by emitting a ray from the optical center of the capture star camera towards the two-dimensional pixel sampling point. The depth value of each three-dimensional spatial sample point is d. i For i = 1, 2, ..., M, query the NeRF network to obtain the corresponding color and occupancy probability α. i =F Θ (p i The color obtained by rendering is... for: depth for: Where ω i For p i The termination probability of a point is determined by the occupancy probability of that point. Calculation yielded: The depth variance calculated along the ray is: Steps 2-4: Loss Function; The photometric loss function L used to optimize network parameters and pose estimation p L1 loss between rendered color and actual color: Geometric loss function L g The L1 loss between the rendered depth and the actual depth is normalized using depth variance: The network parameters are then optimized using a weighted sum of photometric and geometric losses: The optimization function used in the relative pose estimation process is:

2. The method for joint localization and modeling of non-cooperative maneuvering targets in space using NeRF as described in claim 1, characterized in that, The NeRF network structure F Θ From a multilayer perceptron And the position encoder γ is composed of: Among them, multilayer perceptron It contains 4 hidden layers, each with 256 hidden units, and the activation function is ReLU; the position encoder is a sine / cosine periodic function encoder. γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp)) Where L = 8.

Images