On-orbit target three-dimensional reconstruction method based on active neural geometric radiation field

Through the three-dimensional reconstruction method of on-orbit targets, the active neural geometric radiation field and reinforcement learning optimized sampling and calculations are solved, and the problems of low efficiency and complex calculations in the three-dimensional reconstruction process of space targets are achieved, achieving efficient and accurate three-dimensional reconstruction effects.

CN119068102APending Publication Date: 2024-12-03TIANMUSHAN LABORATORY
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411069642.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, there are problems such as long sampling period, low efficiency, large data repetition and high computational complexity in the three-dimensional reconstruction process of space targets. Especially in the spacecraft environment where resources are limited, it is difficult for traditional methods to achieve efficient three-dimensional reconstruction. .

Method used

The three-dimensional reconstruction method of on-orbit target based on active neural geometric radiation field is adopted, and the data set construction and offline model training is used, combining the visual Transformer encoder and reinforcement learning behavioral strategy network, the sampling and calculation process is optimized to achieve efficient and accurate three-dimensional reconstruction.

Benefits of technology

It significantly improves the efficiency and accuracy of three-dimensional reconstruction, and optimizes orbiting and shooting strategies through implicit characterization and reinforcement learning search, achieving optimal multi-task end-to-end learning, suitable for resource-constrained spacecraft environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068102A_ABST
    Figure CN119068102A_ABST
Patent Text Reader

Abstract

The invention provides an in-orbit target three-dimensional reconstruction method based on an active neural geometric radiation field. The method comprises the following steps: active in-orbit surrounding sequential image acquisition; constructing a backbone network encoder of the visual Transform with the shared weight; carrying out implicit vector representation; constructing a geometric decoder of the neural radiation field; and constructing an orbital transfer-shooting strategy network based on reinforcement learning. According to the method, large-scale and high-dimensional data can be efficiently processed in parallel by adopting the geometric radiation field model, feature extraction and three-dimensional reconstruction are automatically carried out, higher accuracy and consistency are achieved, the method fully adapts to the on-orbit environment and task requirements of the spacecraft, and the on-orbit observation period is greatly shortened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the intersecting field of space perception, three-dimensional reconstruction and artificial intelligence technology, and specifically relates to a three-dimensional reconstruction method of an on-orbit target based on an active neural geometric radiation field. Background Art

[0002] Asteroids are rocky relics left over from the early formation of the solar system about 4.6 billion years ago. They are called time capsules and contain original materials left over from the formation of the sun and its planets. The study of gases, dust, ice nuclei and comet nucleus structures associated with comets, as well as sampling of organic matter, will help to reveal the history and evolution of the solar system. Asteroid exploration consists of complex tasks such as flyby, flyby reconstruction, landing sampling and return, and controlled impact. Due to the small mass of asteroids, complex rotation, susceptibility to the influence of large mass objects, and dynamic changes in orbit, the space asteroid exploration mission is very difficult.

[0003] On the other hand, on-orbit docking, control and capture of non-cooperative targets also rely on the three-dimensional result information of the target spacecraft to serve the downstream decision-making and control planning. Therefore, efficient fly-by image acquisition and accurate three-dimensional reconstruction of space targets are the basic functions of many space exploration and on-orbit service missions, and have the dual attributes and requirements of space science exploration and aerospace engineering technology application.

[0004] In the existing technical solution, the orbital trajectory planning and accurate 3D reconstruction of the target are two decoupled subtasks, which are carried out independently. The spacecraft takes a large number of redundant shots on different orbital orbits, and then performs 3D reconstruction based on traditional aerial triangulation. On the one hand, there is no specific guidance optimization clue for the orbital shooting, and the unmeasurable shadow area caused by the rotation, high-speed tumbling of the target, and the single light source in the space environment will lead to a long sampling cycle, low sampling efficiency, and large data duplication; on the other hand, when performing 3D reconstruction based on the above data, traditional aerial triangulation modeling needs to traverse each picture to search for the same-name points for initial camera pose alignment, camera intrinsic parameter estimation, basic matrix solution, triangular mesh vertex estimation, sparse-dense scene creation, and then calculate the depth disparity map, sparse point cloud, dense point cloud, and dense triangular surface mesh based on stereoscopic vision affine transformation. The overall process is complex and nested, and some extreme cases still rely on manual correction.

[0005] Neural Radiance Fields (NeRF) reconstructs 3D scenes based on a sparse set of 2D images and can synthesize new perspectives. It performs very well and can complete reconstruction calculations in a stable time. However, the performance of NeRF is greatly affected by the quality of training samples. Due to the limited pose images in the scene, NeRF usually cannot generalize well to new perspectives and may produce poor results in unobserved areas. As a result, NeRF becomes less practical when resources are limited.

[0006] Spacecraft and other space-based carriers are restricted by many factors such as computing platform computing power, volume, power consumption, weight, and heat dissipation. Traditional solutions have slow calculations, low efficiency, and large reconstruction calculations. The emerging neural radiation field three-dimensional reconstruction model is slow to develop due to lack of data. At the same time, the reliance on high-performance GPUs also makes it impossible to directly deploy the model, which greatly limits the efficiency of space exploration missions. Summary of the invention

[0007] To solve the above problems, the present invention proposes a three-dimensional reconstruction method for on-orbit targets based on active neural geometric radiation field, and performs a series of sampling, modeling and computational optimizations for efficient three-dimensional reconstruction of spatial perception targets, which can greatly accelerate the efficiency of target surround, sampling and reconstruction.

[0008] To achieve the above purpose, the technical solution adopted by the present invention is as follows: A method for three-dimensional reconstruction of an on-orbit target based on active neural geometric radiation field, the method comprising the following steps: Step 1, data set construction: collect 3D models of several space on-orbit targets, perform full-view coverage sampling based on a fixed orbit altitude and a camera orientation centered on the target, sample the obtained full-view coverage sampling map through orbit constraints, and obtain a set of sequential simulated sampling views along a certain orbit; Step 2, offline model training: two consecutive frames of sampled views in the sequential simulated sampled view set are input into the 3D reconstruction model based on the active neural geometric radiation field for model training and real-time inference. After image block slicing, they are input into the vision-based Transformer encoder for convolution processing to obtain a set of implicit vector representations, wherein the geometric decoder based on the active neural geometric radiation field performs convolution decoding calculations on the implicit representation vectors to generate point cloud images and pixel-point cloud confidence estimates corresponding to the two sampled views; Adjust the encoder and implicit vector representation of the vision-based Transformer neural network to obtain a three-dimensional representation of the current observation scene, use the three-dimensional representation as the environmental observation input, decode it using the reinforcement learning behavior strategy network, generate the corresponding "track change-shooting" strategy vector with the help of the designed behavior head, use the "track change-shooting" strategy vector to perform full-view coverage sampling again to obtain a new sampled image, and re-train the offline model according to the above steps based on the new sampled image until the active neural geometric radiation field model converges; Step 3: 3D reconstruction of on-orbit targets: Based on the trained active neural geometric radiation field model, active surround 3D reconstruction of new on-orbit targets is performed.

[0009] Furthermore, the step 2 comprises: Step 2.1: The network receives two frames of image I in sequence t and I t+1 , t represents the time, the Patch segmentation strategy is used to divide the two frames of images into a group of sub-images of the same size, which are respectively input into the vision-based Transformer encoder for convolution processing; Step 2.2, fuse the two encoder outputs through a multi-layer perceptron to obtain a set of implicit representation vectors Z for the current observation space; Step 2.3: Two-way geometric decoders based on active neural geometric radiation fields decode the implicit representation vector Z, i.e., 3D reconstruction, i.e., based on image I t The camera input angle restores the first point cloud image corresponding to the rendered image under the target perspective and the first confidence map of each point, and the first confidence map of each point based on the image I t+1 The camera input angle restores a second point cloud image corresponding to the rendered image under the target viewing angle and a second confidence map corresponding to each point; Step 2.4: Adjust the encoder and implicit vector representation of the vision-based Transformer neural network to obtain a three-dimensional representation of the current observation scene. Use the three-dimensional representation as the environmental observation input. The behavior strategy network performs the optimization search for active fly-around strategy and shooting behavior based on reinforcement learning theory, as well as the optimal planning of fly-around direction, trajectory and shooting behavior. Step 2.5: Collect new sample images based on the optimal plan, repeat steps 2.1 to 2.4, and use alternating iterative training to train the model.

[0010] Furthermore, the alternating iteration in step 2.5 includes: Based on the solidified vision-based Transformer encoder and implicit representation vector Z, the behavior strategy network is inferred to generate the active "track change-shooting" strategy corresponding to the current state, and the next step of sampling search is performed based on the strategy; When the generated strategy includes shooting, the 3D reconstruction network is trained based on the new sampled images; When the generated policy does not take a photo, the previous two frames are sampled as input to drive the behavior generation of the behavior policy network; In this way, the two subtask network combinations are alternately trained, wherein the two subtasks include the training of the 3D reconstruction network and the behavior strategy network; The final stopping condition of the model is that the NeRF-based 3D reconstruction loss function drops to the target threshold, or the overall alternating iterative training steps are greater than a predetermined value, and the 3D reconstruction is judged to have failed, and the model training is restarted until the model converges.

[0011] The beneficial effects of the present invention are: The present invention uses NeRF to perform implicit and efficient representation of spatial three-dimensional information, improving the accurate and efficient expression of the observation environment relied on during reinforcement learning search; at the same time, the behavior strategy network uses implicit spatial expression to search for the most optimized active fly-around-change-trajectory-shooting strategy based on the reward function. Combining the 3D reconstruction model of the neural radiation field with the reinforcement learning active fly-by-orbit change-shooting strategy can achieve optimal end-to-end learning of multi-task shooting and reconstruction; the 3D reconstruction branch uses continuous input image data to estimate camera parameters, and uses the point cloud map and confidence of depth map estimation as loss constraints, which can greatly improve the 3D reconstruction efficiency and reconstruction accuracy; the reinforcement learning search space observes the environment based on implicit feature expression, which can greatly improve the ability to represent spatial information; the designed reward function based on on-orbit fly-by-shooting can drive the policy network to comprehensively learn the optimal combination strategy of fly-by and shooting; the designed alternating iterative training method can meet the learning and training of multi-task shared weight heterogeneous models based on loss function convergence and reinforcement learning reward function search. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a network model architecture diagram of an on-orbit target three-dimensional reconstruction method based on active neural geometric radiation field of the present invention; Figure 2 Schematic diagram of discrete full-view sampling of on-orbit targets; Figure 3 It is a schematic diagram of the sequential sampling results under typical orbital constraints; Figure 4 It is a schematic diagram for illustrating the track change angle; Figure 5 This is a schematic diagram of a typical flyby orbit; Figure 6 Schematic diagram of a single camera-to-target observation. DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0014] The present invention discloses a method for three-dimensional reconstruction of on-orbit targets based on active neural geometric radiation field. The method uses an improved neural radiation field to implicitly represent the scene, and then uses the learned three-dimensional efficient representation of the space as the observation space and search space, and uses a reinforcement learning strategy to search and drive the spacecraft to learn the optimal active orbit change and shooting strategy. Finally, the proposed method is verified on a simulation-generated data set.

[0015] like Figure 1The network model architecture shown is 1) sequential image acquisition for active on-track surround; 2) the backbone observation encoder of the shared-weight visual Transformer, 3) implicit vector representation, 4) the geometric decoder of the Neural Radiation Field (NeRF), and 5) the track change-shooting strategy network based on reinforcement learning.

[0016] The spacecraft continuously photographs the target while flying around, and inputs two consecutive frames of images into the network model in sequence.

[0017] Firstly, based on the target's three-dimensional model and the two-body model orbit constraints, discrete full-angle sampling and shooting of the target are carried out. Then, based on the on-orbit constraints of the spacecraft, a typical flyby orbit and a camera shooting posture facing the target are defined to perform a surrounding shooting sampling strategy for sequential data sampling under the typical flyby orbit for subsequent offline model training.

[0018] Then, the observation encoder segments the input image into image blocks to obtain separated small image blocks, which are then input into the ViT-based encoder for convolutional feature encoding. The encoder results of the two branches are output to the MLP for privacy vector generation. Finally, the NeRF decoder is used to decode the point cloud features, and the reconstruction head is used to generate the point cloud and confidence map of the corresponding image view.

[0019] Then, based on the 3D observation encoder and implicit vector encoding, a 3D representation of the current observation scene can be obtained. Based on the 3D privacy representation as the environmental observation input, the reinforcement learning behavior strategy network is used for decoding, and the corresponding "track change-shooting" strategy vector is generated with the help of the designed behavior head, and the reward function is used to evaluate the fuel and energy consumption value of the behavior generated by the spacecraft.

[0020] At the same time, the “orbit change-shooting” strategy is used to shoot the initially constructed discrete full-angle sampling, and step sampling is performed under the current orbit constraints to obtain new sampling images.

[0021] Finally, the network model is input based on the latest sampled image, and training continues according to the above steps until the sampling step threshold is exceeded, and the model training is stopped and restarted; or the NeRF reconstruction branch network converges and the loss function threshold is lower than the preset target.

[0022] The final model can be used for real-time online testing.

[0023] The present invention proposes a method for three-dimensional reconstruction of an on-orbit target based on active neural geometric radiation field. First, Figure 2As shown in the figure, the discrete full-angle sampling of the target is performed by combining the existing space target with the specific camera parameters and orbital altitude. The discrete angle can be customized. For example, at an orbital altitude of h kilometers, the azimuth and elevation are sampled at a certain interval (such as 5 degrees) to obtain full-angle sampling. At the same time, the typical orbit of the spacecraft is defined, including circular orbit, elliptical orbit, double elliptical spliced ​​fly-by orbit, single water drop fly-by orbit, double water drop spliced ​​fly-by orbit, etc.

[0024] Next, data sampling based on orbital constraints: based on the above-mentioned orbital motion constraints and the random starting position of the spacecraft, fly around and search for the full-angle sampling view closest to the current orbit by distance. If no orbit change operation is performed, the current orbit is followed; if an orbit change operation is performed, the orbit is changed with the current position, new motion direction and new orbit type, and the sampling view closest to the current orbit is searched to construct a shooting sampling data sequence under orbital constraints for model training. At the same time, based on each frame of the image and the corresponding camera parameters (including internal and external parameters), the corresponding pixel depth map is calculated for model training. The active orbit change / shooting operation is triggered by the subsequent deep learning model.

[0025] Then, the sequential sampling of the current track is input into the active neural geometric radiation field. Each time the model receives two frames of continuous sampled images under the track constraint, the image blocks are sliced ​​in sequence, and the images are input into the vision-based Transformer (ViT) encoder for convolution processing. Considering the computational constraints and optimization, the two encoders adopt a shared weight strategy, that is, the convolution encoder uses the same set of convolution kernel function parameters in each round of calculation, thereby achieving further feature compression. Since learning is performed through continuous sampling of images, a set of implicit vector representations Z of the current scene can be obtained. As the learning process progresses, the Z vector will gradually converge and improve.

[0026] Among them, the geometric decoder for NeRF adopts the same shared weight strategy, performs convolution decoding calculations on the implicit representation vector Z in turn, and uses the Transfomer decoder to cooperate with the reconstruction head to continuously generate the point cloud map corresponding to the two frames of images and the corresponding pixel-point cloud confidence estimation. Based on the estimated point cloud map and confidence information, the NeRF loss function is calculated with the true value depth map, and the error is back-propagated. Based on continuous data-driven training and stochastic gradient descent, the encoder, implicit vector and decoder parameters of the neural network are adjusted to make the corresponding part of the network converge.

[0027] The behavior strategy network uses the implicit vector representation of the reconstruction network for calculation, and uses the reinforcement learning strategy network to generate the spacecraft behavior, which includes the selection of [movement direction, movement trajectory, and shooting strategy]. The overall behavior strategy reward function (Reward) is designed based on the direction, trajectory, and shooting cost function. The reward function and the minimized execution steps are used to drive the convergence of the "track change-shooting" behavior strategy network based on reinforcement learning.

[0028] Finally, since the overall network is designed for multi-task, an iterative training method is used during model training. That is, the observation encoder, implicit vector representation and NeRF decoder are first trained using non-track-changing continuously sampled image frames; then some parameters of the NeRF decoder are fixed, and the observation encoder, implicit vector and behavior strategy network are iteratively trained; then the behavior strategy network is fixed, and the previous step is re-iterated, and the training is iteratively alternating in sequence. The final model uses the loss function drop threshold of the NeRF branch and the overall execution steps as the model stop condition.

[0029] Specifically, the present invention provides a method for three-dimensional reconstruction of an on-orbit target based on an active neural geometric radiation field, as well as corresponding data set construction, offline model training and online model testing, comprising the following steps: Step 1: Dataset construction. Based on spacecraft, small satellites and other three-dimensional space targets, full-view sampling is performed. With the target origin as the center, the orientation is locked, the movement orbit height is set, and the full-view sampling is performed by stepping through the azimuth and elevation angles of the camera. Each sampling is performed as follows: Figure 6 As shown, the full sampling viewing angle distribution is as follows Figure 2 shown.

[0030] For each sampled RGB image , there is a corresponding depth point cloud image , there is a one-to-one mapping relationship between image pixels and 3D depth point clouds. When the corresponding camera parameters (including intrinsic and extrinsic parameters, the same below) It is known that the depth map of each pixel of the corresponding sampled image can be calculated based on the following projection formula: To the real world point cloud map mapping. And assume that the camera ray can only hit a 3D point cloud, that is, the targets in the world environment are all non-transparent surfaces, to ensure the uniqueness of the mapping. Among them, It is the depth information that maps the 3D point cloud to the pixel coordinates (i, j) on the camera plane.

[0031] , In the formula, the superscript T represents the transpose of the matrix, and the superscript -1 represents the inverse operation of the matrix.

[0032] Since the camera pose changes during sequential sampling, the first camera coordinate of each sampling is used as the world coordinate system. The spatial correlation constraints of two adjacent frames are as follows, where: Point cloud The corresponding mapping transformation from the camera perspective at time t to the camera perspective coordinate system at time t+1 is: It is a homogeneous mapping (x,y,z)-->(x,y,z,1), where x, y, and z represent the three-dimensional spatial coordinates of the point cloud respectively: , Where P is the pose matrix from the world coordinate system to the camera coordinate system, t, t+1 is the time when the camera samples at the corresponding point; Based on the candidate tracks, initial and end positions, the full-view sampled images are further sampled sequentially. The tracks include Figure 5 Some typical orbits are shown. This step simulates the continuous sampling of on-orbit observations of the two-body model under orbital constraints. This step cooperates with subsequent training to achieve continuous data collection of offline active orbit change.

[0033] Among them, the natural elliptical orbit is as follows Figure 5 As shown in the upper left, the minor axis and major axis of the orbit are defined as l and 2l respectively, and the flyby orbit formula is simplified to the two-dimensional plane where the two-body model is located.

[0034] , Among them, the double elliptical controlled flyby Figure 5 As shown in the upper right, pulses are applied at points C and D to achieve periodic flyby. The minor axis and major axis of a single elliptical orbit are also defined as l and 2l respectively. The solid line part is the orbital period T of the flyby. The coordinates of the simplified two-dimensional plane orbit at time t are [x(t), y(t)]. k is the flyby period coefficient, each interval is T / 2, and n is the angular velocity of the reference spacecraft.

[0035]

[0036] The water droplet orbit is as follows: Figure 5 As shown in the lower left, pulses are applied at points C and D to achieve periodic flyby, OA=l, OB=h, T is the orbit period of the flyby, t=T / 2 is the pulse orbit change moment, and n is the reference spacecraft angular velocity.

[0037]

[0038] Among them, the double water droplet splicing orbit is as follows Figure 5 As shown in the lower right, pulses are applied at points C and D to achieve periodic flyby, OA=l, T is the flyby period, t=T / 4 and t'=3T / 4 are the pulse trajectory change moments, and n is the reference spacecraft angular velocity.

[0039]

[0040]

[0041] The value of the flyby period coefficient k will affect the above orbital period.

[0042] Step 2: Offline model training.

[0043] The model architecture is as follows Figure 1 It consists of 4 submodules: 1) Observation encoder, which is a weight-sharing visual Transformer backbone network; 2) Implicit vector representation module, 3) Geometric decoder based on Neural Radiation Field (NeRF), and 4) Behavior strategy network, which is a track change-shooting strategy network based on reinforcement learning.

[0044] Step 2.1: The observation encoder adopts a shared weight strategy. It is an encoder network based on the visual Transformer architecture. The network receives two frames of images I in sequence. t and I t+1 , and first use the Patch segmentation strategy to divide the image into a group of smaller sub-images of the same size, and then input them into the ViT architecture for the convolution encoder part of the self-attention mechanism. At the same time, constrain the input of the two frames of image I t and I t+1 It is the continuous sampling result of the full sampling view under the track constraint, and the two encoders adopt a shared weight strategy, that is, the network architecture of the two encoders is the same, and the parameters of the convolution kernel training are also consistent.

[0045] Step 2.2, then through the multi-layer perceptron ( , MLP) fuses the outputs of the two encoders to obtain a set of implicit representation vectors Z for the current observation space.

[0046] Step 2.3: For the NeRF 3D reconstruction branch, the two decoders are decoded based on a set of implicit representation vectors Z, that is, based on I t The camera input angle restores the first point cloud image corresponding to the rendered image under the target perspective and the first confidence map of each point, as well as I t+1 The camera input angle restores the second point cloud map corresponding to the rendered image under the target perspective and the second confidence map corresponding to each point.

[0047] Based on step 2.1, we have an image (I t and I t+1 ) and camera extrinsics (K t and K t+1) Calculate the true value point cloud map and , can be calculated pixel by pixel based on Euclidean distance (Corresponding view )’s loss function: , scale and represents the scaling factor and the normalization of the scaling factor, and ||·|| represents the bi-norm.

[0048] At the same time, in order to prevent the random scale difference between the predicted results and the true value, the scaling factor scale=norm and Normalize.

[0049] The average distance between normalized pixels is .

[0050] The confidence loss function is calculated as follows: is the prediction confidence of pixel i, is the parameter that controls the regularization term.

[0051] , The NeRF branch uses an encoder to implicitly represent the vector set Z. The encoder reconstructs the three-dimensional target and uses the confidence loss function to drive the model convergence.

[0052] Step 2.4: With the help of the implicit representation vector set Z of the three-dimensional space, the behavior strategy network performs an optimization search for active circling strategies and shooting behaviors based on reinforcement learning theory, and performs optimal planning of circling directions, orbits, and shooting behaviors.

[0053] The active fly-around reconstruction is reduced to a discrete-time Markov decision process. In the state space S and the behavior set A, the transition probability from state s to s' caused by behavior a can be expressed as T(s'|s,a). At the same time, the spacecraft will give a reward function R(s,a,s') to calculate the reward value of each round of search. To balance the current and future weights, is the initial state of the system. The active fly-around strategy of reinforcement learning finds the optimal strategy by reducing the reward function Unlike traditional reinforcement learning, which uses low-dimensional and simplified spatial representation, the encoding of the target spatial position is used. , target 3D shape encoding , together constitute the state s=( ) can improve the model's use of spatial information. In fact, based on the observation of the scene directly obtained by the camera is an RGB image. The generation of the fly-around strategy based on the observation information by the policy network can be summarized as In step 2.1, two consecutive frames of observation images are used as input, so .

[0054] At the same time, through progressive learning, the encoder network obtains an implicit scene representation of the overall space based on the learning of multiple groups of y, and reduces it to an implicit vector of kk dimensions based on the MLP specification Ω is the behavior strategy mapping function. Reinforcement learning is based on strategy network modeling, and Z is used as the state representation of the search space. , through the implicit feature vector group Z constructed in the middle of the NeRF network, the decision of the behavior a of the active fly-by strategy on orbit can be replaced by the original image-based input y to the implicit representation z of space, that is, . This enables the model to learn the optimal fly-around strategy for the current scene target observation based on the reward function.

[0055] Different from 3D reconstruction relying on loss function learning, the reinforcement learning branch composed of observation encoder, implicit vector and behavior strategy network uses reward function to drive the model to learn the optimal fly-around reconstruction strategy. Specifically, it includes three types of behaviors.

[0056] Direction change behavior, with the current state of the spacecraft as 0, assuming that the spacecraft needs to change the current orbiting direction, such as Figure 4 The required orbit change angle of the spacecraft is shown in The absolute value of is proportional to the fuel consumption, and the reward function can be reduced to .

[0057] The orbit change behavior takes the current orbit of the spacecraft as the initial state. When it needs to switch to another orbit, each switch consumes a fixed amount of fuel. , the reward function can be reduced to .

[0058] Shooting behavior: Each time the spacecraft moves to a certain position, it can decide whether to shoot. Reducing the shooting interval can avoid the introduction of redundant data in reconstruction, but too large a shooting interval will lead to information loss. By actively selecting and sampling dense perspectives, we can learn an optimized sampling interval strategy. Assuming that each sampling consumes a fixed amount of fuel , the reward function can be reduced to .

[0059] The reward function of the overall active fly-around optimization strategy ,in , is the weight constant.

[0060] Step 2.5: Alternating iterative training. Since the entire network includes two subtasks: 3D reconstruction of spatial scenes and learning of fly-around strategies, the model training is performed using alternating iterative training.

[0061] First, in the initial state, continuous data is sampled based on the current track, and two frames of data are input into the observation encoder. Then, the implicit representation Z is calculated based on the MLP, and then the decoder is calculated. The reconstruction head is used to generate the point cloud map and confidence map, and the error calculation and stochastic gradient descent are performed based on the reconstruction loss function to adjust the overall network weights.

[0062] Next, based on the solidified observation encoder and implicit representation Z, the strategy network is inferred to generate the active track change-shooting behavior corresponding to the current state, and the next step of sampling search is carried out based on the strategy.

[0063] When the generated strategy includes shooting, the 3D reconstruction network is trained alternately based on the new samples.

[0064] When the generated policy does not take pictures, the behavior generation of the behavior policy network is driven based on the previous two frame samples as input.

[0065] In this way, the two sub-task network combinations are trained alternately.

[0066] The final stopping condition of the model is that the 3D reconstruction loss function based on NeRF drops to the target threshold, or the total alternating iterative training steps are greater than a certain predetermined value, and the reconstruction is judged to have failed, and the model training is restarted until the model converges.

[0067] Step 3: Based on the online test of the converged model parameters, the new target is actively reconstructed by flying around. Each iterative step drives the current agent to fly around, photograph, and reconstruct the target in 3D. After several photographing steps, the 3D structure of the target is reconstructed based on the 3D reconstruction network branch.

[0068] Example Step 1: First, collect the 3D models of several space targets on orbit, based on the fixed orbit altitude and the camera orientation centered on the target (such as Figure 6 As shown in the figure, full-view coverage sampling is performed. A full sampling view with a fixed interval angle is obtained (as shown in the figure). Figure 2 Then, based on the trajectory constraints (the alternative simulation trajectories are as follows Figure 5 As shown, but not limited to this) and the camera starting position, the nearest sampling view along the corresponding track can be calculated, and the full sampling view can be sampled through the track constraint to obtain the sequential simulation sampling view along a certain track, such as Figure 3As shown in Figure 2, the corresponding actual point cloud depth map is calculated based on the camera external parameter K as training label data.

[0069] Step 2: Based on the sampled images obtained in every two consecutive frames as input, the model is trained, and the image blocks are processed in sequence, and input into the encoder based on ViT, the implicit vector generation module based on the multi-layer perceptron, and the decoder based on NeRF, and the corresponding point cloud map and confidence map are generated based on the reconstructed head. Then, based on the generated point cloud map and the real GT, the error is solved according to the reconstruction loss function, and the parameters of the entire network model are trained based on directional propagation and stochastic gradient descent.

[0070] Then, the network parameters of the observation encoder module and the implicit vector generation module are fixed, and the behavior policy network is trained to generate the corresponding motion direction change, track change and shooting instructions based on the current two frames of information input, and the energy consumption reward function generated by the corresponding behavior is calculated. The reinforcement learning model implicitly judges and evaluates the spatial observation information of the target observation each time, and generates the corresponding track change and shooting behavior driving sampling mode.

[0071] If the orbit is not changed, the spacecraft continues to move along the current orbit, taking the shooting status into consideration; if the instruction is to shoot, sampling continues, and the latest two consecutive frames of images are used as model input for training. If the instruction is not to shoot, sampling is skipped once, and the original two consecutive frames of images are used as model input for training.

[0072] If the orbit is changed, the spacecraft selects a new direction and orbit to move, while taking the shooting status into consideration; if the instruction is to shoot, sampling continues, and the latest two consecutive frames of images are used as model input for training. If the instruction is not to shoot, sampling is skipped once, and the original two consecutive frames of images are used as model input for training.

[0073] Based on the latest sampling results, the 3D reconstruction module and the strategy generation module are trained alternately. If the number of model adoptions reaches the preset upper limit, it means that the current reconstruction has failed, the trained network weights are retained, and the training is restarted; if the loss function of the 3D reconstruction branch drops to the target threshold and the number of samplings does not reach the preset target, it means that the network has converged.

[0074] Step 3: Based on the converged network model parameters, actively reconstruct the three-dimensional orbit of the new on-orbit target, drive the orbiting and shooting behaviors based on the model output, and use the continuous shooting images as model input to drive the model to calculate and track a series of optimized orbiting and shooting behaviors, as well as a set of continuous shooting images, and finally the three-dimensional target results reconstructed by the model.

[0075] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for three-dimensional reconstruction of on-orbit targets based on active neural geometric radiation field, characterized in that: The method comprises the following steps: Step 1, data set construction: collect 3D models of several space on-orbit targets, perform full-view coverage sampling based on a fixed orbit altitude and a camera orientation centered on the target, sample the obtained full-view coverage sampling map through orbit constraints, and obtain a set of sequential simulated sampling views along a certain orbit; Step 2, offline model training: two consecutive frames of sampled views in the sequential simulated sampled view set are input into the 3D reconstruction model based on the active neural geometric radiation field for model training and real-time inference. After image block slicing, they are input into the vision-based Transformer encoder for convolution processing to obtain a set of implicit vector representations, wherein the geometric decoder based on the active neural geometric radiation field performs convolution decoding calculations on the implicit representation vectors to generate point cloud images and pixel-point cloud confidence estimates corresponding to the two sampled views; Adjust the encoder and implicit vector representation of the vision-based Transformer neural network to obtain a three-dimensional representation of the current observation scene, use the three-dimensional representation as the environmental observation input, decode it using the reinforcement learning behavior strategy network, generate the corresponding "track change-shoot" strategy vector with the help of the designed behavior head, use the "track change-shoot" strategy vector to perform full-view coverage sampling again, obtain a new sampled image, and re-train the offline model according to the above steps based on the new sampled image until the active neural geometric radiation field model converges; Step 3: 3D reconstruction of on-orbit targets: Based on the trained active neural geometric radiation field model, active surround 3D reconstruction of new on-orbit targets is performed.

2. The method for three-dimensional reconstruction of on-orbit targets based on active neural geometric radiation field according to claim 1, characterized in that: The step 2 comprises: Step 2.1: The network receives two frames of image I in sequence t and I t+1 , t represents the time, the Patch segmentation strategy is used to divide the two frames of images into a group of sub-images of the same size, which are respectively input into the vision-based Transformer encoder for convolution processing; Step 2.2, fuse the two encoder outputs through a multi-layer perceptron to obtain a set of implicit representation vectors Z for the current observation space; Step 2.3: Two-way geometric decoders based on active neural geometric radiation fields decode the implicit representation vector Z, i.e., 3D reconstruction, i.e., based on image I t The camera input angle restores the first point cloud image corresponding to the rendered image under the target perspective and the first confidence map of each point, and the first confidence map of each point based on the image I t+1 The camera input angle restores a second point cloud image corresponding to the rendered image under the target viewing angle and a second confidence map corresponding to each point; Step 2.4: Adjust the encoder and implicit vector representation of the vision-based Transformer neural network to obtain a three-dimensional representation of the current observation scene. Use the three-dimensional representation as the environmental observation input. The behavior strategy network performs the optimization search for active fly-around strategy and shooting behavior based on reinforcement learning theory, as well as the optimal planning of fly-around direction, trajectory and shooting behavior. Step 2.5: Collect new sample images based on the optimal plan, repeat steps 2.1 to 2.4, and use alternating iterative training to train the model.

3. The method for three-dimensional reconstruction of on-orbit targets based on active neural geometric radiation field according to claim 2, characterized in that: The alternating iterations in step 2.5 include: Based on the solidified vision-based Transformer encoder and implicit representation vector Z, the behavior strategy network is inferred to generate the active "track change-shooting" strategy corresponding to the current state, and the next step of sampling search is performed based on this strategy; When the generated strategy includes shooting, the 3D reconstruction network is trained based on the new sampled images; When the generated policy does not take a photo, the previous two frames are sampled as input to drive the behavior generation of the behavior policy network; In this way, the two subtask network combinations are alternately trained, wherein the two subtasks include the training of the 3D reconstruction network and the behavior strategy network; The final stopping condition of the model is that the NeRF-based 3D reconstruction loss function drops to the target threshold, or the overall alternating iterative training steps are greater than a predetermined value, and the 3D reconstruction is judged to have failed, and the model training is restarted until the model converges.

Citation Information

Cited By

  • Reinforcement learning training method and system for space-time interaction dynamic three-dimensional reconstruction

    CN121562717A

  • A reinforcement learning training method and system for spatiotemporal interactive dynamic three-dimensional reconstruction

    CN121562717B