A Three-Dimensional Reconstruction Method for Space Debris Based on Glancing-Flyby Observation Imaging

Abstract

The present invention provides a method for three-dimensional reconstruction of space debris based on fly-by observation imaging, belonging to the field of three-dimensional reconstruction, which solves the problem of how to accurately predict the future activity trend of space debris, including: obtaining optical observation images after meeting the fly-by observation constraints; performing SFM preprocessing, multi-view internal feature point extraction, multi-view inter-feature point matching and bundle adjustment on the optical observation images to obtain sparse point clouds and pose information; dividing the preprocessed images into a training set, a test set and a validation set; taking the sparse point clouds and pose information as initial conditions and inputting them into the implicit neural representation function of NeRF simultaneously with the training set, mapping the input into a feature vector after position encoding and association positioning, training the MLP, and outputting and rendering the optimal fitting result to obtain a three-dimensional reconstructed optical observation image. The present invention evaluates the working state of space debris by analyzing the three-dimensional reconstructed optical observation image and predicts the future activity trend of space debris.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction of space debris, and particularly relates to a method for three-dimensional reconstruction of space debris based on flyby observation imaging. Background Art

[0002] Due to the continuous increase in the number of satellites in space, carrying out Space Situational Awareness (SSA) tasks, monitoring and obtaining the attitude information of space debris to avoid collisions between spacecraft has become a key link in maintaining space security. At the same time, various space activities have also put forward higher requirements for technical capabilities in aspects such as spacecraft rendezvous and docking, on-orbit diagnosis and repair of faulty satellites, and space debris cleaning. Therefore, the research on methods for imaging and three-dimensional reconstruction of space debris is extremely urgent, which helps to improve the overall autonomy and detection ability of the space system.

[0003] At present, the imaging observation methods for space debris in China mainly include radar detection imaging and optical detection imaging. Radar detection belongs to an active imaging method. However, since the energy amplitude of the received signal is inversely proportional to the square of the distance of the space debris, it is often limited by the detection distance and difficult to perform imaging detection on space debris in the high-orbit band. Therefore, for geostationary orbit (GEO) satellites, the current mainstream is still to adopt the optical imaging detection method. As a passive detection method, it is divided into two types of platforms: space-based and ground-based. However, ground-based optical imaging is limited by factors such as atmospheric disturbance, weather conditions, geographical location, and time window, resulting in unstable image quality and limited resolution, and it is difficult to perform high-quality and high-frequency observations. For example, ground-based optical detection is easily interfered by factors such as atmospheric turbulence and cloud clusters, and the duration of the observation window is limited, resulting in poor imaging quality.

[0004] Compared with ground-based optical observations, space-based optical detection is not limited by geographical location and meteorological conditions, can provide more stable and clear images, is not affected by the atmosphere, and can achieve global coverage, all-weather, and high temporal resolution observations, thus showing significant advantages in the field of space debris monitoring; and currently, the detection equipment capabilities in China already have the function of high-resolution imaging at medium and long distances. In addition, compared with infrared observations, optical observations have the advantages of being passive and passive, low energy consumption, and suitable for long-term operation. Therefore, the data of space debris imaging by space-based optical detection is an important data source in the field of space debris monitoring such as the three-dimensional structure, attitude change, and motion state estimation of space debris. Exploring efficient and accurate three-dimensional reconstruction methods based on time-series space-based visible light images can provide information such as structural dimensions and surface texture details required for on-orbit services in real time, which helps to improve the overall autonomy, unmanned, and intelligent level of the space system.

[0005] According to the different orbital heights and motion characteristics of the observed objects, space-based optical imaging is mainly divided into the following three typical imaging modes: low-orbit rendezvous observation, high-orbit flyby observation, and high-orbit orbiting observation. Among them, the optical observation satellite conducts optical imaging of the target satellite at the orbital position that meets the imaging conditions. Among them, in the patent with the publication number CN114970180A and the invention title "An In-orbit Optimization Method for Spacecraft Flyby Observation", the constraint conditions during the imaging process are divided into imaging part angle constraints, sunlight constraints, attitude angular velocity constraints, observation distance constraints, and safety distance constraints, etc., which provides certain inspiration for the design of the imaging mode. In Li Mengxi. Research on Space Target Attitude Estimation Technology Based on Space-based Optical Imaging [D]. National University of Defense Technology, 2022, in order to estimate the in-orbit three-dimensional attitude of spacecraft, two space target attitude estimation problems based on space-based optical imaging are mainly considered, namely the attitude estimation problems in the rendezvous scenario of low-Earth orbit (LEO) space targets and the flyby scenario of geosynchronous orbit (GEO) space targets. The feature association between the target and the known three-dimensional model is constructed to calculate the target attitude, but the three-dimensional structure and surface texture details of the space target are not completely reconstructed. In the patent with the publication number CN118604840A and the invention title "A Satellite System and Method for Cooperative Detection and Imaging of Low-Earth Orbit Targets from Low-Earth Orbit", the satellite is provided with three working modes: flyby imaging, orbiting imaging, and joint detection and positioning modes, but only the imaging mode is discussed, and the further processing of visible light images is not involved.

[0006] After obtaining the space-based visible light image, image preprocessing, three-dimensional reconstruction, and attitude estimation are the key technologies for on-orbit services. In recent years, the three-dimensional reconstruction technology of space debris based on sequential visible light images has gradually become a research hotspot in the field of space debris identification and monitoring. Relevant research is of great significance for improving the situation awareness ability and accurate identification level of space debris. The essence of three-dimensional reconstruction of space debris is to reconstruct the three-dimensional model of space debris through multi-view two-dimensional visible light images, and realize the identification of features such as the surface material, structure, and texture of space debris. Since three-dimensional reconstruction was first proposed by Roberts in 1963, it has experienced the development from traditional methods, deep learning methods to end-to-end methods.

[0007] The following process is the main step of the traditional 3D reconstruction method. Refer to: Zhao Shujia. Research on Key Technologies of 3D Reconstruction of Space Targets under Low Illumination [D]. Xidian University, 2022. The traditional 3D reconstruction methods are sorted out, mainly including the method combining Structure from Motion (SFM) and Multi-View Stereo (MVS). The main forms of expression of space targets are sparse and dense reconstruction of point clouds. The common process of 3D reconstruction of space targets includes but is not limited to the following steps: (1) Image acquisition: Obtain images of the target through a multi-view camera; (2) Camera calibration: Determine the internal and external parameters of the camera; (3) Feature extraction and matching: Extract features from the images and perform matching; (4) Camera pose estimation: Estimate the relative pose of the camera through the matched feature points; (5) Triangulation: Calculate the positions of feature points in 3D space; (6) Sparse point cloud construction: Generate a sparse 3D point cloud; (7) Dense point cloud generation: Generate a dense point cloud through multi-view matching; (8) Mesh reconstruction: Convert the point cloud into a triangular mesh model; (9) Texture mapping: Map the texture information of the image onto the mesh surface; (10) Model optimization: Denoise and smooth the 3D model for optimization; (11) Result evaluation: Evaluate the reconstruction quality of the model. Publication number: CN106408650A, Invention name: A Method for 3D Reconstruction and Measurement of Space Targets by On-Orbit Flyby Imaging. The advantage of the disclosed technical solution is that space-based imaging can quickly respond to the needs of reconnaissance, without strict observation climate conditions and observation windows. However, the method adopted is mainly based on traditional methods, and it is slightly inferior in terms of reconstruction effect and timeliness.

[0008] Since 2021, at the International Conference on Computer Vision, the most widely studied method is the Neural Radiance Fields (NeRF) method, which has reached a certain degree of maturity after years of accumulation. Ma Hansheng, Zhu Yuhua, Li Zhihui, Yan Lei, Si Yiyi, Lian Yimeng, Zhang Yuhan. Review of Neural Radiance Fields for Multi-View Synthesis [J]. Computer Engineering and Applications, 2024, 60(4): 21-38. This paper reviews and summarizes the key algorithms in the field of neural radiance fields in recent years. First, it introduces the background and principle of neural radiance fields, and then classifies and discusses subsequent key improved models. Neural Radiance Fields (NeRF) is an end-to-end 3D reconstruction method based on neural networks. It generates high-quality 3D views by learning the radiance field of the scene. The core idea of NeRF is to map each position and direction in the scene to color and volume density. By training on images collected from different angles, the model can learn the complete 3D scene structure and achieve realistic rendering results from any perspective. Different from traditional 3D reconstruction methods, NeRF does not rely on explicit geometric data or mesh models, but generates 3D scenes in an implicit representation way. This method can better handle complex details and lighting changes, making it perform well in fields such as view synthesis, scene reconstruction, and virtual reality.

[0009] However, when the above solution is applied in practice, the following technical defects exist:

[0010] 1. Observation platform

[0011] Compared with ground-based optical detection, space-based optical detection is not restricted by geographical location and meteorological conditions, and is an important data source for estimating the three-dimensional structure, attitude change, and motion state of space debris. Ground-based optical imaging is limited by factors such as atmospheric disturbance, weather conditions, geographical location, and time window, resulting in unstable image quality, limited resolution, and difficulty in performing high-quality and high-frequency observations. In contrast, space-based optical imaging can break through these limitations, provide more stable and clear images, be unaffected by the atmosphere, and can achieve global coverage, all-weather, and high temporal resolution observations, thus showing significant advantages in the field of space debris monitoring.

[0012] 2. Space-based optical imaging modes

[0013] Space-based optical imaging modes are mainly divided into three working modes: rendezvous, fly-around, and fly-by observation imaging.

[0014] (1) Rendezvous observation imaging

[0015] The rendezvous observation satellite can only observe space debris within a specific time window. Usually, multiple satellites need to cooperate to achieve full coverage, and there is a limited coverage range; the observation time for each transit is short, only about ten seconds, resulting in a small number of acquired observation images and a small proportion of effective pixels of the debris in the observation images, making it difficult to meet the image perspective conditions required for three-dimensional reconstruction; and in order to maintain continuous observation of the debris, orbital maneuvers need to be carried out through continuous small thrust, resulting in a relatively high observation cost and consuming certain resources.

[0016] (2) Fly-around observation imaging

[0017] Although fly-around observation imaging can continuously track and observe debris in a specific area, due to the relatively long distance from the debris, the image resolution is low, and the ratio of the debris in the image is relatively low; it is difficult to quickly adjust the orbit for emergency response observations of sudden events; and the cost of launch and operation and maintenance management is relatively high.

[0018] (3) Fly-by observation imaging

[0019] The flyby observation satellite has a relatively long observation time for each transit, enabling complex observation tasks such as multiple imaging and high-precision tracking. It combines the advantages of low-orbit rendezvous and high-orbit flyby imaging modes, can frequently access the same debris, provides a high temporal resolution, and is suitable for detecting the dynamic changes of space debris. It has strong flexibility and mobility, can quickly adjust the orbit and observation angle to meet the observation requirements of different tasks, and can capture high-resolution visible light images with image quality meeting the requirements of three-dimensional reconstruction.

[0020] 3. Three-dimensional reconstruction method

[0021] Three-dimensional reconstruction based on dense point clouds performs well in terms of accuracy and robustness, can capture the details of objects. Especially in complex scenes, the algorithm can adaptively process noise and occlusion to a certain extent, ensuring the accuracy of three-dimensional reconstruction and having good adaptability. In addition, point cloud data does not require pre-defined grids, so it is suitable for processing objects with complex shapes, and its disorder and sparsity make parallel processing possible, improving the computational efficiency. However, the sparsity of point cloud data often leads to a reduction in reconstruction accuracy, especially in long-distance or complex scenes, so it may not perform well in scenes with sparse images in the field of space non-cooperative targets; the disorder of point cloud data makes feature extraction more complex, and due to the lack of topological information, the texture details of complex geometric shapes may not be effectively presented during reconstruction. Under complex lighting conditions, the accuracy of three-dimensional reconstruction based on depth point clouds may be greatly restricted; the lack of labeled data is also a great challenge. Point cloud data is huge in quantity, and manual annotation is very time-consuming and costly. Especially in high-precision tasks, the lack of labeled data may limit the performance and application effect of the model.

[0022] Multi-view Stereo (MVS) in three-dimensional reconstruction, as a core issue in computer vision and photogrammetry, has been widely studied. Traditional MVS methods have shown good results in ideal environments (Lambertian surfaces and rich textures). However, in practical applications, due to the existence of variables such as weak textures, non-Lambertian surfaces, and photometric changes, especially for complex space environments, the reconstruction effect is often poor, so there is still much room for improvement in this method.

[0023] In summary, there are thousands of abandoned satellites, rocket upper stages, and some other objects distributed in the geosynchronous orbit. These space debris pose a certain threat to the normal operation of satellites. Therefore, space-based situation awareness is particularly important for ensuring the safe and stable operation of future spacecraft. The present invention mainly adopts a space-based optical imaging mode of flyby imaging for observing space debris, and on this basis, conducts research on a three-dimensional reconstruction method based on the neural radiance field algorithm. Summary of the invention

[0024] Among the commonly used space debris monitoring methods, the high-resolution visible light images obtained by space-based optical measurement are suitable for attitude estimation and three-dimensional reconstruction, which are of great significance for enhancing space visual perception ability. In addition, the optical detection system has a powerful passive monitoring ability and does not need to actively emit signals, performing particularly well in complex or concealed detection environments. Therefore, in order to safeguard space interests and space security, ensure space activities, and assist space decision-making, it is urgent to obtain the current characteristic information of space debris and predict the future activity trends of space debris. The space characteristic analysis method mainly based on optical detection is one of the key technologies for space situation awareness.

[0025] To solve the technical problem of how to obtain the current characteristic information of space debris and predict the future activity trends of space debris, the present invention provides a three-dimensional reconstruction method for space debris based on flyby observation imaging. This method takes the neural radiance field algorithm as the basic principle and the space-based optical flyby imaging as the working mode of the typical scenario, providing an end-to-end space-based optical imaging method for the typical scenario, as well as a three-dimensional reconstruction process and method based on the combination of structure from motion (SFM) and an improved neural radiance field algorithm, providing an important reference for on-orbit services. The space-based platform can obtain high-definition images of specific perspective parts of the debris through flyby observation at close range in orbit, and can evaluate the working state of the debris by constructing the fine component-level features of the space debris, thereby predicting the future activity trends of the space target. The technical solution disclosed by the present invention safeguards space interests and space security, ensures space activities, and can also assist space decision-making.

[0026] To achieve the above object, the present invention provides a three-dimensional reconstruction method for space debris based on flyby observation imaging, including the following steps:

[0027] Step 1: The flyby observation constraint is composed of the observation distance constraint, the observation geometry constraint, the illumination condition constraint, and the detection limit constraint. After the flyby observation imaging mode of the space-based platform satisfies the flyby observation constraint, an optical observation image of the space debris is obtained and transmitted to the ground station.

[0028] Step 2: After preprocessing operations of image enhancement and image restoration are performed on the received optical observation image through the structure from motion algorithm, a preprocessed image is obtained; multi-view internal feature points are extracted from the preprocessed image by a multi-scale feature extraction network; the ORB algorithm is used to match the extracted feature points between multiple views, and then a sparse point cloud and pose information are obtained through bundle adjustment.

[0029] Step 3: Divide the preprocessed image into a training set, a test set, and a validation set according to a preset ratio; Prune the external points of the sparse point cloud by the confidence loss function to obtain an optimized point cloud; Use the optimized point cloud and pose information as initial conditions and input them into the implicit neural representation function of the neural radiance field representing the flyby observation scenario simultaneously with the training set. After position encoding and association positioning, map all the inputs into feature vectors, and input the feature vectors into the fully connected network of the neural radiance field algorithm for training to obtain a trained fully connected network model. After testing and validating the trained fully connected network model with the test set and the validation set, output the optimal fitting result; Use volume rendering technology to render the optimal fitting result in three-dimensional space and output the three-dimensional reconstructed optical observation image of the space debris.

[0030] In Step 1, the observation distance constraints include: observation position constraint, optical axis angle constraint, and communication security distance constraint; among them, the observation position constraint is: where d min is the minimum effective observation distance, ξ is the spatially transformable scale coefficient, R target is the observation range radius of the space debris, A FOV is the field of view angle of the optical camera;

[0031] The optical axis angle constraint is: where β is the optical axis angle between the optical axis and the optical camera, β ob is the half field of view angle of the observation platform, and β is less than β ob ; r CM is the optical axis connection line, that is, the observation line of sight, r TC is the distance vector from the space debris to the optical camera;

[0032] The communication security distance constraint is: the distance between each observation platform is less than the effective communication security distance threshold;

[0033] In Step 1, the observation geometric constraints include the observation platform constraint and the angle constraint between the observation platform and the space debris; among them, the observation platform constraint is:

[0034] The angle constraint between the observation platform and the space debris is:

[0035] where r C is the geocentric vector of the observation platform carrying the optical camera, r TC is the distance vector from the space debris to the optical camera, R E is the radius of the earth, θ is the angle between the geocenter-optical camera-space debris, and the angle range is 0 to 180°;

[0036] In step one, the illumination condition constraints include: avoiding direct sunlight constraint, avoiding the space debris being in the earth's shadow area constraint, and observation attitude constraint; where

[0037] The avoiding direct sunlight constraint is: being in the front-light condition, the illumination angle conforms to the threshold, and the sun does not directly shine on the lens of the observation platform during observation, satisfying 0 < α < β < π or In the formula, α is the sum of the apparent radius of the sun and the light scattering angle and is a non-negative acute angle, r CS is the distance vector from the observation platform to the sun, r TC is the distance vector from the space debris to the optical camera, β is the angle between the optical axis and the optical axis of the optical camera, that is, the angle between r CS and r TC ;

[0038] The avoiding the space debris being in the earth's shadow area constraint is: or In the formula, λ is the angle between the geocentric vector r T of the space debris and the geocentric vector r S of the sun, R E is the radius of the earth;

[0039] The observation attitude constraint is: the angle between the optical axis of the optical camera carried by each observation platform and the ideal observation line of sight is less than the angle threshold, ensuring that the activity area of the space debris is within the field of view of the optical camera;

[0040] In step one, the detection limit constraint is:

[0041]

[0042] In the formula, M is the stellar magnitude corresponding to the space debris, R S,N,0 is the signal-to-noise ratio of the system when reaching the system detection limit, is the average photon flux of the sporadic stellar magnitude, A is the area of the light-passing aperture of the receiver, η is the quantum efficiency of the detector for the spectrum of the space debris, t e is the integration time, τ is the transmittance of the optical system to the space debris signal, N D is the number of dark signal electrons, N R is the number of readout noise electrons, N s is the number of signal electrons output by the optical camera during the detection time of the optical camera, that is, the integration time, is the average photon flux of the m-th stellar magnitude, and m is the number of the stellar magnitude.

[0043] In step two, the method for preprocessing the received optical observation image by using the structure from motion algorithm for image enhancement includes:

[0044] Regarding the optical observation image whose image brightness does not reach the threshold as a low-illumination image;

[0045] Input the low-light image into the encoder of the Structure from Motion (SFM) algorithm to generate an illumination-invariant color map;

[0046] Map the normal exposure image into a reversible network of latent codes;

[0047] Optimize the reversible network using maximum likelihood estimation. Utilize the illumination-invariant color map and learn the latent mapping between the low-light image and the normal exposure image through the optimized reversible network for image enhancement. Through end-to-end iterative training, make the regression loss function converge and output an optical observation image whose image brightness reaches the threshold;

[0048] In step two, the method for preprocessing the received optical observation image through the Structure from Motion algorithm for image restoration includes:

[0049] Simplify the estimated blur kernel through the regression loss. Use an adaptive least squares filter to remove noise, blur, and / or artifact features in the optical observation image and output the restored optical observation image;

[0050] After all the optical observation images have undergone the preprocessing operations of image enhancement and image restoration, a preprocessed image is obtained.

[0051] In step two, the method for optimizing the reversible network using maximum likelihood estimation, utilizing the illumination-invariant color map, and learning the latent mapping between the low-light image and the normal exposure image through the optimized reversible network for image enhancement includes:

[0052]

[0053] where L(x l , x ref ) is the image similarity between the low-light image and the normal exposure image, f flow (x ref |x l ) is the probability density function of the normal exposure image, x l is the low-light image, x ref is the normal exposure image, f z (Θ(x ref ; x l )) is the probability density function of the reversible network learning the mapping between the normal exposure image and the low-light image based on the latent code z, f z is the probability density function of the latent code z, Θ(x ref ; x l ) is the reversible network for learning the relationship between the normal exposure image and the low-light image, Θ is the reversible network, which is divided into a sequence of N reversible layers {θ 1 , θ 2,...θ i ...,θ n}, N = 1 to n, i = 0 to N - 1; z n is the nth potential code, g n (x l ) is the nth illumination-invariant color map generated by the encoder g from the low-light image, with a shape compatible with the Nth reversible layer θ n ;

[0054] A method for achieving convergence of the regression loss function through end-to-end iterative training and outputting an optical observation image with the image brightness reaching a threshold includes:

[0055] Collect paired samples (x l , x ref ), and train Θ by minimizing the l1 reconstruction regression loss function:

[0056] The regression loss function is: argminE[l1(Θ(x l ), x ref )] = argmaxE[logf(Θ(x l )|x ref )];

[0057] where E is the expected value, l1 is the L1 norm distance, also known as the absolute value distance, which is a method for measuring the distance between two sets of vectors or points; Θ(x l ) is the optical observation image with the image brightness reaching the threshold output by Θ, and f(Θ(x l )|x ref ) is the probability density function conditional on x ref , defined as follows:

[0058]

[0059] where b is the learning rate constant;

[0060] A method for estimating the blurred kernel by simplifying the regression loss and outputting the restored optical observation image of the image by removing the noise, blur, and / or artifact features in the optical observation image using an adaptive least squares filter includes:

[0061] D a = H a y;

[0062]

[0063] where D α is the denoised, deblurred, and / or de-artifacted feature corresponding to x h ↓ s , x hIt is the optical observation image after the output image is restored, ↓ s represents downsampling; H α is a feature-specific operator, a is the number of the optical observation image; y is the degraded image, F(·) represents the Fourier transform, F -1 (·) represents the inverse Fourier transform, represents the conjugate of F(·), k b is the blurred kernel estimated by simplifying the regression loss, p b represents the filtering parameter; b is the number of the blurred kernel, represents the convolution operation, k h represents the blurred kernel in the high-resolution space, and μ refers to the noise.

[0064] In step two, the method for extracting multi-view internal feature points from the preprocessed image by the multi-scale feature extraction network includes:

[0065] Input the preprocessed image into the multi-scale feature extraction network MsFe;

[0066] Starting from the resolution of the initially input preprocessed image, perform downsampling multiple times according to preset different scales; after reaching the lowest resolution, start upsampling and combine the features of the previous layer as the input for the next upsampling to obtain a high-quality feature map; remove the overlapping corner points in the high-quality feature map through non-maximum suppression (NMS) to obtain the feature points extracted within the multi-view.

[0067] In step two, the method for performing multi-view inter-feature point matching on the extracted feature points and obtaining sparse point cloud and pose information through bundle adjustment includes:

[0068] Use the ORB algorithm to calculate the distance between the feature descriptors corresponding to each feature point, and match the corresponding feature points between multi-views by whether the distance range of the feature descriptors is within the threshold to obtain the matching feature point pairs;

[0069] Perform geometric verification on the matching feature point pairs to ensure that the matching points satisfy the observation geometric constraints;

[0070] Use the EPnP algorithm to estimate the relative pose between the preprocessed images of each pair of matching feature points that satisfy the observation geometric constraints; calculate the mean of the relative poses to obtain the initial camera pose; use the initial camera pose and the matching feature point pairs to calculate the positions of the 3D points through the triangulation method;

[0071] Optimize the reprojection error of the positions of the 3D points and the initial camera pose through bundle adjustment BA, and output the sparse point cloud and pose information.

[0072] In step 3, the method of pruning the external points of the sparse point cloud by the confidence loss function to obtain the optimized point cloud includes:

[0073] P = {(p j , f j , γ j ) | j = 1, 2,...,, N p};

[0074]

[0075] Among them, P is the optimized point cloud, p j represents the position of the three-dimensional point of the sparse point cloud, f j represents the feature vector of the three-dimensional point of the sparse point cloud, γ j ∈ [0, 1] represents the confidence value of the three-dimensional point of the sparse point cloud, j is the number of sparse point clouds, and N p is the total number of sparse point clouds; L sparse is the confidence loss value of the confidence loss function, γ is the confidence or probability value, between 0 and 1.

[0076] In step 3, the implicit neural expression function of the neural radiance field is:

[0077] F β : (γ(x), γ(d)) → (c, σ);

[0078] Among them, F β is the implicit neural expression function of the neural radiance field with respect to the coordinate x and the feature vector d of the line-of-sight direction; γ(x) is the confidence of the spatial position, γ(d) is the confidence along the ray direction, the weight parameter β is optimized to generate the volume density σ and the directional emission color c, γ is the confidence or probability value, between 0 and 1, and is applied to the predefined position embedding of the coordinate x and the feature vector d of the line-of-sight direction to map the input to a higher-dimensional space;

[0079] The method of mapping all inputs to feature vectors after position encoding and association positioning includes:

[0080] x' = [x, γ L (x)];

[0081] γ L (x) = [sin(x), cos(x),..., sin(2 L-1 x), cos(2 L-1 x)];

[0082] In the formula, x' is the feature vector, x is the coordinate, and γ L (x) is the mapping function, and L is the hyperparameter that controls the maximum encoding frequency.

[0083] In step 3, the method of inputting the feature vector into the fully connected network of the neural radiance field algorithm for training to obtain a trained fully connected network model includes:

[0084] Select sampling points for each pixel in the feature vector;

[0085] Use the fully connected network MLP of the neural radiance field algorithm NeRF to generate the density and color of the sampling points, and use the volume rendering technique to render the color of each sampling point and integrate along the sampling ray to generate a rendered image;

[0086] The fully connected network is divided into the first-layer network sampling and the second-layer network sampling. The total number of samples for a single sampling ray sample is N c +N f , N c is the number of hierarchical sampling positions of the first-layer network; N f is the number of hierarchical sampling positions of the second-layer network; The first-layer network hierarchically samples N c positions to obtain the weights of the first-layer network hierarchical sampling samples; The second-layer network continues to sample N f positions based on the weights of the first-layer network hierarchical sampling samples to obtain the weights of the second-layer network hierarchical sampling samples; Optimize the scene representation by minimizing the L2 norm distance loss value between the rendered image and the real image through the L2 norm distance loss function of the second-layer network hierarchical sampling samples to obtain a trained fully connected network model;

[0087] Among them, the calculation method of the weight is: In the formula, is the actual value of the weight, w n' is the theoretical value of the weight, w m' is the initial value of the weight of each sampling ray sample in the network, n' is the weight number, and m' is the initial value number of the weight;

[0088] The L2 norm distance loss function is:

[0089] Among them, L is the L2 norm distance loss value, r represents a single sampling ray sample, represents all sampling rays; C(r) is the real image, is the rendered image output by the first-layer network, is the rendered image output by the second-layer network.

[0090] In step 3, the volume rendering technique is:

[0091]

[0092] Where \(C(r)\) is the real image. In three-dimensional space, \(r(t)\) represents a point, and \(d\) represents the line-of-sight direction feature vector; the proximal end of the scene is \(t\). n , and the distal end is \(t\). f ; \(c(r(t),d)\) is the color value seen when observing the point \(r(t)\) from the line-of-sight direction feature vector \(d\); and \(\sigma(r(t))\) represents the volume density function, which reflects the light absorption ability of the material at the point \(r(t)\); \(T(t)\) represents the cumulative transmittance from the starting \(t\). n to the position \(t\), \(\sigma(r(s))\) is the change value of the integral path region during the integral calculation of the volume density function, \(s\) is the integral upper limit variable, and \(t\) is a single integration step length.

[0093] The present invention has the following beneficial effects:

[0094] 1. The present invention designs a flyby observation imaging mode for a space-based platform in a typical scenario and analyzes in detail the flyby observation constraints visible to the optical system of the observation satellite. Due to the limitations of the real space environment and combining the working characteristics of different orbits, the flyby observation imaging mode of the space-based platform can have the advantages of a high revisit coverage rate, a small energy consumption rate, and long-term operation, and can focus on high-resolution imaging and three-dimensional reconstruction of space debris and malfunctioning satellites.

[0095] 2. After preprocessing operations such as image enhancement and image restoration on the received optical observation images through the Structure from Motion (SFM) algorithm, the present invention obtains a preprocessed image. By iteratively minimizing the reprojection error, sparse point clouds and pose information are generated. The method based on the learning model has obvious advantages over the traditional iterative traversal method, in that the algorithm has strong adaptability and generalization ability, and the accuracy of the point cloud coordinates and pose matrix obtained on unseen datasets is relatively high, solving the problem of point cloud loss caused by occlusion and measurement errors in complex scenes.

[0096] 3. The present invention considers the problem of space waste. In blank areas, due to data uncertainty, reconstruction defects may occur, such as floating objects. The sparse point clouds obtained through the Structure from Motion (SFM) algorithm are used as the optimized input conditions for the Neural Radiance Field (NeRF) method to obtain the initial pose information. On this basis, combined with the neural radiance field method, all scene attribute information is stored on the points through neural point clouds. Through position encoding and 3D segmentation, a feature query method for any point in space is continuously generated, and a new view image is rendered, greatly improving the rendering speed of the three-dimensional reconstruction of space targets and saving the resource consumption of data transmission and ground station data processing.

[0097] 4. The combination of the neural radiance field and point clouds greatly improves the rendering speed of three-dimensional reconstruction, which has great practical significance for limited on-board resources and transmission rates, and lays a foundation for realizing lightweight on-board automatic processing. Brief Description of the Drawings

[0098] Figure 1 is a schematic flow chart of a three-dimensional reconstruction method for space debris based on fly-by observation imaging provided by the present invention.

[0099] Figure 2 is a schematic diagram of an example of a fully-connected network architecture with positional encoding for a neural radiance field provided by the present invention. Detailed Embodiments

[0100] As Figure 1 shown, in an embodiment of the present invention, a three-dimensional reconstruction method for space debris based on fly-by observation imaging is provided, including the following steps:

[0101] Step 1: A fly-by observation constraint is constituted by an observation distance constraint, an observation geometry constraint, an illumination condition constraint, and a detection limit constraint. After the fly-by observation imaging mode of the space-based platform satisfies the fly-by observation constraint, an optical observation image of the space debris is obtained and transmitted to the ground station.

[0102] Step 2: After preprocessing operations of image enhancement and image restoration are performed on the received optical observation image by the Structure from Motion (SFM) algorithm, a preprocessed image is obtained; multi-view internal feature points are extracted from the preprocessed image by a multi-scale feature extraction network (MsFe); the feature points extracted are matched between multi-views by the ORB algorithm, and then sparse point clouds and pose information are obtained through bundle adjustment.

[0103] Step 3: The preprocessed image is divided into a training set, a test set, and a validation set according to a preset ratio (such as 7:2:1); the outlier points of the sparse point cloud are pruned by a confidence loss function to obtain an optimized point cloud; the optimized point cloud and pose information are used as initial conditions and input into an implicit neural representation function of a neural radiance field (NeRF) representing the fly-by observation scene together with the training set, so that the output results are made compatible to achieve the purpose of efficient rendering; after positional encoding and association for positioning, all inputs are mapped into feature vectors, and the feature vectors are input into a fully-connected network (MLP) of the neural radiance field algorithm (NeRF) for training to obtain a trained fully-connected network model. After testing and validating the trained fully-connected network model with the test set and the validation set, an optimal fitting result is output; volume rendering technology is used to render the optimal fitting result in three-dimensional space, and a three-dimensional reconstructed optical observation image of the space debris is output. Finally, information such as the three-dimensional structure, surface texture, and material details of the space debris can be inverted.

[0104] In Step 1, the observation distance constraint includes: an observation position constraint, an optical axis angle constraint, and a communication security distance constraint; among them, the observation position constraint is: where d minis the minimum effective observation distance, ensuring that the field of view of the optical camera should cover the space debris activity area as much as possible; ξ is the space-transformable scale coefficient, and R target is the observation range radius of the space debris, and A FOV is the field of view angle of the optical camera;

[0105] The constraint of the optical axis angle is: In the formula, β is the optical axis angle between the optical axis and the optical camera, and β ob is the half field of view angle of the observation platform, and β is less than β ob ; r CM is the optical axis connection line, that is, the observation line of sight, and r TC is the distance vector from the space debris to the optical camera;

[0106] The communication security distance constraint is: the distance between each observation platform is less than the effective communication security distance threshold;

[0107] In step one, the observation geometric constraint includes the observation platform constraint and the included angle constraint between the observation platform and the space debris, ensuring that the space-based observation platform and the space debris can be visible to each other; among them, the observation platform constraint is:

[0108] The included angle constraint between the observation platform and the space debris is:

[0109] In the formula, r C is the geocentric vector of the observation platform carrying the optical camera, and r TC is the distance vector from the space debris to the optical camera, and R E is the radius of the earth, θ is the included angle between the geocenter-optical camera-space debris, and the included angle range is 0 to 180°;

[0110] In step one, the space debris can be optically visible when the lighting conditions are met in space. Usually, when observing the space debris by the satellite carrying the optical camera, it is necessary to avoid the background of direct sunlight, the space debris cannot appear in the earth's shadow area, and ensure that the activity area of the space debris is as much as possible within the field of view of the optical camera, so as to ensure that the sunlight reflected by the space debris can be captured by the detection system of the observation station. The influence of the moon is not considered here for the time being. The lighting condition constraints include: avoiding direct sunlight constraint, avoiding the space debris being in the earth's shadow area constraint and observation attitude constraint; among them,

[0111] The avoiding direct sunlight constraint is: being in the front light condition, the lighting angle conforms to the threshold, the sun does not directly shine on the lens of the observation platform during observation, and avoid the observation platform from appearing in the solar disk and its nearby areas, satisfying 0 < α < β < π or In the formula, α is the sum of the solar apparent radius and the light scattering angle and is a non-negative acute angle, and r CSis the distance vector from the observation platform to the sun, r TC is the distance vector from the space debris to the optical camera, and β is the angle between the optical axis and the optical axis of the optical camera, that is, r CS and r TC the included angle;

[0112] If the space debris is in the earth's shadow area, the space debris cannot reflect sunlight, and the observation platform cannot image. Therefore, when observing and imaging space debris, it is necessary to avoid the space debris being in the earth's shadow area. The constraint to avoid the space debris being in the earth's shadow area is: or In the formula, λ is the included angle between the geocentric vector r T of the space debris and the heliocentric vector r S of the sun, and R E is the radius of the earth;

[0113] The observation attitude constraint is: the angle between the optical axis of the optical camera carried by each observation platform and the ideal observation line of sight is less than the angle threshold to ensure that the active area of the space debris is within the field of view of the optical camera;

[0114] The space-based detection system has a certain detection limit. From an energy perspective, if the visible light signal intensity of the space debris meets the minimum detection threshold of the detector, the detection system can detect the space debris. Whether the observation platform is equipped with a CCD camera or a CMOS camera, the principle of imaging the space debris is to receive the sunlight reflected by the space debris and convert the optical signal into an electrical signal through structures such as photosensitive elements, so as to image the space debris. In step one, the detection limit constraint is:

[0115]

[0116] In the formula, M is the magnitude value corresponding to the space debris, and R S,N,0 is the signal-to-noise ratio of the system when reaching the system detection limit, is the average photon flux of the zero magnitude, A is the area of the light-passing aperture of the receiver, η is the quantum efficiency of the detector for the spectrum of the space debris, t e is the integration time, τ is the transmittance of the optical system for the space debris signal, N D is the number of dark signal electrons, N R is the number of readout noise electrons, N s is the number of signal electrons output by the optical camera during the detection time of the optical camera, that is, the integration time, is the average photon flux of the m magnitude, and m is the magnitude number.

[0117] Due to the special nature of the illumination in the space environment, the optical observation images of space debris obtained under low-light conditions often carry noise and have varying degrees of degradation. The image degradation process can be represented as a combination of blurring, downsampling, and noise. Therefore, preprocessing operations need to be performed first. In step two, the method for preprocessing the received optical observation images through the Structure from Motion (SFM) algorithm for image enhancement includes:

[0118] Regarding the optical observation images whose image brightness does not reach the threshold as low-light images;

[0119] Inputting the low-light images into the encoder of the Structure from Motion (SFM) algorithm to generate an illumination-invariant color map;

[0120] Mapping the normally exposed images into a reversible network of latent codes;

[0121] Optimizing the reversible network using maximum likelihood estimation, using the illumination-invariant color map, learning the latent mapping between the low-light images and the normally exposed images through the optimized reversible network for image enhancement, and achieving convergence of the regression loss function through end-to-end iterative training, and outputting the optical observation images whose image brightness reaches the threshold;

[0122] In step two, the method for preprocessing the received optical observation images through the Structure from Motion (SFM) algorithm for image restoration includes:

[0123] Simplifying the estimated blurring kernel through the regression loss, and using an adaptive least squares filter to remove the noise, blurring, and / or artifact features existing in the optical observation images, and outputting the optical observation images after image restoration;

[0124] Among them, in the process of image restoration, the blurring kernel refers to the convolution kernel that causes image blurring. The present invention regards the regression loss of blurring kernel estimation as a sub-problem of the regression loss describing the difference between images. By the regression loss, describing the difference between images, the estimation process of the blurring kernel can be simplified. The regression loss of blurring kernel estimation can evaluate and optimize the estimation of blurring kernel parameters and the quality of the restored images, making them closer to the real images.

[0125] After all the optical observation images have undergone the preprocessing operations of image enhancement and image restoration, preprocessed images are obtained.

[0126] In order to better characterize high-quality normally exposed images, in step two, the method for optimizing the reversible network using maximum likelihood estimation, using the illumination-invariant color map, and learning the latent mapping between the low-light images and the normally exposed images through the optimized reversible network for image enhancement includes:

[0127]

[0128] Among them, L(x l , x ref ) is the image similarity between the low-light image and the normally exposed image, f flow (x ref |x l ) is the probability density function of the normally exposed image, x l is the low-light image, x ref is the normally exposed image, f z (Θ(x ref ; x l )) is the probability density function of the reversible network based on the latent code z for learning the mapping between the normally exposed image and the low-light image, f z is the probability density function of the latent code z, Θ(x ref ; x l ) is the reversible network for learning the relationship between the normally exposed image and the low-light image, Θ is the reversible network, which is divided into a sequence of N reversible layers {θ 1 , θ 2 ,... θ i ..., θ n}, N = 1 to n, i = 0 to N - 1; z n is the nth latent code, g n (x l ) is the nth illumination-invariant color map generated by the encoder g from the low-light image, and its shape is compatible with the Nth reversible layer θ n ;

[0129] The method of making the regression loss function converge through end-to-end iterative training and outputting an optical observation image with the image brightness reaching the threshold includes:

[0130] Collect paired samples (x l , x ref ), and train Θ by minimizing the l1 reconstruction regression loss function:

[0131] The regression loss function is: argminE[l1(Θ(x l ), x ref )] = argmaxE[logf(Θ(x l )|x ref )]; The regression loss function is used to minimize the similarity between the low-light image and the normally exposed image, minimize this objective function, and obtain the trained Θ, aiming to minimize the expected value of the L1 norm distance between the predicted value and the reference value of the model.

[0132] Among them, E is the expected value, l1 is the L1 norm distance, also known as the absolute value distance, which is a method for measuring the distance between two sets of vectors or points. Θ(x l) is the optical observation image where the image brightness output by Θ reaches the threshold, f(Θ(x l )|x ref ) is the probability density function conditional on x ref and is defined as follows:

[0133]

[0134] where b is the learning rate constant;

[0135] The method for estimating the blurred kernel by regression loss simplification and using an adaptive least squares filter to remove noise, blur, and / or artifact features in the optical observation image and output the restored optical observation image of the image includes:

[0136] D a =H a y;

[0137]

[0138]

[0139] where D α is the denoised, deblurred, and / or de-artifacted feature corresponding to x h ↓ s , x h is the restored optical observation image of the output image, ↓ s represents downsampling; H α is the feature-specific operator, a is the number of the optical observation image; y is the degraded image, F(·) represents the Fourier transform, F -1 (·) represents the inverse Fourier transform, represents the conjugate of F(·), k b is the blurred kernel estimated by regression loss simplification, p b represents the filtering parameter; b is the number of the blurred kernel, represents the convolution operation, k h represents the blurred kernel in the high-resolution space, and μ refers to the noise.

[0140] In step two, the method for extracting multi-view internal feature points from the preprocessed image by the multi-scale feature extractor (MsFe) includes:

[0141] Input the preprocessed image into the multi-scale feature extractor MsFe;

[0142] Starting from the preprocessed image resolution of the initial input, downsampling is performed multiple times at preset different scales; after reaching the lowest resolution, upsampling begins and the features of the previous layer are combined as the input for the next upsampling to obtain a high-quality feature map; overlapping corner points in the high-quality feature map are removed through Non-Maximum Suppression (NMS) to ensure a more uniform distribution of corner points, and the feature points extracted within multiple views are obtained. By detecting corner points on images of different scales, the scale invariance of the detection is improved.

[0143] In step two, the ORB (Oriented FAST and Rotated BRIEF) algorithm is used to perform feature point matching among multiple views for the extracted feature points. The ORB algorithm is an efficient feature detection and description algorithm widely used in real-time computer vision tasks. It combines the advantages of a multi-scale feature extraction network and the BRIEF descriptor, and at the same time introduces some improvements to enhance the robustness and accuracy of the features. The method of obtaining sparse point clouds and pose information through Bundle Adjustment (BA), that is, the method of iteratively minimizing the reprojection error, includes:

[0144] Calculate the distance between the feature descriptors corresponding to each feature point using the ORB algorithm, and match the corresponding feature points among multiple views according to whether the distance range of the feature descriptors is within the threshold to obtain matching feature point pairs;

[0145] Perform geometric verification on the matching feature point pairs to ensure that the matching points satisfy the observed geometric constraints;

[0146] Preferably, geometric verification of the matching feature point pairs is performed through the fundamental matrix or the essential matrix; among them, the fundamental matrix is used to process uncalibrated images, that is, the case where the camera internal parameters are unknown. The fundamental matrix describes the geometric relationship between the matching feature point pairs in two images, and is especially suitable for images taken from two different perspectives. It is a 3x3 non-singular matrix with a rank of 2.

[0147] The essential matrix is similar to the fundamental matrix. The essential matrix is a matrix based on the assumption that two cameras have been calibrated, that is, the internal parameters of the two cameras are known, such as focal length, principal point position, etc. The essential matrix describes the rotation and translation relationship between two cameras, and is the relationship matrix after converting the matching feature point pairs in two images from the pixel coordinate system to the normalized coordinate system. The essential matrix is also a 3x3 matrix, and its form can be composed of a rotation matrix and a translation vector.

[0148] Use the EPnP algorithm to estimate the relative pose between pre - processed images of each pair of matched feature point pairs that satisfy the observed geometric constraints; calculate the mean of the relative poses to obtain the initial camera pose; use the initial camera pose and the matched feature point pairs to calculate the positions of the 3D points through triangulation;

[0149] Optimize the reprojection error of the positions of the 3D points and the initial camera pose through bundle adjustment (BA), refine the initial camera pose and the positions of the 3D points, and output the sparse point cloud and pose information.

[0150] In step three, the method of pruning the outlier points of the sparse point cloud by the confidence loss function to obtain the optimized point cloud includes:

[0151] P = {(p j , f j , γ j ) | j = 1, 2,...,, N p};

[0152]

[0153] where P is the optimized point cloud, p j represents the position of the 3D point of the sparse point cloud, f j represents the feature vector of the 3D point of the sparse point cloud, γ j ∈[0, 1] represents the confidence value of the 3D point of the sparse point cloud, j is the number of sparse point clouds, and N p is the total number of sparse point clouds; L sparse is the confidence loss value of the confidence loss function, γ is the confidence or probability value, between 0 and 1.

[0154] In step three, the implicit neural representation function of the neural radiance field is:

[0155] F β : (γ(x), γ(d)) → (c, σ); represents a continuous scene as a 5D vector - valued function, whose input is a 3D position and a 2D viewing direction, and whose output is color and volume density.

[0156] where F β is the implicit neural representation function of the neural radiance field with respect to the coordinate x and the feature vector d of the viewing direction; γ(x) is the confidence of the spatial position, γ(d) is the confidence along the ray direction, the weight parameter β is optimized to generate the volume density σ and the directional emission color c, γ is the confidence or probability value, between 0 and 1, and is applied to the predefined position embedding of the coordinate x and the feature vector d of the viewing direction to map the input to a higher - dimensional space;

[0157] Directly optimizing NeRF on the original input (x, d) usually leads to difficulties in synthesizing high-frequency details. To solve this problem, sine functions with different frequencies are used to map the input to a higher-dimensional space to enhance the details and textures in the rendering and achieve better 3D reconstruction accuracy. Therefore, the present invention uses the method of mapping all inputs into feature vectors after position encoding association positioning to solve this technical problem. The method includes:

[0158] x′ = [x, γ L (x)];

[0159] γ L (x) = [sin(x), cos(x),..., sin(2 L-1 x), cos(2 L-1 x)];

[0160] In the formula, x′ is the feature vector, x is the coordinate, and γ L (x) is the mapping function, and L is the hyperparameter that controls the maximum encoding frequency.

[0161] In step three, the method of inputting the feature vector into the fully connected network MLP of the neural radiance field algorithm NeRF for training to obtain a trained fully connected network model includes:

[0162] Select sampling points for each pixel in the feature vector;

[0163] Use the fully connected network MLP of the neural radiance field algorithm NeRF to generate the density and color of the sampling points, and use the volume rendering technique to render the color of each sampling point and integrate along the sampling ray to generate a rendered image;

[0164] The fully connected network is divided into the first-layer network sampling and the second-layer network sampling. The total number of samples for a single sampling ray sample is N c +N f , N c is the number of positions for the first-layer network hierarchical sampling; N f is the number of positions for the second-layer network hierarchical sampling; The first-layer network hierarchically samples N c positions to obtain the weights of the first-layer network hierarchical sampling samples; The second-layer network continues to sample N f positions based on the weights of the first-layer network hierarchical sampling samples to obtain the weights of the second-layer network hierarchical sampling samples; The L2 norm distance loss value between the rendered image and the real image is optimized by minimizing the L2 norm distance loss function with the weights of the second-layer network hierarchical sampling samples to obtain a trained fully connected network model;

[0165] Among them, the calculation method of the weight is: In the formula, is the actual value of the weight, wn' is the theoretical value of the weight, w m' is the initial value of the weight of each sampled ray sample in the network, n' is the number of the weight, and m' is the number of the initial value of the weight;

[0166] The L2 norm distance loss function is:

[0167] where L is the L2 norm distance loss value, r represents a single sampled ray sample, represents all sampled rays; C(r) is the real image, is the rendered image output by the first-layer network, is the rendered image output by the second-layer network. The L2 norm distance: the square of the Euclidean distance, which is the square root of the sum of the squares of the differences of the two vector components.

[0168] The volume rendering technique is used to render each color of the pixel. For a given ray origin and the viewing direction along the pixel, the i-th sample on the ray is defined as where t i is extracted by stratified sampling from N uniformly spaced bounded partitions [t n , t f . In step three, the volume rendering technique is:

[0169]

[0170] In the formula, C(r) is the real image. In three-dimensional space, r(t) represents a point, and d represents the view direction feature vector; the near end of the scene is t n , and the far end is t f ; c(r(t), d) is the color value seen when observing the point r(t) from the view direction feature vector d; and σ(r(t)) represents the volume density function, which reflects the absorption ability of the material at the point r(t) to light; T(t) represents the cumulative transmittance from the starting t n to the t position, σ(r(s)) is the change value of the integral path region during the integral calculation of the volume density function, s is the integral upper limit variable, and t is a single integration step.

[0171] After volume rendering in three-dimensional space, a new perspective visible light two-dimensional image with high resolution is finally generated and output, that is, the three-dimensional reconstruction optical observation image of the space debris.

[0172] To more clearly illustrate the technical solution of the present invention, specific examples are provided below to detail the technical solution of the present invention:

[0173] Example 1: Analysis of the grazing imaging mode

[0174] Taking the typical scenario of observing and imaging in the forward sunlight of a GEO orbit as an example:

[0175] 1) Visible window analysis

[0176] For a GEO satellite with an inclination angle not equal to 0°, it passes through the equatorial plane twice a day, at local times from 6:00 to 18:00 and from 18:00 to 6:00 the next day respectively.

[0177] 2) Observation geometric constraints

[0178] When observing at the moment when the space debris passes near the equator, considering the position of the sun, when the sun is located in the upper rear when the space debris passes the equator, the satellite needs to maneuver to the upper rear of the space debris for imaging; when the space debris is located in the upper front when passing the equator, the satellite needs to maneuver to the upper front of the space debris for imaging.

[0179] 3) Illumination condition constraints

[0180] If the inspection orbit is located below the space debris, it is preferred to conduct a close-range observation of it during the time period from 18:00 to 6:00 the next day, and conduct forward sunlight imaging of it from below or the lower side of the space debris. At this time, it meets the illumination conditions of the forward sunlight angle. According to the simulation results, when the local time is from 23:24 to 00:36, the sun is located below the space debris, and the satellite should image the space debris from below; but with the revolution of the satellite, during the spring equinox or autumn equinox of each year, the space debris is in the earth's shadow, and at this time the visible light camera cannot image it, and this observation period should be excluded, and a multi-satellite cooperation method should be adopted to conduct fly-by imaging observations from other perspectives.

[0181] Example 2: Fully connected network MLP and position encoding

[0182] Although a neural network can be used as a universal function approximator, when directly inputting vector coordinates into the network, the rendering effect is not good in terms of high-frequency changes in color and geometric shape. Deep networks tend to learn low-frequency functions. By mapping the input into a high-frequency function in a high-dimensional space and then passing it to the network, data containing high-frequency changes can be better fitted. As Figure 2 shown, it is a schematic diagram of a fully connected network architecture.

[0183] 1) Fully connected network architecture

[0184] It mainly includes an input vector, an intermediate hidden layer, and an output vector. All layers are standard fully connected layers. Layers with ReLU activation are represented by black arrows, layers without activation are represented by orange arrows, layers with Sigmoid activation are represented by dashed black arrows, and “+” represents vector concatenation.

[0185] 2) Position encoding process

[0186] ①Follow the structure of DeepSDF and add a skip connection to connect the input to the activation of the fifth layer;

[0187] ②The output volume density σ is corrected by ReLU to ensure non - negative values, and at the same time, a 256 - dimensional feature vector is output;

[0188] ③This feature vector is concatenated with the positional encoding of the input view direction, that is, the confidence γ(d) along the ray direction, and is processed by an additional fully - connected ReLU layer with 128 channels;

[0189] ④The final layer outputs the RGB radiance observed at the position coordinate x along the line - of - sight direction feature vector d with a Sigmoid activation layer.

[0190] 3) Positional encoding method

[0191] The way of performing positional encoding on the neural radiance field is as follows:

[0192] γ L (πx)=(sin(2 0 πx),cos(2 0 πx),...,sin(2 L-1 πx),cos(2 L-1 πx))

[0193] 4) Positional encoding results

[0194] γ(·) is independently applied to each dimension of the vector to be encoded. When encoding the position vector x, L is taken as 10, and when encoding the view vector d, L is taken as 4. After positional encoding, the encoding dimension of x is 60, and the encoding dimension of d is 24.

[0195] The embodiments of the present invention have the following beneficial effects:

[0196] 1. The present invention designs a fly - by observation imaging mode for a space - based platform in a typical scenario and details the fly - by observation constraints for the optical visibility of the observation satellite. Due to the limitations of the real space environment and combining the working characteristics of different orbits, the fly - by observation imaging mode of the space - based platform can have the advantages of high revisit coverage rate, small energy consumption rate, and long - term operation, and can focus on high - resolution imaging and 3D reconstruction of space debris and malfunctioning satellites.

[0197] 2. After performing preprocessing operations of image enhancement and image restoration on the received optical observation images through the Structure from Motion (SFM) algorithm for motion recovery structure, a preprocessed image is obtained. By iteratively minimizing the reprojection error, sparse point clouds and pose information are generated. The method based on the learning model has obvious advantages over the traditional iterative traversal method, which lies in the strong adaptive ability and generalization of the algorithm. The accuracy of the point cloud coordinates and pose matrices obtained on unseen datasets is relatively high, and the problem of point cloud loss caused by occlusion and measurement errors in complex scenes is solved.

[0198] 3. The present invention takes into account the problem of space waste. In blank areas, due to the uncertainty of data, reconstruction defects may occur, such as floating objects. The sparse point clouds obtained through the Structure from Motion (SFM) algorithm are used as the optimized input conditions for the NeRF method to obtain the initial pose information. On this basis, combined with the neural radiance field method, all scene attribute information is stored on points through neural point clouds. Through position encoding and 3D segmentation, a feature query method for any point in space is continuously generated, and a new perspective image is rendered, greatly improving the rendering speed of 3D reconstruction of space targets and saving the resource consumption of data transmission and ground station data processing.

[0199] 4. The combination of neural radiance fields and point clouds greatly improves the rendering speed of 3D reconstruction, which has great practical significance for limited on-board resources and transmission rates, and lays a foundation for realizing lightweight on-board automatic processing.

[0200] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A three-dimensional reconstruction method for space debris based on grazing flight observation imaging, characterized in that, It includes the following steps: Step 1: The flyby observation constraint is composed of the observation distance constraint, the observation geometry constraint, the illumination condition constraint, and the detection limit constraint. After the flyby observation imaging mode of the space-based platform satisfies the flyby observation constraint, an optical observation image of the space debris is obtained and transmitted to the ground station; Step 2: After preprocessing operations of image enhancement and image restoration are performed on the received optical observation image through the Structure from Motion (SFM) algorithm, a preprocessed image is obtained; multi-view internal feature points are extracted from the preprocessed image by a multi-scale feature extraction network; the ORB algorithm is used to match the feature points between multiple views, and then sparse point clouds and pose information are obtained through bundle adjustment; Step 3: The preprocessed image is divided into a training set, a test set, and a validation set according to a preset ratio; the outlier points of the sparse point cloud are pruned by a confidence loss function to obtain an optimized point cloud; the optimized point cloud and pose information are used as initial conditions and input into the implicit neural representation function of the neural radiance field representing the flyby observation scenario together with the training set. After position encoding and correlation positioning, all inputs are mapped into feature vectors, and the feature vectors are input into the fully connected network of the neural radiance field algorithm for training to obtain a trained fully connected network model. After testing and validating the trained fully connected network model through the test set and the validation set, the optimal fitting result is output; the volume rendering technology is used to render the optimal fitting result in three-dimensional space, and a three-dimensional reconstructed optical observation image of the space debris is output.

2. The method according to claim 1, characterized in that, In step one, the observation distance constraint includes: observation position constraint, optical axis angle constraint, and communication safety distance constraint; among which, the observation position constraint is: In the formula, d min is the minimum effective observation distance, ξ is the space-transformable scale coefficient, R target is the observation range radius of the space debris, A FOV is the field of view angle of the optical camera; The constraint of the optical axis angle is as follows: In the formula, β is the optical axis angle between the optical axis and the optical camera, and β ob is the half field of view angle of the observation platform, and β is less than β ob ; r CM is the optical axis connection line, that is, the observation line of sight, and r TC is the distance vector from the space debris to the optical camera; The communication security distance constraint is: the distance between each observation platform is less than the effective communication security distance threshold; In Step 1, the observation geometric constraints include the observation platform constraints and the angle constraints between the observation platform and the space debris; among them, the observation platform constraints are: The included angle constraint between the observation platform and the space debris is: where r C is the geocentric vector of the observation platform carrying the optical camera, and r TC is the distance vector from the space debris to the optical camera, R E is the radius of the Earth, and θ is the angle between the geocenter-optical camera-space debris, and the range of the angle is 0 to 180°; In Step 1, the illumination condition constraint includes: the constraint of avoiding direct sunlight, the constraint of avoiding the space debris being in the earth's shadow area, and the observation attitude constraint; among them, The constraint for avoiding direct sunlight is as follows: under frontlighting conditions, the illumination angle meets the threshold, and the sun does not directly shine on the lens of the observation platform during observation, satisfying 0 < α < β < π or where α is the sum of the apparent radius of the sun and the light scattering angle and is a non-negative acute angle, r CS is the distance vector from the observation platform to the sun, r TC is the distance vector from the space debris to the optical camera, β is the angle between the optical axis and the optical axis between the optical cameras, that is, the angle between r CS and r TC ; The constraint for avoiding space debris in the earth's shadow region is: or where λ is the angle between the geocentric vector r T of the space debris and the heliocentric vector r S , and R E is the radius of the earth; The observation attitude constraint is: the angle between the optical axis of the optical camera carried by each observation platform and the ideal observation line of sight is less than the angle threshold, ensuring that the activity area of the space debris is within the field of view of the optical camera; In Step 1, the detection limit constraint is: where M is the stellar magnitude corresponding to the space debris, and R S,N,0 is the signal-to-noise ratio of the system when reaching the system detection limit, is the average photon flux of the sporadic magnitude, A is the area of the light-gathering aperture of the receiver, η is the quantum efficiency of the detector for the space debris spectrum, t e is the integration time, τ is the transmittance of the optical system for the space debris signal, N D is the number of dark signal electrons, N R is the number of readout noise electrons, N s is the number of signal electrons output by the optical camera during the detection time of the optical camera, i.e., the integration time, is the average photon flux of the m-th stellar magnitude, and m is the number of the stellar magnitude.

3. The method according to claim 2, wherein In Step 2, the method for performing the preprocessing operation of image enhancement on the received optical observation image through the Structure from Motion (SFM) algorithm includes: The optical observation image with an image brightness not reaching the threshold is regarded as a low-light image; The low-light image is input into the encoder of the Structure from Motion (SFM) algorithm to generate an illumination-invariant color map; The normal exposure image is mapped into a reversible network of latent codes; The reversible network is optimized using maximum likelihood estimation. Using the illumination-invariant color map, the latent mapping between the low-light image and the normal exposure image is learned through the optimized reversible network for image enhancement, and the regression loss function is made to converge through end-to-end iterative training, and an optical observation image with an image brightness reaching the threshold is output; In Step 2, the method for performing the preprocessing operation of image restoration on the received optical observation image through the Structure from Motion (SFM) algorithm includes: The estimated blur kernel is simplified through the regression loss, and an adaptive least squares filter is used to remove the noise, blur, and / or artifact features in the optical observation image, and an optical observation image after image restoration is output; After preprocessing operations of image enhancement and image restoration are performed on all optical observation images, preprocessed images are obtained.

4. The method according to claim 3, wherein In step two, the method for optimizing the reversible network using maximum likelihood estimation, using an illumination-invariant color map, and learning the potential mapping between low-light images and normally exposed images through the optimized reversible network for image enhancement includes: where \(L(x\) l , x\) ref ) is the image similarity between the low - light image and the normal - exposure image, \(f\) flow (x\) ref |x\) l ) is the probability density function of the normal - exposure image, \(x\) l is the low - light image, \(x\) ref is the normal - exposure image, \(f\) z (\varTheta(x\) ref ; x\) l )) is the probability density function of the reversible network that learns the mapping between the normal - exposure image and the low - light image based on the latent code \(z\), \(f\) z is the probability density function of the latent code \(z\), \(\varTheta(x\) ref ; x\) l ) is the reversible network that learns the relationship between the normal - exposure image and the low - light image. \(\varTheta\) is a reversible network, which is divided into a sequence of \(N\) reversible layers \(\{\theta\) 1 , \theta\) 2 ,... \theta\) i ..., \theta\) n}\), where \(N = 1\) to \(n\), \(i = 0\) to \(N - 1\); \(z\) n is the \(n\)th latent code, \(g\) n (x\) l ) is the \(n\)th illumination - invariant color map generated by the encoder \(g\) from the low - light image, and its shape is compatible with the \(N\)th reversible layer \(\theta\) n ; The method for making the regression loss function converge through end-to-end iterative training and outputting optical observation images with image brightness reaching the threshold includes: Collect paired samples (x l , x ref ), and train Θ by minimizing the l1 reconstruction regression loss function: The regression loss function is: argminE[l1(Θ(x l ), x ref )] = argmaxE[logf(Θ(x l )|x ref )]; where E is the expected value, l1 is the L1 norm distance, also known as the absolute value distance, which is a method for measuring the distance between two sets of vectors or points; Θ(x l ) is the optical observation image where the image brightness output by Θ reaches the threshold, f(Θ(x l )|x ref ) is the probability density function conditional on x ref and is defined as follows: where b is the learning rate constant; The method for simplifying the estimated blur kernel through regression loss, removing noise, blur, and / or artifact features existing in the optical observation image using an adaptive least squares filter, and outputting the restored optical observation image of the image includes: D a = H a y; Among them, D α is the denoising, deblurring, and / or artifact-removing feature corresponding to x h ↓ s , where x h is the optical observation image after the output image is restored, and ↓ s represents downsampling; H α is a feature-specific operator, a is the number of the optical observation image; y is the degraded image, F(·) represents the Fourier transform, and F -1 (·) represents the inverse Fourier transform, represents the conjugate of F(·), k b is the blurred kernel estimated by simplifying the regression loss, p b represents the filtering parameter; b is the number of the blurred kernel, represents the convolution operation, and k h represents the blurred kernel in the high-resolution space, and μ refers to the noise.

5. The method according to claim 3 or 4, characterized in that In step two, the method for the multi-scale feature extraction network to extract in-feature points of multiple views from the preprocessed image includes: Input the preprocessed image into the multi-scale feature extraction network MsFe; Starting from the resolution of the initially input preprocessed image, perform downsampling multiple times according to preset different scales; after reaching the lowest resolution, start upsampling and combine the features of the previous layer as the input for the next upsampling to obtain high-quality feature maps; remove overlapping corner points in the high-quality feature maps through non-maximum suppression to obtain the in-feature points extracted from multiple views.

6. The method according to claim 5, characterized in that, In step two, the method for performing multi-view feature point matching on the extracted feature points using the ORB algorithm and obtaining sparse point clouds and pose information through bundle adjustment includes: Use the ORB algorithm to calculate the distances between the feature descriptors corresponding to each feature point, and match the corresponding feature points between multiple views by whether the distance range of the feature descriptors is within the threshold to obtain matching feature point pairs; Perform geometric verification on the matching feature point pairs to ensure that the matching points satisfy the observation geometric constraints; Use the EPnP algorithm to estimate the relative pose between the preprocessed images of each pair of matching feature points that satisfy the observation geometric constraints; calculate the mean of the relative poses to obtain the initial camera pose; use the initial camera pose and the matching feature point pairs to calculate the positions of the 3D points through triangulation; Optimize the reprojection error of the positions of the 3D points and the initial camera pose through bundle adjustment, and output sparse point clouds and pose information.

7. The method according to claim 6, wherein In step three, the method for pruning the outlier points of the sparse point cloud by the confidence loss function to obtain the optimized point cloud includes: P = {(p j , f j , γ j ) | j = 1, 2,...,N p}; Among them, P is the optimized point cloud, and p j represents the position of the three-dimensional point of the sparse point cloud, and f j represents the feature vector of the three-dimensional point of the sparse point cloud, and γ j ∈[0,1] represents the confidence value of the three-dimensional point of the sparse point cloud, j is the number of sparse point clouds, and N p is the total number of sparse point clouds; L sparse is the confidence loss value of the confidence loss function, γ is the confidence or probability value, between 0 and 1.

8. The method according to claim 7, wherein In step three, the implicit neural representation function of the neural radiance field is: F β :(γ(x),γ(d))→(c,σ); where F β is an implicit neural representation function of the neural radiance field with respect to the coordinate x and the view direction eigenvector d; γ(x) is the confidence of the spatial position, γ(d) is the confidence along the ray direction, the weight parameter β is optimized to generate the volume density σ and the directional emission color c, γ is a confidence or probability value, between 0 and 1, applied to the predefined positional embedding of the coordinate x and the view direction eigenvector d to map the input to a higher-dimensional space; The method for mapping all inputs to feature vectors after position encoding and association positioning includes: x′ = [x, γ L (x)]; γ L γ(x) = [sin(x), cos(x),..., sin(2 L-1 x), cos(2 L-1 x)]; where \(x'\) is the feature vector, \(x\) is the coordinate, and \(\gamma\) L (x) is the mapping function, and \(L\) is the hyperparameter that controls the maximum coding frequency.

9. The method according to claim 8, wherein In step three, the method for inputting the feature vectors into the fully connected network of the neural radiance field algorithm for training to obtain the trained fully connected network model includes: Select sampling points for each pixel in the feature vector; Use the fully connected network MLP of the neural radiance field algorithm NeRF to generate the density and color of the sampling points, use volume rendering technology to render the color of each sampling point, and integrate along the sampling rays to generate the rendered image; The fully connected network is divided into the first-layer network sampling and the second-layer network sampling. The total number of samples for a single sampled ray is N c +N f , N c is the number of stratified sampling positions in the first-layer network; N f is the number of stratified sampling positions in the second-layer network; the first-layer network stratifies and samples N c positions to obtain the weights of the first-layer network stratified sampling samples; the second-layer network continues to sample N f positions based on the weights of the first-layer network stratified sampling samples to obtain the weights of the second-layer network stratified sampling samples; the L2 norm distance loss value between the rendered image and the real image is optimized by minimizing the L2 norm distance loss function with the weights of the second-layer network stratified sampling samples to obtain a trained fully connected network model; Among them, the calculation method of the weight is as follows: In the formula, is the actual value of the weight, w n' is the theoretical value of the weight, w m' is the initial value of the weight of each sampled ray sample in the network, n' is the number of the weight, and m' is the number of the initial value of the weight; The L2-norm distance loss function is as follows: Among them, L is the L2 norm distance loss value, r represents a single sampled ray sample, represents all sampled rays; C(r) is the real image, is the rendered image output by the first-layer network, is the rendered image output by the second-layer network.

10. The method according to claim 1 or 9, characterized in that, In step three, the volume rendering technology is: Wherein, C(r) is the real image. In three-dimensional space, r(t) represents a point, and d represents the line-of-sight direction feature vector; the near end of the scene is t n , and the far end is t f ; c(r(t), d) is the color value seen when observing the point r(t) from the line-of-sight direction feature vector d; and σ(r(t)) represents the volume density function, which reflects the light absorption ability of the material at the point r(t); T(t) represents the cumulative transmittance from the starting t n to the position t, σ(r(s)) is the change value of the integral path region during the integral calculation of the volume density function, s is the integral upper limit variable, and t is a single integration step size.

Images