Space debris three-dimensional reconstruction method based on grazing observation imaging

Abstract

The invention provides a space debris three-dimensional reconstruction method based on grazing observation imaging, belongs to the field of three-dimensional reconstruction, solves the problem of how to accurately predict the future activity trend of space debris, and comprises the following steps: obtaining an optical observation image after meeting a grazing observation constraint; sFM preprocessing, multi-view internal feature point extraction, multi-view feature point matching and bundle adjustment are carried out on the optical observation image to obtain sparse point cloud and pose information; dividing the preprocessed image into a training set, a test set and a verification set; and inputting the sparse point cloud and the pose information as initial conditions and the training set into an implicit neural expression function of NeRF at the same time, mapping the input into a feature vector after position coding correlation positioning, training the MLP, and outputting and rendering an optimal fitting result to obtain a three-dimensional reconstruction optical observation image. The working state of the space debris is evaluated by analyzing the three-dimensional reconstruction optical observation image, and the future activity trend of the space debris is predicted.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction of space debris, and in particular relates to a three-dimensional reconstruction method of space debris based on flyby observation imaging. Background Art

[0002] Due to the increasing number of satellites in space, carrying out Space Situational Awareness (SSA) missions and monitoring and obtaining attitude information of space debris to avoid collisions between spacecraft have become key links in maintaining space safety. At the same time, various space activities have also put forward higher requirements on technical capabilities in aspects such as spacecraft rendezvous and docking, on-orbit diagnosis and repair of faulty satellites, and space debris cleaning. Therefore, the research on methods for imaging and three-dimensional reconstruction of space debris is imminent, which will help improve the overall autonomy and detection capability of the aerospace system.

[0003] At present, my country's imaging observation methods for space debris mainly include radar detection imaging and optical detection imaging. Radar detection is an active imaging method, but because the energy amplitude of its received signal is inversely proportional to the square of the distance to the space debris, it is often limited by the detection distance, making it difficult to perform imaging detection of space debris in the high orbit belt. Therefore, for high-orbit (GEO) satellites, the current mainstream is still optical imaging detection. As a passive detection method, it is divided into two types of platforms: space-based and ground-based. However, ground-based optical imaging is limited by factors such as atmospheric disturbances, weather conditions, geographical location, and time windows, resulting in unstable image quality and limited resolution, making it difficult to conduct high-quality, high-frequency observations. For example, ground-based optical detection is easily interfered by factors such as atmospheric turbulence and cloud meteorology, and the observation window is limited in duration, resulting in poor imaging quality;

[0004] Compared with ground-based optical observation, space-based optical detection is not restricted by geographical location and meteorological conditions, can provide more stable and clear images, is not affected by the atmosphere, can achieve global coverage, all-weather and high-temporal resolution observation, thus showing significant advantages in the field of space debris monitoring; and currently my country's detection equipment capabilities have already had the function of medium and long-distance high-resolution imaging. In addition, compared with infrared observation, optical observation has the advantages of being passive, low energy consumption, and suitable for long-term work. Therefore, the data of space-based optical detection imaging of space debris is an important source of data in the field of space debris monitoring, such as the three-dimensional structure, attitude changes, and motion state estimation of space debris. Based on time-series space-based visible light images, exploring efficient and accurate three-dimensional reconstruction methods can provide real-time information such as structural dimensions and surface texture details required for on-orbit services, which will help improve the overall autonomy, unmanned operation, and intelligence of the space system.

[0005] According to the different orbital heights and motion characteristics of the observed objects, space-based optical imaging is mainly divided into the following three typical imaging modes: low-orbit rendezvous observation, high-orbit flyby observation and high-orbit flyby observation; among them, the optical observation satellite performs optical imaging of the target satellite at an orbital position that meets the imaging conditions. Among them, the publication number is CN114970180A, and the name of the invention is: In an on-orbit optimization method for spacecraft flyby observation, the constraints of the imaging process are divided into imaging part angle constraints, sunlight constraints, attitude angular velocity constraints, observation distance constraints and safety distance constraints, which has certain inspiration for the design of imaging modes. Li Mengxi. Research on Space Target Attitude Estimation Technology Based on Space-based Optical Imaging [D]. National University of Defense Technology, 2022. In order to estimate the on-orbit three-dimensional attitude of a spacecraft, two space target attitude estimation problems based on space-based optical imaging are mainly considered, namely, the attitude estimation problems in the rendezvous scenario of low-orbit (LEO) space targets and the flyby scenario of geosynchronous orbit (GEO) space targets. The feature association between the target and the known three-dimensional model is constructed to calculate the target attitude, but the three-dimensional structure and surface texture details of the space target are not completely reconstructed. The publication number is CN118604840A, and the name of the invention is: A satellite system and method for collaborative detection and imaging of low-orbit targets from a low-orbit orbit, which provides three working modes of satellites: flyby imaging, flyby imaging and joint detection and positioning mode, but only the imaging mode is discussed, and no further processing of visible light images is involved.

[0006] After obtaining space-based visible light images, image preprocessing, three-dimensional reconstruction and attitude estimation are key technologies for on-orbit services. In recent years, three-dimensional reconstruction technology of space debris based on sequence visible light images has gradually become a research hotspot in the field of space debris identification and monitoring. Related research is of great significance to improving the situational awareness and accurate identification level of space debris. The essence of three-dimensional reconstruction of space debris is to reconstruct the three-dimensional model of space debris through multi-view two-dimensional visible light images, and realize the recognition of surface material, structure and texture features of space debris. Three-dimensional reconstruction was first proposed by Roberts in 1963, and has experienced the development from traditional methods, deep learning methods to end-to-end methods.

[0007] The following process is the main steps of the traditional 3D reconstruction method. Reference: Zhao Shujia. Research on key technologies for 3D reconstruction of low-light images of space targets [D]. Xidian University, 2022. The traditional 3D reconstruction methods are sorted out, mainly including the combination of motion recovery structure (SFM) and multi-view system (MVS). The main expression form of space targets is sparse and dense reconstruction of point clouds. The commonly used process of three-dimensional reconstruction of space targets includes but is not limited to the following steps: (1) Image acquisition: acquiring images of the target through a multi-view camera; (2) Camera calibration: determining the internal and external parameters of the camera; (3) Feature extraction and matching: extracting features from the image and matching them; (4) Camera pose estimation: estimating the relative pose of the camera by matching feature points; (5) Triangulation: calculating the position of feature points in three-dimensional space; (6) Sparse point cloud construction: generating a sparse three-dimensional point cloud; (7) Dense point cloud generation: generating a dense point cloud through multi-view matching; (8) Mesh reconstruction: converting the point cloud into a triangular mesh model; (9) Texture mapping: mapping the texture information of the image to the mesh surface; (10) Model optimization: performing denoising, smoothing and other optimizations on the three-dimensional model; (11) Result evaluation: evaluating the reconstruction quality of the model. Publication number: CN106408650A, invention name: A method for three-dimensional reconstruction and measurement of space targets by on-orbit flyby imaging. The advantage of the disclosed technical solution is that space-based imaging can quickly respond to reconnaissance needs, without strict observation climate conditions and observation windows, but the method adopted is mainly based on traditional methods, which is slightly inferior in reconstruction effect and timeliness.

[0008] Since 2021, the most widely studied method at the International Computer Vision Conference is the Neural Radiance Field (NeRF) method, which has reached a certain maturity. Ma Hansheng, Zhu Yuhua, Li Zhihui, Yan Lei, Si Yiyi, Lian Yimeng, Zhang Yuhan. A Review of Neural Radiance Field Multi-View Synthesis Technology [J]. Computer Engineering and Applications, 2024, 60(4): 21-38, reviewed and summarized the key algorithms in the field of neural radiance fields in recent years. First, the background and principle of the generation of neural radiance fields are introduced, and then the subsequent key improvement models are classified and discussed. Neural Radiance Fields (NeRF) is an end-to-end 3D reconstruction method based on neural networks. It generates high-quality 3D views by learning the radiance field of the scene. The core idea of ​​NeRF is to map each position and direction of the scene into color and volume density. By training images collected from different angles, the model can learn the complete 3D scene structure and generate realistic rendering results from any perspective. Unlike traditional 3D reconstruction methods, NeRF does not rely on explicit geometric data or mesh models, but generates 3D scenes through implicit representation. This method can better handle complex details and lighting changes, making it perform well in areas such as view synthesis, scene reconstruction, and virtual reality.

[0009] However, the above solution has the following technical defects when it is actually applied:

[0010] 1. Observation platform

[0011] Compared with ground-based optical detection, space-based optical detection is not restricted by geographical location and meteorological conditions, and is an important source of data for estimating the three-dimensional structure, attitude changes, and motion state of space debris. Ground-based optical imaging is limited by factors such as atmospheric disturbances, weather conditions, geographical location, and time windows, resulting in unstable image quality and limited resolution, making it difficult to conduct high-quality, high-frequency observations. In contrast, space-based optical imaging can break through these limitations, provide more stable and clear images, is not affected by the atmosphere, and can achieve global coverage, all-weather, and high-temporal resolution observations, thus showing significant advantages in the field of space debris monitoring.

[0012] 2. Space-based optical imaging mode

[0013] Space-based optical imaging modes are mainly divided into three working modes: rendezvous, flyby and flyby observation imaging.

[0014] (1) Rendezvous Observation Imaging

[0015] Rendezvous observation satellites can only observe space debris within a specific time window, and usually require the cooperation of multiple satellites to achieve full coverage, so there is a limited coverage range; the observation time for each transit is short, only a dozen seconds, resulting in a small number of observation images obtained and a small proportion of effective pixels of debris in the observation images, which makes it difficult to meet the image viewing angle conditions required for three-dimensional reconstruction; and in order to maintain continuous observation of debris, orbital maneuvers are required in the form of continuous small thrust energy, which results in a high cost of observation and consumes certain resources.

[0016] (2) Fly-by observation and imaging

[0017] Although flyby observation imaging can continuously track and observe debris in a specific area, the debris is far away from the debris, the image resolution is low, and the proportion of debris in the image is low; it is difficult to quickly adjust the orbit for emergency response observation of sudden events; and the cost of launch and operation and maintenance management is high.

[0018] (3) Flyby Observation Imaging

[0019] The observation time of each flyby observation satellite is relatively long, and it can carry out complex observation tasks, such as multiple imaging and high-precision tracking. It combines the advantages of low-orbit rendezvous and high-orbit flyby imaging modes, can frequently visit the same debris, and provides a high temporal resolution, which is suitable for detecting the dynamic changes of space debris. It is highly flexible and maneuverable, and can quickly adjust the orbit and observation angle to meet the observation needs of different tasks. It can take high-resolution visible light images, and the image quality meets the requirements of three-dimensional reconstruction.

[0020] 3. 3D Reconstruction Method

[0021] The 3D reconstruction based on dense point cloud performs well in terms of accuracy and robustness, and can capture the details of objects. Especially in complex scenes, the algorithm can adaptively handle noise and occlusion to a certain extent, ensuring the accuracy of 3D reconstruction and having good adaptability. In addition, point cloud data does not require predefined grids, so it is suitable for processing objects of complex shapes, and its disorder and sparsity make parallel processing possible, improving computational efficiency. However, the sparsity of point cloud data often leads to reduced reconstruction accuracy, especially in long distances or complex scenes, so it may not work well in scenes with sparse images in the field of non-cooperative targets in space; the disorder of point cloud data makes feature extraction more complicated. Due to the lack of topological information, the texture details of complex geometric shapes may not be effectively displayed during reconstruction. Under complex lighting conditions, the accuracy of 3D reconstruction based on deep point cloud may be greatly restricted; the lack of labeled data is also a great challenge. The amount of point cloud data is huge, and manual labeling is very time-consuming and costly. Especially in high-precision tasks, the lack of labeled data may limit the performance and application effect of the model.

[0022] Multi-view Stereo (MVS) in 3D reconstruction has been widely studied as a core problem in computer vision and photogrammetry. Traditional MVS methods have shown good results in ideal environments (Lambertian surfaces and rich textures). However, in practical applications, due to the existence of variables such as weak textures, non-Lambertian surfaces, and photometric changes, especially in complex spatial environments, reconstruction results are often poor. Therefore, this method still has a lot of room for improvement.

[0023] In summary, there are thousands of abandoned satellites, rocket upper stages and other objects distributed in the geosynchronous orbit. These space debris pose a certain threat to the normal operation of satellites. Therefore, space-based situational awareness is particularly important to ensure the safe and stable operation of future spacecraft. The present invention mainly adopts the space-based optical imaging mode of flyby imaging to observe space debris, and on this basis, studies the three-dimensional reconstruction method based on the neural radiation field algorithm. Summary of the invention

[0024] Among the commonly used means of monitoring space debris, high-resolution visible light images obtained by space-based optical measurements are suitable for attitude estimation and three-dimensional reconstruction, which is of great significance for improving space visual perception capabilities. In addition, optical detection systems have strong passive monitoring capabilities and do not need to actively transmit signals. They perform particularly well in complex environments or those that require covert detection. Therefore, in order to maintain space interests and space security, ensure space activities, and assist space decision-making, it is urgent to obtain the current characteristic information of space debris and predict the future activity trends of space debris. The space characteristic analysis method with optical detection as the main means is one of the key technologies for space situational awareness.

[0025] In order to solve the technical problem of how to obtain the current characteristic information of space debris and predict the future activity trend of space debris, the present invention provides a three-dimensional reconstruction method of space debris based on flyby observation imaging. The method takes the neural radiation field algorithm as the basic principle and takes space-based optical flyby imaging as the working mode of typical scenes. It provides an end-to-end typical scene space-based optical imaging method, as well as a three-dimensional reconstruction process and method based on the combination of motion recovery structure (SFM) and improved neural radiation field algorithm, which provides an important reference for on-orbit service. The space-based platform can perform high-definition imaging of specific viewing angles of debris at close range by on-orbit flyby. It can evaluate the working status of the debris and predict the future activity trend of space targets by constructing fine component-level features of the space debris. The technical solution disclosed by the present invention maintains space interests and space security, ensures space activities, and can assist space decision-making.

[0026] To achieve the above object, the present invention provides a method for three-dimensional reconstruction of space debris based on flyby observation imaging, comprising the following steps:

[0027] Step 1: The flyby observation constraint is formed by the observation distance constraint, the observation geometry constraint, the illumination condition constraint and the detection limit constraint. After the flyby observation imaging mode of the space-based platform satisfies the flyby observation constraint, an optical observation image of the space debris is obtained, and the optical observation image is transmitted to the ground station;

[0028] Step 2: Perform image enhancement and image restoration preprocessing operations on the received optical observation image by using the motion recovery structure algorithm to obtain a preprocessed image; extract feature points in multiple views of the preprocessed image by a multi-scale feature extraction network; use the ORB algorithm to match the extracted feature points between multiple views, and then use the bundle method to obtain a sparse point cloud and posture information;

[0029] Step three, divide the preprocessed image into training set, test set and validation set according to a preset ratio; prune the external points of the sparse point cloud by the confidence loss function to obtain the optimized point cloud; input the optimized point cloud and posture information as initial conditions and the training set into the implicit neural expression function of the neural radiation field used to represent the flyby observation scene, map all inputs into feature vectors after position encoding association positioning, and input the feature vectors into the fully connected network of the neural radiation field algorithm for training to obtain a trained fully connected network model, test and verify the trained fully connected network model through the test set and validation set, and output the optimal fit result; use volume rendering technology to render the optimal fit result in three-dimensional space, and output a three-dimensional reconstructed optical observation image of the space fragment.

[0030] In step 1, the observation distance constraint includes: observation position constraint, optical axis angle constraint and communication safety distance constraint; wherein the observation position constraint is: Where, d min is the minimum effective observation distance, ξ is the spatial transformable scale coefficient, R target is the observation range radius of space debris, A FOV is the field of view of the optical camera;

[0031] The optical axis angle constraint is: Where β is the angle between the optical axis and the optical camera, β ob is the half angle of the field of view of the observation platform, and β is less than β ob ; r CM is the optical axis line, i.e. the observation line of sight, r TC is the distance vector from the space debris to the optical camera;

[0032] The communication safety distance constraint is: the distance between each observation platform is less than the effective communication safety distance threshold;

[0033] In step 1, the observation geometry constraint includes the observation platform constraint and the angle constraint between the observation platform and the space debris; wherein the observation platform constraint is:

[0034] The angle constraint between the observation platform and the space debris is:

[0035] In the formula, r C is the geocentric vector of the observation platform equipped with the optical camera, r TC is the distance vector from the space debris to the optical camera, R E is the radius of the Earth, θ is the angle between the Earth's center, the optical camera, and the space debris, and the angle range is 0 to 180°;

[0036] In step 1, the lighting condition constraints include: avoiding direct sunlight constraints, avoiding space debris in the earth's shadow area constraints and observation attitude constraints; wherein,

[0037] Avoid direct sunlight constraints: in the front light condition, the illumination angle meets the threshold, the sun does not directly hit the lens of the observation platform during observation, and satisfies 0<α<β<π or Where α is the sum of the apparent radius of the sun and the light scattering angle and is a non-negative acute angle, r CS is the distance vector from the observation platform to the sun, r TC is the distance vector from the space debris to the optical camera, β is the angle between the optical axis and the optical camera, that is, r CS With r TC The angle of

[0038] The constraints to prevent space debris from being in the Earth's shadow area are: or Where λ is the geocentric vector r of the space debris T and the sun's geocentric vector r S The angle, R E is the radius of the Earth;

[0039] The observation attitude constraint is: the angle between the optical axis of the optical camera carried by each observation platform and the ideal observation line of sight is less than the angle threshold, ensuring that the space debris activity area is within the field of view of the optical camera;

[0040] In step 1, the detection limit constraint is:

[0041]

[0042] Where M is the magnitude value corresponding to space debris, R S,N,0 To achieve the signal-to-noise ratio of the system at the detection limit of the system, is the average photon flux of sporadic magnitude, A is the aperture area of ​​the receiver, η is the quantum efficiency of the detector for the space debris spectrum, te is the integration time, τ is the transmittance of the optical system to the space debris signal, N D is the number of dark signal electrons, N R is the number of read noise electrons, N s is the number of signal electrons output by the optical camera during the optical camera detection time, i.e., the integration time, is the signal photon flux.

[0043] In step 2, the method of performing a preprocessing operation of image enhancement on the received optical observation image by using a structure-from-motion algorithm includes:

[0044] The optical observation image whose image brightness does not reach the threshold is regarded as a low-light image;

[0045] The low-light image is fed into the encoder of the Structure from Motion (SFM) algorithm to generate an illumination-invariant color map.

[0046] A reversible network that maps normal exposure images to latent codes;

[0047] The reversible network is optimized by maximum likelihood estimation. The illumination-invariant color map is used to learn the potential mapping between low-light images and normal-exposure images through the optimized reversible network for image enhancement. The regression loss function is converged through end-to-end iterative training, and the optical observation image with image brightness reaching the threshold is output.

[0048] In step 2, the method of performing a preprocessing operation of image restoration on the received optical observation image by using a structure-from-motion algorithm includes:

[0049] The estimated blur kernel is simplified by regression loss, and an adaptive least squares filter is used to remove noise, blur and / or artifact features in the optical observation image, and the restored optical observation image is output;

[0050] After all the optical observation images are subjected to the preprocessing operations of image enhancement and image restoration, the preprocessed images are obtained.

[0051] In step 2, the reversible network is optimized by using maximum likelihood estimation, and the illumination-invariant color map is used to learn the potential mapping between the low-light image and the normal exposure image through the optimized reversible network to perform image enhancement. The method includes:

[0052]

[0053] Among them, L(x l ,x ref ) is the image similarity between the low-light image and the normal exposure image, f flow (x ref |x l) is the probability density function of the normal exposure image, x l For low light images, x ref is a normal exposure image, Θ is a reversible network, which is divided into N reversible layer sequences {θ 1 ,θ 2 ,...θ i ...,θ n}, N = 1 to n, i = 0 to N-1; h i+1 =θ i (h i ; g i (x l )) is the i-th reversible layer θ i The output, h i is the output of the i-1th reversible layer, g i (x l ) is the i-th illumination-invariant color map generated by encoder g; h 0 =x ref And z = h N ;h N is the output of the N-1th reversible layer, z n is the nth potential code, g n (x l ) is the nth illumination-invariant color map generated by encoder g, whose shape is similar to that of the Nth reversible layer θ n Compatible; f z is the probability density function of the potential code z;

[0054] The method of achieving convergence of the regression loss function through end-to-end iterative training and outputting an optical observation image whose image brightness reaches a threshold includes:

[0055] Collect paired samples (x l ,x ref ), and train Θ by minimizing the l1 reconstruction regression loss function:

[0056] The regression loss function is: argminE[l1(Θ(x l ),x ref )] = argmaxE[logf(Θ(x l )|x ref )];

[0057] Where E is the expected value, l1 is the L1 norm distance, also known as the absolute value distance, which is a method of measuring the distance between two sets of vectors or points; Θ(x l ) is the optical observation image whose brightness output by Θ reaches the threshold value, f(Θ(x l )|x ref ) is x ref is the conditional probability density function, which is defined as follows:

[0058]

[0059] Where b is the learning rate constant;

[0060] The method of simplifying the estimated blur kernel by regression loss, removing noise, blur and / or artifact features in the optical observation image by using an adaptive least squares filter, and outputting the restored optical observation image includes:

[0061] D a =H a y;

[0062]

[0063] Among them, D α is the value corresponding to x h ↓ s De-noising, de-blurring and / or de-artifacting features, x h is the optical observation image after the output image is restored,↓ s represents downsampling; H α is a feature-specific operator, a is the number of the optical observation image; y is the degraded image, F(·) represents the Fourier transform, and F -1 (·) represents the inverse Fourier transform, F(·) represents the conjugate of F(·), k b is the blur kernel estimated by simplifying the regression loss, p b represents the filtering parameter; b is the number of the blur kernel, represents the convolution operation, k h represents the blur kernel in high-resolution space, and μ refers to the noise.

[0064] In step 2, the method of extracting feature points in multiple views of the preprocessed image by a multi-scale feature extraction network includes:

[0065] Input the preprocessed image into the multi-scale feature extraction network MsFe;

[0066] Starting from the initial input preprocessed image resolution, downsampling is performed multiple times according to different preset scales; upsampling begins after reaching the lowest resolution and the features of the previous layer are combined as the input for the next upsampling to obtain a high-quality feature map; non-maximum suppression (NMS) is used to remove overlapping corner points in the high-quality feature map to obtain feature points extracted in multiple views.

[0067] In step 2, the method of using the ORB algorithm to match the extracted feature points between multiple views and then using the bundle adjustment to obtain the sparse point cloud and pose information includes:

[0068] The ORB algorithm is used to calculate the distance between the feature descriptors corresponding to each feature point, and the corresponding feature points are matched between multiple views according to whether the distance range of the feature descriptors is within the threshold to obtain matching feature point pairs;

[0069] Perform geometric verification on the matching feature point pairs to ensure that the matching points meet the observed geometric constraints;

[0070] Use the EPnP algorithm to estimate the relative pose between each pair of preprocessed images that meet the observation geometry constraints; calculate the average value of the relative pose to obtain the initial camera pose; use the initial camera pose and the matching feature point pairs to calculate the position of the 3D point through triangulation method;

[0071] The position of the 3D points and the reprojection error of the initial camera posture are optimized through bundle adjustment (BA), and sparse point cloud and posture information are output.

[0072] In step 3, the method of pruning the external points of the sparse point cloud by the confidence loss function to obtain the optimized point cloud includes:

[0073] P={(p j ,f j ,γ j )|j=1,2,...,N p};

[0074]

[0075] Among them, P is the optimized point cloud, p j Represents the position of the 3D point of the sparse point cloud, f j The feature vector representing the 3D point of the sparse point cloud, γ j ∈[0,1] represents the confidence value of the 3D point of the sparse point cloud, j is the number of sparse point clouds, N p is the total number of sparse point clouds; L sparse is the confidence loss value of the confidence loss function, and γ is the confidence or probability value, which is between 0 and 1.

[0076] In step 3, the implicit neural expression function of the neural radiation field is:

[0077] F β :(γ(x),γ(d))→(c,σ);

[0078] Among them, F βis the implicit neural expression function of the neural radiation field with respect to the coordinate x and the sight direction feature vector d; γ(x) is the confidence of the spatial position, γ(d) is the confidence along the light direction, and the weight parameter β is optimized to generate the volume density σ and the directional emission color c. γ is the confidence or probability value between 0 and 1, which is applied to the predefined position embedding of the coordinate x and the sight direction feature vector d to map the input to a higher dimensional space;

[0079] Methods for mapping all inputs into feature vectors after positional encoding and association positioning include:

[0080] x′=[x,γ L (x)];

[0081] γ L (x)=[sin(x),cos(x),...,sin(2 L-1 x),cos(2 L-1 x)];

[0082] In the formula, x′ is the eigenvector, x is the coordinate, γ L (x) is the mapping function and L is a hyperparameter that controls the maximum encoding frequency.

[0083] In step 3, the method of inputting the feature vector into the fully connected network of the neural radiation field algorithm for training to obtain a trained fully connected network model includes:

[0084] A sampling point is selected for each pixel in the feature vector;

[0085] Use the fully connected network MLP of the Neural Radiation Field algorithm NeRF to generate the density and color of the sampling points, use volume rendering technology to render the color of each sampling point and integrate along the sampling light to generate the rendered image;

[0086] The fully connected network is divided into the first layer network sampling and the second layer network sampling. The total number of samples of a single sampling light is N c +N f , N c N is the number of sampling positions for the first layer of the network; f is the number of stratified sampling positions for the second layer of the network; N is the number of stratified sampling positions for the first layer of the network c The weights of the first-layer network stratified sampling samples are obtained; the second-layer network continues to sample N according to the weights of the first-layer network stratified sampling samples. f The weights of the second-layer network layered sampling samples are obtained; the weights of the second-layer network layered sampling samples are used to optimize the scene representation by minimizing the L2 norm distance loss value between the rendered image and the real image through the L2 norm distance loss function, and a trained fully connected network model is obtained;

[0087] The weight calculation method is: In the formula, is the actual value of the weight, w n is the theoretical value of weight, w m The initial value of the weight of each sampled light sample in the network;

[0088] The L2 norm distance loss function is:

[0089] Among them, L is the L2 norm distance loss value, r represents a single sampling light sample, represents all sampling rays; C(r) is the real image, To render the image.

[0090] In step 3, the volume rendering technique is:

[0091]

[0092] In the formula, C(r) is the real image. In three-dimensional space, r(t) represents a point, and d represents the line of sight direction feature vector. The near end of the scene is t n , the far end is t f ; c(r(t),d) is the color value observed at point r(t) from the line of sight eigenvector d; σ(r(t)) represents the volume density function, which reflects the light absorption capacity of the material at point r(t); T(t) represents the color value observed at point r(t) from the line of sight eigenvector d; n The cumulative transmittance to position t, σ(r(s)) is the change in the integral path area during the integral calculation of the volume density function, s is the integral upper limit variable, and t is a single integration step.

[0093] The present invention has the following beneficial effects:

[0094] 1. The present invention designs a flyby observation imaging mode of a space-based platform for a typical scenario and analyzes in detail the flyby observation constraints that are optically visible to the observation satellite. Limited by the real space environment and combined with the working characteristics of different orbits, the flyby observation imaging mode of the space-based platform can have the advantages of high revisit coverage, low energy consumption rate and long-term operation, and can focus on high-resolution imaging and three-dimensional reconstruction of space debris and faulty satellites.

[0095] 2. The present invention proposes a motion recovery structure algorithm SFM to perform image enhancement and image restoration preprocessing operations on the received optical observation image to obtain a preprocessed image, and generates sparse point cloud and pose information by iteratively minimizing the reprojection error. The learning model-based method has obvious advantages over the traditional iterative traversal method in that the algorithm has strong adaptability and generalization. The point cloud coordinates and pose matrices obtained on unseen data sets have high accuracy, which solves the problem of point cloud loss caused by occlusion and measurement errors in complex scenes.

[0096] 3. The present invention takes into account the problem of space waste. In blank areas, due to data uncertainty, reconstruction defects such as floating objects may occur. The sparse point cloud obtained by the motion recovery structure algorithm SFM is used as the optimization input condition of the NeRF method to obtain the initial pose information. On this basis, combined with the neural radiation field method, all scene attribute information is stored at the point through the neural point cloud. Through position encoding and 3D segmentation, the feature query method of any point in space is continuously generated, and the new perspective image is rendered, which greatly improves the rendering speed of three-dimensional reconstruction of space targets and saves the resource consumption of data transmission and ground station data processing.

[0097] 4. The combination of neural radiation field and point cloud greatly improves the rendering speed of 3D reconstruction, which has great practical significance for limited satellite resources and transmission rates, and lays the foundation for realizing lightweight automatic processing on board. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 It is a schematic flow chart of a method for three-dimensional reconstruction of space debris based on flyby observation imaging provided by the present invention.

[0099] Figure 2 It is a schematic diagram of an example of a fully connected network architecture using position encoding for a neural radiation field provided by the present invention. DETAILED DESCRIPTION

[0100] like Figure 1 As shown, in an embodiment of the present invention, a method for three-dimensional reconstruction of space debris based on flyby observation imaging is provided, comprising the following steps:

[0101] Step 1: The flyby observation constraint is formed by the observation distance constraint, the observation geometry constraint, the illumination condition constraint and the detection limit constraint. After the flyby observation imaging mode of the space-based platform satisfies the flyby observation constraint, an optical observation image of the space debris is obtained, and the optical observation image is transmitted to the ground station;

[0102] Step 2: Perform image enhancement and image restoration preprocessing operations on the received optical observation image by using the motion recovery structure algorithm SFM to obtain a preprocessed image; extract feature points in multiple views of the preprocessed image by using the multi-scale feature extraction network MsFe; use the ORB algorithm to match the extracted feature points between multiple views, and then use the bundle method to obtain sparse point cloud and posture information;

[0103] Step 3: Divide the preprocessed image into training set, test set and validation set according to the preset ratio (such as 7:2:1); prune the external points of the sparse point cloud by the confidence loss function to obtain the optimized point cloud; input the optimized point cloud and pose information as the initial conditions and the training set into the implicit neural expression function of the neural radiation field NeRF used to represent the flyby observation scene, so that the output results are compatible and the purpose of efficient rendering is achieved; after the position encoding association positioning, all inputs are mapped into feature vectors, and the feature vectors are input into the fully connected network MLP of the neural radiation field algorithm NeRF for training to obtain a trained fully connected network model, and the trained fully connected network model is tested and verified by the test set and validation set, and the optimal fitting result is output; the optimal fitting result is rendered in three-dimensional space using volume rendering technology, and the three-dimensional reconstructed optical observation image of the space debris is output. Finally, the three-dimensional structure, surface texture, material details and other information of the space debris can be inverted.

[0104] In step 1, the observation distance constraint includes: observation position constraint, optical axis angle constraint and communication safety distance constraint; wherein the observation position constraint is: Where, d min is the minimum effective observation distance, ensuring that the optical camera field of view should cover the space debris activity area as much as possible; ξ is the spatial transformable scale coefficient, R target is the observation range radius of space debris, A FOV is the field of view of the optical camera;

[0105] The optical axis angle constraint is: Where β is the angle between the optical axis and the optical camera, β ob is the half angle of the field of view of the observation platform, and β is less than β ob ; r CM is the optical axis line, i.e. the observation line of sight, r TC is the distance vector from the space debris to the optical camera;

[0106] The communication safety distance constraint is: the distance between each observation platform is less than the effective communication safety distance threshold;

[0107] In step 1, the observation geometry constraint includes the observation platform constraint and the angle constraint between the observation platform and the space debris, so as to ensure that the space-based observation platform and the space debris can be visible to each other; wherein the observation platform constraint is:

[0108] The angle constraint between the observation platform and the space debris is:

[0109] In the formula, r C is the geocentric vector of the observation platform equipped with the optical camera, r TC is the distance vector from the space debris to the optical camera, R E is the radius of the Earth, θ is the angle between the Earth's center, the optical camera, and the space debris, and the angle range is 0 to 180°;

[0110] In step 1, space debris can only be optically visible when the lighting conditions are met in space. This usually means that when a satellite equipped with an optical camera observes space debris, it is necessary to avoid the background of direct sunlight, and space debris cannot appear in the shadow area of ​​the earth. In addition, the activity area of ​​the space debris must be within the field of view of the optical camera as much as possible, so as to ensure that the sunlight reflected by the space debris can be captured by the observation station detection system. The influence of the moon is not considered here. The lighting condition constraints include: avoiding direct sunlight constraints, avoiding space debris in the shadow area constraints, and observation attitude constraints; among them,

[0111] Avoid direct sunlight constraints: in the front light condition, the illumination angle meets the threshold, the sun does not directly hit the lens of the observation platform during observation, avoid the observation platform appearing in the sun's apparent disk and its vicinity, and satisfy 0<α<β<π or Where α is the sum of the apparent radius of the sun and the light scattering angle and is a non-negative acute angle, r CS is the distance vector from the observation platform to the sun, r TC is the distance vector from the space debris to the optical camera, β is the angle between the optical axis and the optical camera, that is, r CS With r TC The angle of

[0112] If the space debris is in the shadow area, the space debris cannot reflect sunlight, and the observation platform cannot form an image. Therefore, when observing and imaging space debris, it is necessary to avoid the space debris being in the shadow area. The constraint to avoid the space debris being in the shadow area is: or Where λ is the geocentric vector r of the space debris T and the sun's geocentric vector r S The angle, R E is the radius of the Earth;

[0113] The observation attitude constraint is: the angle between the optical axis of the optical camera carried by each observation platform and the ideal observation line of sight is less than the angle threshold, ensuring that the space debris activity area is within the field of view of the optical camera;

[0114] Space-based detection systems have certain detection limits. From an energy perspective, if the intensity of the visible light signal of space debris meets the minimum detection threshold of the detector, the detection system can detect space debris. Regardless of whether the observation platform carries a CCD camera or a CMOS camera, the principle of imaging space debris is to receive sunlight reflected by space debris and convert the light signal into an electrical signal through structures such as photosensitive elements, thereby imaging the space debris. In step one, the detection limit constraint is:

[0115]

[0116] Where M is the magnitude value corresponding to space debris, R S,N,0 To achieve the signal-to-noise ratio of the system at the detection limit of the system, is the average photon flux of sporadic magnitude, A is the aperture area of ​​the receiver, η is the quantum efficiency of the detector for the space debris spectrum, t e is the integration time, τ is the transmittance of the optical system to the space debris signal, N D is the number of dark signal electrons, N R is the number of read noise electrons, N s is the number of signal electrons output by the optical camera during the optical camera detection time, i.e., the integration time, is the signal photon flux.

[0117] Due to the particularity of space environment illumination, the optical observation images of space debris obtained under weak light conditions are often noisy and degraded to varying degrees. The image degradation process can be expressed as a combination of blur, downsampling and noise, so it needs to be preprocessed first. In step 2, the method of preprocessing the received optical observation image for image enhancement by using the motion restoration structure algorithm includes:

[0118] The optical observation image whose image brightness does not reach the threshold is regarded as a low-light image;

[0119] The low-light image is fed into the encoder of the Structure from Motion (SFM) algorithm to generate an illumination-invariant color map.

[0120] A reversible network that maps normal exposure images to latent codes;

[0121] The reversible network is optimized by maximum likelihood estimation. The illumination-invariant color map is used to learn the potential mapping between low-light images and normal-exposure images through the optimized reversible network for image enhancement. The regression loss function is converged through end-to-end iterative training, and the optical observation image with image brightness reaching the threshold is output.

[0122] In step 2, the method of performing a preprocessing operation of image restoration on the received optical observation image by using a structure-from-motion algorithm includes:

[0123] The estimated blur kernel is simplified by regression loss, and an adaptive least squares filter is used to remove noise, blur and / or artifact features in the optical observation image, and the restored optical observation image is output;

[0124] In the image restoration process, blur kernel refers to the convolution kernel that causes the image to blur. The present invention regards the regression loss of blur kernel estimation as a sub-problem of the regression loss describing the difference between images. By describing the difference between images through regression loss, the estimation process of blur kernel can be simplified. The regression loss of blur kernel estimation can evaluate and optimize the estimation of blur kernel parameters and the quality of restored images, making them closer to the real image.

[0125] After all the optical observation images are subjected to the preprocessing operations of image enhancement and image restoration, the preprocessed images are obtained.

[0126] In order to better characterize the high-quality normal exposure image, in step 2, the reversible network is optimized by using the maximum likelihood estimation, and the illumination-invariant color map is used to learn the potential mapping between the low-light image and the normal exposure image through the optimized reversible network to perform image enhancement. The method includes:

[0127]

[0128] Among them, L(x l ,x ref ) is the image similarity between the low-light image and the normal exposure image, f flow (x ref |x l ) is the probability density function of the normal exposure image, x l For low light images, x ref is a normal exposure image, Θ is a reversible network, which is divided into N reversible layer sequences {θ 1 ,θ 2 ,...θ i ...,θ n}, N = 1 to n, i = 0 to N-1; h i+1 =θ i (h i ; gi (x l )) is the i-th reversible layer θ i The output, h i is the output of the i-1th reversible layer, g i (x l ) is the i-th illumination-invariant color map generated by encoder g; h 0 =x ref And z = h N ;h N is the output of the N-1th reversible layer, z n is the nth potential code, g n (x l ) is the nth illumination-invariant color map generated by encoder g, whose shape is similar to that of the Nth reversible layer θ n Compatible; f z is the probability density function of the potential code z;

[0129] The method of achieving convergence of the regression loss function through end-to-end iterative training and outputting an optical observation image whose image brightness reaches a threshold includes:

[0130] Collect paired samples (x l ,x ref ), and train Θ by minimizing the l1 reconstruction regression loss function:

[0131] The regression loss function is: argminE[l1(Θ(x l ),x ref )] = argmaxE[logf(Θ(x l )|x ref )]; The regression loss function is used to minimize the similarity between the low-light image and the normal exposure image, so that the objective function is minimized and the training Θ is obtained, which aims to minimize the expected value between the model's predicted value and the reference value under the L1 norm distance.

[0132] Where E is the expected value and l1 is the L1 norm distance, also known as the absolute value distance, which is a method of measuring the distance between two sets of vectors or points. l ) is the optical observation image whose brightness output by Θ reaches the threshold value, f(Θ(x l )|x ref ) is x ref is the conditional probability density function, which is defined as follows:

[0133]

[0134] Where b is the learning rate constant;

[0135] The method of simplifying the estimated blur kernel by regression loss, removing noise, blur and / or artifact features in the optical observation image by using an adaptive least squares filter, and outputting the restored optical observation image includes:

[0136] D a =H a y;

[0137]

[0138] Among them, D α is the value corresponding to x h ↓ s De-noising, de-blurring and / or de-artifacting features, x h is the optical observation image after the output image is restored,↓ s represents downsampling; H α is a feature-specific operator, a is the number of the optical observation image; y is the degraded image, F(·) represents the Fourier transform, and F -1 (·) represents the inverse Fourier transform, represents the conjugate of F(·), k b is the blur kernel estimated by simplifying the regression loss, p b represents the filtering parameter; b is the number of the blur kernel, represents the convolution operation, k h represents the blur kernel in high-resolution space, and μ refers to the noise.

[0139] In step 2, the method of extracting feature points in multiple views of the preprocessed image by a multi-scale feature extractor (MsFe) network includes:

[0140] Input the preprocessed image into the multi-scale feature extraction network MsFe;

[0141] Starting from the initial input preprocessed image resolution, downsampling is performed multiple times according to preset different scales; upsampling is started after reaching the lowest resolution and the features of the previous layer are combined as the input of the next upsampling to obtain a high-quality feature map; non-maximum suppression (NMS) is used to remove overlapping corner points in the high-quality feature map to ensure a more uniform distribution of corner points and obtain feature points extracted in multiple views. By detecting corner points on images of different scales, the scale invariance of the detection is improved.

[0142] In step 2, the ORB (Oriented FAST and Rotated BRIEF) algorithm is used to match the extracted feature points between multiple views. The ORB algorithm is an efficient feature detection and description algorithm, which is widely used in real-time computer vision tasks. It combines the advantages of multi-scale feature extraction networks and BRIEF descriptors, and introduces some improvements to improve the robustness and accuracy of features. The methods for obtaining sparse point clouds and pose information through bundle adjustment (BA), that is, iterative minimization of reprojection errors, include:

[0143] The ORB algorithm is used to calculate the distance between the feature descriptors corresponding to each feature point, and the corresponding feature points are matched between multiple views according to whether the distance range of the feature descriptors is within the threshold to obtain matching feature point pairs;

[0144] Perform geometric verification on the matching feature point pairs to ensure that the matching points meet the observed geometric constraints;

[0145] Preferably, the matching feature point pairs are geometrically verified by a basic matrix or an essential matrix; wherein the basic matrix is ​​used to process non-calibrated images, that is, when the camera intrinsic parameters are unknown. The basic matrix describes the geometric relationship between the matching feature point pairs in the two images, and is particularly suitable for images taken from two different perspectives. It is a 3x3 non-singular matrix with a rank of 2.

[0146] The essential matrix is ​​similar to the basic matrix. It is based on the assumption that the two cameras have been calibrated, that is, the internal parameters of the two cameras are known, such as focal length, principal point position, etc. The essential matrix describes the rotation and translation relationship between the two cameras, and is the relationship matrix after converting the matching feature point pairs in the two images from the pixel coordinate system to the normalized coordinate system. The essential matrix is ​​also a 3x3 matrix, and its form can be composed of a rotation matrix and a translation vector.

[0147] Use the EPnP algorithm to estimate the relative pose between each pair of preprocessed images that meet the observation geometry constraints; calculate the average value of the relative pose to obtain the initial camera pose; use the initial camera pose and the matching feature point pairs to calculate the position of the 3D point through triangulation method;

[0148] The position of the 3D points and the reprojection error of the initial camera pose are optimized through bundle adjustment (BA), the initial camera pose and the position of the 3D points are refined, and sparse point cloud and pose information are output.

[0149] In step 3, the method of pruning the external points of the sparse point cloud by the confidence loss function to obtain the optimized point cloud includes:

[0150] P={(p j ,fj ,γ j )|j=1,2,...,N p};

[0151]

[0152] Among them, P is the optimized point cloud, p j Represents the position of the 3D point of the sparse point cloud, f j The feature vector representing the 3D point of the sparse point cloud, γ j ∈[0,1] represents the confidence value of the 3D point of the sparse point cloud, j is the number of sparse point clouds, N p is the total number of sparse point clouds; L sparse is the confidence loss value of the confidence loss function, and γ is the confidence or probability value, which is between 0 and 1.

[0153] In step 3, the implicit neural expression function of the neural radiation field is:

[0154] F β :(γ(x),γ(d))→(c,σ); represents a continuous scene as a 5D vector-valued function whose input is a 3D position and 2D viewing direction, and whose output is color and volume density.

[0155] Among them, F β is the implicit neural expression function of the neural radiation field with respect to the coordinate x and the sight direction feature vector d; γ(x) is the confidence of the spatial position, γ(d) is the confidence along the light direction, and the weight parameter β is optimized to generate the volume density σ and the directional emission color c. γ is the confidence or probability value between 0 and 1, which is applied to the predefined position embedding of the coordinate x and the sight direction feature vector d to map the input to a higher dimensional space;

[0156] Directly optimizing NeRF on the original input (x, d) usually leads to difficulties in synthesizing high-frequency details. In order to solve this problem, sine functions of different frequencies are used to map the input to a higher-dimensional space to enhance the details and textures after rendering and achieve better 3D reconstruction accuracy. Therefore, the present invention uses all inputs to be mapped into feature vectors after position coding association positioning to solve this technical problem. The method includes:

[0157] x′=[x,γ L (x)];

[0158] γ L (x)=[sin(x),cos(x),...,sin(2 L-1 x),cos(2 L-1 x)];

[0159] In the formula, x′ is the eigenvector, x is the coordinate, γ L (x) is the mapping function and L is a hyperparameter that controls the maximum encoding frequency.

[0160] In step 3, the method of inputting the feature vector into the fully connected network MLP of the neural radiation field algorithm NeRF for training to obtain a trained fully connected network model includes:

[0161] A sampling point is selected for each pixel in the feature vector;

[0162] Use the fully connected network MLP of the Neural Radiation Field algorithm NeRF to generate the density and color of the sampling points, use volume rendering technology to render the color of each sampling point and integrate along the sampling light to generate the rendered image;

[0163] The fully connected network is divided into the first layer network sampling and the second layer network sampling. The total number of samples of a single sampling light is N c +N f , N c N is the number of sampling positions for the first layer of the network; f is the number of stratified sampling positions for the second layer of the network; N is the number of stratified sampling positions for the first layer of the network c The weights of the first-layer network stratified sampling samples are obtained; the second-layer network continues to sample N according to the weights of the first-layer network stratified sampling samples. f The weights of the second-layer network layered sampling samples are obtained; the weights of the second-layer network layered sampling samples are used to optimize the scene representation by minimizing the L2 norm distance loss value between the rendered image and the real image through the L2 norm distance loss function, and a trained fully connected network model is obtained;

[0164] The weight calculation method is: In the formula, is the actual value of the weight, w n is the theoretical value of weight, w m The initial value of the weight of each sampled light sample in the network;

[0165] The L2 norm distance loss function is:

[0166] Among them, L is the L2 norm distance loss value, r represents a single sampling light sample, represents all sampling rays; C(r) is the real image, For the rendered image. L2 norm distance: The square of the Euclidean distance, which is the square root of the sum of the squares of the differences between the two vector components.

[0167] Using volume rendering techniques to render each color of a pixel, for a given ray origin and viewing direction along the pixel, the i-th sample on the ray is defined as where t i is a bounded partition from N evenly spaced nodes from near to far [t n , t f ] is extracted by layered sampling. In step 3, the volume rendering technology is:

[0168]

[0169] In the formula, C(r) is the real image. In three-dimensional space, r(t) represents a point, and d represents the line of sight direction feature vector. The near end of the scene is t n , the far end is t f ; c(r(t),d) is the color value observed at point r(t) from the line of sight eigenvector d; σ(r(t)) represents the volume density function, which reflects the light absorption capacity of the material at point r(t); T(t) represents the color value observed at point r(t) from the line of sight eigenvector d; n The cumulative transmittance to position t, σ(r(s)) is the change in the integral path area during the integral calculation of the volume density function, s is the integral upper limit variable, and t is a single integration step.

[0170] After volume rendering in three-dimensional space, a new-perspective visible light two-dimensional image with high resolution is finally generated and output, that is, the three-dimensional reconstructed optical observation image of space debris.

[0171] In order to more clearly illustrate the technical solution of the present invention, specific examples are now provided to illustrate the technical solution of the present invention in detail:

[0172] Example 1: Flyby imaging mode analysis

[0173] Take the typical scene of GEO orbital direct light observation imaging as an example:

[0174] 1) Visual window analysis

[0175] GEO satellites with an inclination angle other than 0° pass through the equatorial plane twice a day, from 6:00 to 18:00 local time and from 18:00 to 6:00 the next day.

[0176] 2) Observation geometry constraints

[0177] Observations are conducted when space debris passes near the equator, taking into account the position of the sun. When the space debris passes the equator and the sun is located above and behind, the satellite needs to maneuver to the upper and rear of the space debris for imaging. When the space debris passes the equator and the sun is located above and in front, the satellite needs to maneuver to the upper and front of the space debris for imaging.

[0178] 3) Lighting condition constraints

[0179] If the patrol orbit is located below the space debris, the time period from 18:00 to 6:00 the next day should be selected for close observation, and the space debris should be imaged from below or below the side, which meets the lighting conditions of the light angle. According to the simulation results, when the local time is 23:24-00:36, the sun is below the space debris, and the satellite should image the space debris from below; but as the satellite revolves, the space debris is in the shadow of the earth at the vernal or autumnal equinox every year, and the visible light camera cannot image it at this time. This observation period should be excluded, and multi-satellite collaboration should be adopted to conduct flyby imaging observations from other perspectives.

[0180] Example 2: Fully connected network MLP and position encoding

[0181] Although neural networks can be used as universal function approximators, they perform poorly in rendering high-frequency changes in color and geometry when vector coordinates are directly input into the network. Deep networks tend to learn low-frequency functions, and by mapping the input to a high-frequency function in a high-dimensional space and then passing it to the network, they can better fit data containing high-frequency changes. Figure 2 As shown, this is a schematic diagram of a fully connected network architecture.

[0182] 1) Fully connected network architecture

[0183] It mainly includes input vectors, intermediate hidden layers, and output vectors. All layers are standard fully connected layers. Layers with ReLU activation are represented by black arrows, layers without activation are represented by orange arrows, layers with Sigmoid activation are represented by dotted black arrows, and "+" represents vector concatenation.

[0184] 2) Position encoding process

[0185] ① Follow the structure of DeepSDF and add a skip connection to connect the input to the activation of the fifth layer;

[0186] ② The output volume density σ is corrected by ReLU to ensure a non-negative value, and a 256-dimensional feature vector is output;

[0187] ③ This feature vector is concatenated with the position encoding of the input view direction, i.e., the confidence γ(d) along the light direction, and processed by an additional fully connected ReLU layer with 128 channels;

[0188] ④The final layer, i.e. the Sigmoid activation layer, outputs the RGB radiation observed along the line of sight feature vector d at the position coordinate x.

[0189] 3) Position encoding method

[0190] Neural radiation fields encode position as follows:

[0191] γL (πx)=(sin(2 0 πx),cos(2 0 πx),...,sin(2 L-1 πx),cos(2 L-1 πx))

[0192] 4) Position encoding results

[0193] γ(·) is applied to each dimension of the vector to be encoded independently. When L is used to encode the position vector x, it is 10, and when L is used to encode the view vector d, it is 4. After position encoding, the encoding dimension of x is 60, and the encoding dimension of d is 24.

[0194] The embodiments of the present invention have the following beneficial effects:

[0195] 1. The present invention designs a flyby observation imaging mode of a space-based platform for a typical scenario and analyzes in detail the flyby observation constraints that are optically visible to the observation satellite. Limited by the real space environment and combined with the working characteristics of different orbits, the flyby observation imaging mode of the space-based platform can have the advantages of high revisit coverage, low energy consumption rate and long-term operation, and can focus on high-resolution imaging and three-dimensional reconstruction of space debris and faulty satellites.

[0196] 2. The present invention proposes a motion recovery structure algorithm SFM to perform image enhancement and image restoration preprocessing operations on the received optical observation image to obtain a preprocessed image, and generates sparse point cloud and pose information by iteratively minimizing the reprojection error. The learning model-based method has obvious advantages over the traditional iterative traversal method in that the algorithm has strong adaptability and generalization. The point cloud coordinates and pose matrices obtained on unseen data sets have high accuracy, which solves the problem of point cloud loss caused by occlusion and measurement errors in complex scenes.

[0197] 3. The present invention takes into account the problem of space waste. In blank areas, due to data uncertainty, reconstruction defects such as floating objects may occur. The sparse point cloud obtained by the motion recovery structure algorithm SFM is used as the optimization input condition of the NeRF method to obtain the initial pose information. On this basis, combined with the neural radiation field method, all scene attribute information is stored at the point through the neural point cloud. Through position encoding and 3D segmentation, the feature query method of any point in space is continuously generated, and the new perspective image is rendered, which greatly improves the rendering speed of three-dimensional reconstruction of space targets and saves the resource consumption of data transmission and ground station data processing.

[0198] 4. The combination of neural radiation field and point cloud greatly improves the rendering speed of 3D reconstruction, which has great practical significance for limited satellite resources and transmission rates, and lays the foundation for realizing lightweight automatic processing on board.

[0199] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for three-dimensional reconstruction of space debris based on flyby observation imaging, characterized in that: The following steps are involved: Step 1: The flyby observation constraint is formed by the observation distance constraint, the observation geometry constraint, the illumination condition constraint and the detection limit constraint. After the flyby observation imaging mode of the space-based platform satisfies the flyby observation constraint, an optical observation image of the space debris is obtained, and the optical observation image is transmitted to the ground station; Step 2: Perform image enhancement and image restoration preprocessing operations on the received optical observation image by using the motion recovery structure algorithm to obtain a preprocessed image; extract feature points in multiple views of the preprocessed image by a multi-scale feature extraction network; use the ORB algorithm to match the extracted feature points between multiple views, and then use the bundle method to obtain a sparse point cloud and posture information; Step three, divide the preprocessed image into training set, test set and validation set according to a preset ratio; prune the external points of the sparse point cloud by the confidence loss function to obtain the optimized point cloud; input the optimized point cloud and posture information as initial conditions and the training set into the implicit neural expression function of the neural radiation field used to represent the flyby observation scene, map all inputs into feature vectors after position encoding association positioning, and input the feature vectors into the fully connected network of the neural radiation field algorithm for training to obtain a trained fully connected network model, test and verify the trained fully connected network model through the test set and validation set, and output the optimal fit result; use volume rendering technology to render the optimal fit result in three-dimensional space, and output a three-dimensional reconstructed optical observation image of the space fragment.

2. The method according to claim 1, characterized in that In step 1, the observation distance constraint includes: observation position constraint, optical axis angle constraint and communication safety distance constraint; wherein the observation position constraint is: Where, d min is the minimum effective observation distance, ξ is the spatial transformable scale coefficient, R target is the observation range radius of space debris, A FOV is the field of view of the optical camera; The optical axis angle constraint is: Where β is the angle between the optical axis and the optical camera, β ob is the half angle of the field of view of the observation platform, and β is less than β ob ; r CM is the optical axis line, i.e. the observation line of sight, r TC is the distance vector from the space debris to the optical camera; The communication safety distance constraint is: the distance between each observation platform is less than the effective communication safety distance threshold; In step 1, the observation geometry constraint includes the observation platform constraint and the angle constraint between the observation platform and the space debris; wherein the observation platform constraint is: The angle constraint between the observation platform and the space debris is: In the formula, r C is the geocentric vector of the observation platform equipped with the optical camera, r TC is the distance vector from the space debris to the optical camera, R E is the radius of the Earth, θ is the angle between the Earth's center, the optical camera, and the space debris, and the angle range is 0 to 180°; In step 1, the lighting condition constraints include: avoiding direct sunlight constraints, avoiding space debris in the earth's shadow area constraints and observation attitude constraints; wherein, Avoid direct sunlight constraints: in the front light condition, the illumination angle meets the threshold, the sun does not directly hit the lens of the observation platform during observation, and satisfies 0<α<β<π or Where α is the sum of the apparent radius of the sun and the light scattering angle and is a non-negative acute angle, r CS is the distance vector from the observation platform to the sun, r TC is the distance vector from the space debris to the optical camera, β is the angle between the optical axis and the optical camera, that is, r CS With r TC The angle of The constraints to prevent space debris from being in the Earth's shadow area are: or Where λ is the geocentric vector r of the space debris T and the sun's geocentric vector r S The angle, R E is the radius of the Earth; The observation attitude constraint is: the angle between the optical axis of the optical camera carried by each observation platform and the ideal observation line of sight is less than the angle threshold, ensuring that the space debris activity area is within the field of view of the optical camera; In step 1, the detection limit constraint is: Where M is the magnitude value corresponding to space debris, R S,N,0 To achieve the signal-to-noise ratio of the system at the detection limit of the system, is the average photon flux of sporadic magnitude, A is the aperture area of ​​the receiver, η is the quantum efficiency of the detector for the space debris spectrum, t e is the integration time, τ is the transmittance of the optical system to the space debris signal, N D is the number of dark signal electrons, N R is the number of read noise electrons, N s is the number of signal electrons output by the optical camera during the optical camera detection time, i.e., the integration time, is the signal photon flux.

3. The method according to claim 2, characterized in that In step 2, the method of performing a preprocessing operation of image enhancement on the received optical observation image by using a structure-from-motion algorithm includes: The optical observation image whose image brightness does not reach the threshold is regarded as a low-light image; The low-light image is fed into the encoder of the Structure from Motion (SFM) algorithm to generate an illumination-invariant color map. A reversible network that maps normal exposure images to latent codes; The reversible network is optimized by maximum likelihood estimation. The illumination-invariant color map is used to learn the potential mapping between low-light images and normal-exposure images through the optimized reversible network for image enhancement. The regression loss function is converged through end-to-end iterative training, and the optical observation image with image brightness reaching the threshold is output. In step 2, the method of performing a preprocessing operation of image restoration on the received optical observation image by using a structure-from-motion algorithm includes: The estimated blur kernel is simplified by regression loss, and an adaptive least squares filter is used to remove noise, blur and / or artifact features in the optical observation image, and the restored optical observation image is output; After all the optical observation images are subjected to the preprocessing operations of image enhancement and image restoration, the preprocessed images are obtained.

4. The method according to claim 3, characterized in that In step 2, the reversible network is optimized by using maximum likelihood estimation, and the illumination-invariant color map is used to learn the potential mapping between the low-light image and the normal exposure image through the optimized reversible network to perform image enhancement. The method includes: Among them, L(x l ,x ref ) is the image similarity between the low-light image and the normal exposure image, f flow (x ref |x l ) is the probability density function of the normal exposure image, x l For low light images, x ref is a normal exposure image, Θ is a reversible network, which is divided into N reversible layer sequences {θ 1 ,θ 2 ,...θ i ...,θ n }, N = 1 to n, i = 0 to N-1; h i+1 =θ i (h i ; g i (x l )) is the i-th reversible layer θ i The output, h i is the output of the i-1th reversible layer, g i (x l ) is the i-th illumination-invariant color map generated by encoder g; h 0 =x ref And z = h N ;h N is the output of the N-1th reversible layer, z n is the nth potential code, g n (x l ) is the nth illumination-invariant color map generated by encoder g, whose shape is similar to that of the Nth reversible layer θ n Compatible; f z is the probability density function of the potential code z; The method of achieving convergence of the regression loss function through end-to-end iterative training and outputting an optical observation image whose image brightness reaches a threshold includes: Collect paired samples (x l ,x ref ), and train Θ by minimizing the l1 reconstruction regression loss function: The regression loss function is: argminE[l1(Θ(x l ),x ref )] = argmaxE[logf(Θ(x l )|x ref )]; Where E is the expected value, l1 is the L1 norm distance, also known as the absolute value distance, which is a method of measuring the distance between two sets of vectors or points; Θ(x l ) is the optical observation image whose brightness output by Θ reaches the threshold value, f(Θ(x l )|x ref ) is x ref is the conditional probability density function, which is defined as follows: Where b is the learning rate constant; The method of simplifying the estimated blur kernel by regression loss, removing noise, blur and / or artifact features in the optical observation image by using an adaptive least squares filter, and outputting the restored optical observation image includes: D a =H a y; Among them, D α is the value corresponding to x h ↓ s De-noising, de-blurring and / or de-artifacting features, x h is the optical observation image after the output image is restored,↓ s represents downsampling; H α is a feature-specific operator, a is the number of the optical observation image; y is the degraded image, F(·) represents the Fourier transform, and F -1 (·) represents the inverse Fourier transform, represents the conjugate of F(·), k b is the blur kernel estimated by simplifying the regression loss, p b represents the filtering parameter; b is the number of the blur kernel, represents the convolution operation, k h represents the blur kernel in high-resolution space, and μ refers to the noise.

5. The method according to claim 3 or 4, characterized in that In step 2, the method of extracting feature points in multiple views of the preprocessed image by a multi-scale feature extraction network includes: Input the preprocessed image into the multi-scale feature extraction network MsFe; Starting from the initial input preprocessed image resolution, downsampling is performed multiple times according to different preset scales; upsampling begins after reaching the lowest resolution and the features of the previous layer are combined as the input for the next upsampling to obtain a high-quality feature map; non-maximum suppression is used to remove overlapping corner points in the high-quality feature map to obtain feature points extracted in multiple views.

6. The method according to claim 5, characterized in that In step 2, the method of using the ORB algorithm to match the extracted feature points between multiple views and then using the bundle adjustment to obtain the sparse point cloud and pose information includes: The ORB algorithm is used to calculate the distance between the feature descriptors corresponding to each feature point, and the corresponding feature points are matched between multiple views according to whether the distance range of the feature descriptors is within the threshold to obtain matching feature point pairs; Perform geometric verification on the matching feature point pairs to ensure that the matching points meet the observed geometric constraints; Use the EPnP algorithm to estimate the relative pose between each pair of preprocessed images that meet the observation geometry constraints; calculate the average value of the relative pose to obtain the initial camera pose; use the initial camera pose and the matching feature point pairs to calculate the position of the 3D point through triangulation method; The position of the 3D points and the reprojection error of the initial camera posture are optimized through bundle adjustment, and sparse point cloud and posture information are output.

7. The method according to claim 6, characterized in that In step 3, the method of pruning the external points of the sparse point cloud by the confidence loss function to obtain the optimized point cloud includes: P={(p j ,f j ,c j )|j=1,2,...,N p }; Among them, P is the optimized point cloud, p j Represents the position of the 3D point of the sparse point cloud, f j The feature vector representing the 3D point of the sparse point cloud, γ j ∈[0,1] represents the confidence value of the 3D point of the sparse point cloud, j is the number of sparse point clouds, N p is the total number of sparse point clouds; L sparse is the confidence loss value of the confidence loss function, and γ is the confidence or probability value, which is between 0 and 1.

8. The method according to claim 7, characterized in that In step 3, the implicit neural expression function of the neural radiation field is: F β :(γ(x),γ(d))→(c,σ); Among them, F β is the implicit neural expression function of the neural radiation field with respect to the coordinate x and the sight direction feature vector d; γ(x) is the confidence of the spatial position, γ(d) is the confidence along the light direction, and the weight parameter β is optimized to generate the volume density σ and the directional emission color c. γ is the confidence or probability value between 0 and 1, which is applied to the predefined position embedding of the coordinate x and the sight direction feature vector d to map the input to a higher dimensional space; Methods for mapping all inputs into feature vectors after positional encoding and association positioning include: x′=[x,γ L (x)]; γ L (x)=[sin(x),cos(x),...,sin(2 L-1 x),cos(2 L-1 x)]; In the formula, x′ is the eigenvector, x is the coordinate, γ L (x) is the mapping function and L is a hyperparameter that controls the maximum encoding frequency.

9. The method according to claim 8, characterized in that In step 3, the method of inputting the feature vector into the fully connected network of the neural radiation field algorithm for training to obtain a trained fully connected network model includes: A sampling point is selected for each pixel in the feature vector; Use the fully connected network MLP of the Neural Radiation Field algorithm NeRF to generate the density and color of the sampling points, use volume rendering technology to render the color of each sampling point and integrate along the sampling light to generate the rendered image; The fully connected network is divided into the first layer network sampling and the second layer network sampling. The total number of samples of a single sampling light is N c +N f , N c N is the number of sampling positions for the first layer of the network; f is the number of stratified sampling positions for the second layer of the network; N is the number of stratified sampling positions for the first layer of the network c The weights of the first-layer network stratified sampling samples are obtained; the second-layer network continues to sample N according to the weights of the first-layer network stratified sampling samples. f The weights of the second-layer network layered sampling samples are obtained; the weights of the second-layer network layered sampling samples are used to optimize the scene representation by minimizing the L2 norm distance loss value between the rendered image and the real image through the L2 norm distance loss function, and a trained fully connected network model is obtained; The weight calculation method is: In the formula, is the actual value of the weight, w n is the theoretical value of weight, w m The initial value of the weight of each sampled light sample in the network; The L2 norm distance loss function is: Among them, L is the L2 norm distance loss value, r represents a single sampling light sample, represents all sampling rays; C(r) is the real image, To render the image.

10. The method according to claim 1 or 9, characterized in that: In step 3, the volume rendering technique is: In the formula, C(r) is the real image. In three-dimensional space, r(t) represents a point, and d represents the line of sight direction feature vector. The near end of the scene is t n , the far end is t f ; c(r(t),d) is the color value observed at point r(t) from the line of sight eigenvector d; σ(r(t)) represents the volume density function, which reflects the light absorption capacity of the material at point r(t); T(t) represents the color value observed at point r(t) from the line of sight eigenvector d; n The cumulative transmittance to position t, σ(r(s)) is the change in the integral path area during the integral calculation of the volume density function, s is the integral upper limit variable, and t is a single integration step.

Images