Method for three-dimensional reconstruction of satellites from videos taken on the ground with an astronomical telescope

Abstract

The application discloses a method for three-dimensional reconstruction of a satellite by using a video shot on the ground by an astronomical telescope, the method uses a non-professional ground-based telescope to track and shoot the satellite, acquires video data, pre-processes the video data, and manually labels image feature points in the video data; uses a structure-from-motion algorithm to acquire relative poses R and T corresponding to the feature points and images new ; obtains point cloud position information in a three-dimensional space of each image, and uses a 3D-Gaussian algorithm to realize three-dimensional reconstruction of the satellite. By using the application, a three-dimensional model of the satellite can be obtained, and accurate estimation of lengths and relative angles of various cabin sections of the satellite can be achieved.

Description

Technical Field

[0001] This invention belongs to the field of satellite 3D reconstruction, specifically involving a method for 3D reconstruction of satellites using videos captured on the ground by amateur astronomical telescopes. Background Technology

[0002] Currently, satellites can be tracked and photographed using astronomical telescopes, and the data obtained is in the form of images and videos. However, satellite status analysis is limited to two-dimensional images. At present, three-dimensional models can be reconstructed based on image and video information, allowing for more measurements and analyses. There are relatively complete technical solutions for three-dimensional reconstruction using two-dimensional image sequences, but various factors have rendered it completely infeasible in the process of three-dimensional reconstruction of satellites.

[0003] Due to atmospheric disturbances, video images captured by ground-based astronomical telescopes often exhibit visual distortion and blurring. To mitigate this effect, existing technologies employ lucky imagery, which involves rapidly capturing a series of images with short exposure times, then selecting the images least affected by atmospheric disturbances for superposition, thereby improving the resolution of celestial images from ground-based telescopes. This method takes advantage of the relatively short duration of atmospheric disturbances to obtain clearer celestial images. Alternatively, wavelet decomposition enhancement techniques can be used. These techniques decompose the image into multiple scales using wavelet transform, then enhance specific frequency components in the wavelet domain, and finally reconstruct the image using inverse wavelet transform. This process can enhance image edges and details, making the image appear sharper.

[0004] Furthermore, existing technologies use pose estimation techniques to reconstruct the camera's position at the time of image capture based on multiple frames of images of the same object taken from different angles. Then, a dense point cloud or a continuous neural radiation field or other form of 3D scene representation is calculated, enabling the 3D scene to generate a corresponding image from the perspective of the captured photograph. This allows for high-quality rendering results even from non-observational perspectives. Gaussian splashing-based 3D reconstruction technology uses a 3D Gaussian to explicitly represent the scene, enabling rapid scene fitting.

[0005] However, due to the strong atmospheric disturbances, complex background and noise, as well as the ultra-long focal length and limited observation angle, even the most advanced current 3D reconstruction process cannot complete the 3D reconstruction of satellites. Summary of the Invention

[0006] The purpose of this invention is to provide a method for three-dimensional reconstruction of a satellite using satellite data observed by a ground-based amateur telescope, thereby obtaining a three-dimensional model of the satellite and accurate estimations of the length and relative angles of each segment of the satellite.

[0007] The technical solution provided by this invention is as follows:

[0008] A method for 3D reconstruction of satellites using videos captured by astronomical telescopes on the ground includes the following steps:

[0009] 1) Use ground-based, non-specialized telescopes to track and photograph satellites to acquire video data;

[0010] 2) Preprocess the video data obtained in step 1);

[0011] 3) Manually label the image feature points in the video data;

[0012] 4) Use the structure to obtain the point cloud corresponding to the feature points and the relative pose corresponding to the image from the motion algorithm, and further optimize the relative pose corresponding to the image based on the distance from the telescope to the satellite recorded when the video was captured;

[0013] 5) Based on the point cloud location information in three-dimensional space obtained in step 4), perform satellite three-dimensional reconstruction using the 3D-Gaussian algorithm, specifically including the following steps:

[0014] 5-1) A new image is obtained by rasterizing and rendering along the pose of each image using the 3D-Gaussian algorithm.

[0015] 5-2) Apply loss functions to the images obtained in 5-1) and 2) and alternately optimize the 3D Gaussian and pose information. During the optimization process, limit the order of the coefficients of the spherical harmonic function, the growth and splitting speed of the Gaussian to prevent overfitting.

[0016] 5-3) After filtering out noise, the structural information of the satellite based on Gaussian point cloud representation and the three-dimensional information that can be rendered from any viewpoint are finally obtained, thus completing the three-dimensional reconstruction of the satellite.

[0017] Furthermore, step 2) specifically involves:

[0018] 2-1) For video data of extremely poor quality, the image is centered according to the centroid of the image. Then, the lucky imaging technique is used to process the captured video data and the image with weak atmospheric disturbance is selected from all the images for superposition.

[0019] 2-2) Use wavelet decomposition enhancement technology to sharpen the edges and details of images in video data. Perform wavelet decomposition on the image and enhance the high-frequency wavelet coefficients to improve the edge and texture details of the image.

[0020] 2-3) Use a pre-trained neural network model to reduce noise in the video data.

[0021] Furthermore, in step 4), the camera position is adjusted and optimized using the distance from the camera to the space station recorded during shooting, and the estimated camera position T is updated using the following formula. new :

[0022]

[0023] d camera,i,new =scale×d camera,i

[0024] T new =center+d camera,i,new ·r

[0025] Where N represents the number of images used for pose estimation, and d camera,i,new and d telescope,i,new These represent the distance between the camera and the satellite calculated from the motion algorithm based on the structure, and the actual distance between the telescope and the satellite during the image capture, respectively. The recalculation after scaling makes the new camera-to-satellite distance more continuous and smooth. center represents the center of the feature point cloud, and r represents the camera unit orientation vector indicated by the camera rotation matrix R.

[0026] The technical effects of this invention are as follows:

[0027] The present invention is used to construct three-dimensional reconstruction information of a satellite, and to estimate and measure the satellite's attitude and the length of each segment of the satellite. The estimation results obtained have high accuracy. Attached Figure Description

[0028] Figure 1 It is the entire process of processing satellite data.

[0029] Figure 2 This is a schematic diagram illustrating the measurement of satellite length and the relative angles of various parts of the satellite according to a specific embodiment of the present invention. Detailed Implementation

[0030] This invention verified its feasibility through photographing and 3D reconstruction of the Chinese space station, such as... Figure 1 As shown.

[0031] a) Using ground-based, non-specialized telescopes to track and photograph satellites to acquire video data. For example, the video of the Tiangong space station passing over the north bank of Miyun Reservoir in Beijing, taken between 20:48:30 and 20:51:17 on September 15, 2023, is shown. Due to atmospheric turbulence, light pollution, rapid satellite movement, and the telescope's ultra-long focal length, the images obtained exhibit severe distortion, high noise, and significant jitter.

[0032] b) Preprocess the video data as follows:

[0033] 2-1. For data of extremely poor quality, the images are centered based on their centroids. Then, lucky image processing technology is used to process the captured video, selecting images with less atmospheric disturbance (i.e., higher contrast and sharpness) from all images for superposition to improve the signal-to-noise ratio. This method can eliminate image jitter and distortion, but it can lead to image blurring.

[0034] 2-2. Using wavelet decomposition enhancement technology to sharpen the edges and details of an image involves performing wavelet decomposition on the image and increasing the high-frequency wavelet coefficients to improve the edge and texture details of the image and improve the image clarity. However, this will aggravate the noise in the image.

[0035] 2-3. Further denoising is performed on the images using a neural network model pre-trained on a large number of original images and noisy images. This results in high-quality images that are jitter-free, distortion-free, and relatively clear.

[0036] c) Manually label the image feature points.

[0037] Assuming the satellite is stationary, the camera rotates around it to take pictures. To obtain the camera's relative position and attitude, this invention manually labels feature points (such as the apex and edges of the solar array) in the image sequence.

[0038] d) The Structure from Motion (SfM) algorithm is used to obtain the point cloud corresponding to the feature points and the relative pose of the image. The relative pose of the image is further optimized based on the distance between the telescope and the satellite recorded during video capture. This yields the shooting pose of each image relative to the satellite and the point cloud position information in three-dimensional space corresponding to the labeled two-dimensional feature points.

[0039] Because the data captured by the telescope has a super-long focal length compared to ordinary data, and the camera is not sensitive to movement along the camera's orientation, the pose (R-rotation, T-position) estimated by the SfM algorithm for each image has a small error in R, but a large error in T. This invention uses the distance from the camera to the space station recorded during shooting to adjust and optimize the camera's position, updating the estimated camera position T using the following formula. new :

[0040]

[0041] d camera,i,new =scale×d camera,i

[0042] T new =center+d camera,i,new ·r

[0043] Where N represents the number of images used for pose estimation, and d camera,i,new and d telescope,i,new These represent the distance between the camera and the satellite calculated from the motion algorithm based on the structure, and the actual distance from the telescope to the satellite during image capture, respectively. The recalculation after scaling makes the new camera-to-satellite distance more continuous and smooth. `center` represents the center of the feature point cloud, and `r` represents the camera unit orientation vector indicated by the camera rotation matrix `R`. Finally, the new camera pose (R and T) is obtained. new The combination of these methods is used, followed by spherical interpolation and quadratic interpolation to interpolate R and T, respectively, to obtain the poses of images whose poses cannot be obtained through annotation. This yields feature point clouds, relatively clear images, and their relative positions to the point clouds. However, due to image quality and annotation errors, the pose still contains some errors.

[0044] e) Perform satellite 3D reconstruction using the 3D-Gaussian reconstruction algorithm. Specific steps include:

[0045] This invention initializes a 3D Gaussian point for each point cloud, with each Gaussian point containing parameters such as position, shape, and transparency, all of which are differentiable. Then, a loss function is applied to the preprocessed image and the rasterized rendered image along the camera pose of each image, gradually optimizing the Gaussian points to minimize the loss function values ​​of both. Simultaneously, the camera pose R and T are... new The search is performed. The two optimization parts are performed iteratively and alternately, and during this process, the order of the coefficients of the spherical harmonic function, the growth of the Gaussian, and the splitting rate are limited to prevent overfitting.

[0046] f) As training progresses, the point cloud distribution gradually converges to that of a real satellite, yielding structural information of the space station based on Gaussian point cloud representation, as well as 3D information that can be rendered from any viewpoint.

[0047] g) Filtering out discontinuous Gaussian points representing background noise: Post-processing the Gaussian point cloud involves first calculating the center and radius of the initial point cloud to determine a sphere to filter points far from the satellite. Second, outliers are identified based on the average distance of each point to its k nearest neighbors: given the mean (μ) and standard deviation (σ) of these distances across the entire dataset, a threshold τ = μ + α·σ is set, where α is a multiplier (set to 1 in our method). Finally, points with an average distance greater than τ are removed.

[0048] Using the satellite 3D reconstruction information obtained by this invention, the structure on the space station is calculated.

[0049] 1) Calculate the length of each structure: Based on the telescope's focal length, sensor size, image resolution, and the distance between the telescope and the space station recorded during shooting, calculate the actual length corresponding to each pixel in the first frame (or any frame). Perform dense sampling with the center of the space station as the center and the distance between the camera and the center of the space station in the first frame as the radius. Select the frame with the longest structure to be measured from the sampling results (representing that this structure is parallel to the camera plane, and the result is most accurate in this case). Calculate the pixel distance of this structure, and then calculate the actual length based on the pixel distance.

[0050] 2) Calculate the angle of the solar array: Manually segment the point cloud of the solar array and the main structure on the space station, decompose the segmented point cloud using PCA, calculate the orientation, determine a main plane based on the orientation of the main structure, and then calculate the angle between the solar array of each module and the main plane as the angle of the solar array.

[0051] like Figure 2 As shown, the lengths of various modules of the space station (Tianhe core module, Mengtian experimental module, Wentian experimental module, and Tianzhou 6) in a specific embodiment of the present invention were measured, with results of 16.11, 17.30, 17.30, and 9.71 meters, respectively, and actual values ​​of 16.6, 17.88, 17.88, and 10.6 meters. Diameters were also measured, with results of 5.65, 5.50, 5.63, and 3.30 meters, and actual values ​​of 4.2, 4.2, 4.2, and 3.35 meters. The solar array angles of the Mengtian and Wentian experimental modules were also measured to be approximately 63.01° and 68.71°, respectively.

Claims

1. A method for three-dimensional reconstruction of a satellite using videos captured by a ground-based astronomical telescope, specifically including the following steps: 1) Use ground-based, non-specialized telescopes to track and photograph satellites to acquire video data; 2) Preprocess the video data obtained in step 1); 3) Manually label the image feature points in the video data; 4) Use the structure to obtain the point cloud corresponding to the feature points and the relative pose corresponding to the image from the motion algorithm. Based on the distance from the telescope to the satellite recorded when the video was captured, further optimize the relative pose corresponding to the image to obtain the point cloud position information in the satellite's three-dimensional space. 5) Based on the point cloud location information obtained in step 4), perform satellite 3D reconstruction using the 3D-Gaussian algorithm, specifically including the following steps: 5-1) A new image is obtained by rasterizing and rendering along the pose of each image using the 3D Gaussian algorithm. 5-2) Apply loss functions to the images obtained in 5-1) and 2) respectively, and alternately optimize the 3D Gaussian and pose information; 5-3) After filtering out noise, the structural information of the satellite based on point cloud representation and the three-dimensional information that can be rendered from any viewpoint are finally obtained, thus completing the three-dimensional reconstruction of the satellite.

2. The method as described in claim 1, characterized in that, Step 2) specifically involves: 2-1) For video data of extremely poor quality, the image is centered according to the centroid of the image. Then, the lucky imaging technique is used to process the captured video data and the image with weak atmospheric disturbance is selected from all the images for superposition. 2-2) Use wavelet decomposition enhancement technology to sharpen the edges and details of images in video data. Perform wavelet decomposition on the image and enhance the high-frequency wavelet coefficients to improve the edge and texture details of the image. 2-3) Use a pre-trained neural network model to reduce noise in the video data.

3. The method as described in claim 1, characterized in that, Step 4) Use the distance from the camera to the space station recorded during shooting to adjust and optimize the camera position, and update the estimated camera position using the following formula. : Where N represents the number of images used for pose estimation. and These represent the distance between the camera and the satellite calculated from the motion algorithm based on the structure, and the actual distance between the telescope and the satellite during the image capture, respectively. The recalculation after scaling makes the new camera-to-satellite distance more continuous and smooth. center represents the center of the feature point cloud, and r represents the camera unit orientation vector indicated by the camera rotation matrix R.

4. The method as described in claim 3, characterized in that, Use spherical interpolation and quadratic interpolation to apply R and Interpolation is performed to obtain the pose corresponding to the unlabeled image.

5. The method as described in claim 1, characterized in that, In step 5-2), during the optimization process, the coefficient order of the spherical harmonic function, the growth rate of the Gaussian, and the splitting rate are limited to prevent overfitting.

6. The method as described in claim 1, characterized in that, In step 5-3), the Gaussian point cloud is further processed. First, the center and radius of the initial point cloud are calculated to determine a sphere to filter out those points that are far from the satellite. Second, outliers are identified based on the average distance of each point to its k nearest neighbors: given the mean μ and standard deviation σ of these distances over the entire dataset, a threshold τ = μ + α · σ is set, where α is a multiplier, and points with an average distance greater than τ are removed.

Images