Space target point cloud three-dimensional reconstruction and rendering method

Abstract

The invention discloses a space target point cloud three-dimensional reconstruction and rendering method, belongs to the field of space target scene reconstruction and rendering, solves the problem of low accuracy of space target scene reconstruction and rendering, and comprises the following steps: generating an initial sparse point cloud and a camera pose of a space target by using an SFM; performing denoising and outlier processing on the initial sparse point cloud by using a bilateral filtering algorithm to obtain an initial point cloud; distributing an initial gray Gaussian volume for each point in the initial point cloud, and projecting the initial gray Gaussian volume to a two-dimensional plane by using a projection formula in combination with a camera pose; rendering the two-dimensional projection image by using a micro-rasterization rendering technology; fusing the luminosity loss function, the structural similarity loss function and the depth loss function to construct a total loss function, and optimizing an initial gray Gaussian body attribute in the rendered image by using a stochastic gradient descent method; a self-adaptive density control mechanism is introduced to adjust the density of the initial point cloud, and a space target point cloud three-dimensional reconstruction structure is generated; according to the invention, the reconstruction and rendering accuracy of the space target scene is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of space target scene reconstruction and rendering, and relates to a method for three-dimensional reconstruction and rendering of space target point cloud. Background Art

[0002] 3D Gaussian Splatting (3D GS) is an emerging and efficient scene reconstruction and rendering technology that uses Gaussian distribution to represent point cloud data in the scene and achieves high-quality 3D reconstruction by optimizing these Gaussian distributions. This method was published by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis in ACM Transactions on Graphics (TOG) Volume 42, Issue 4, Pages 1-14. The core of the 3D GS method is to convert the image in the scene into a sparse point cloud, and then represent the point cloud as a Gaussian volume. Each Gaussian volume is described by parameters such as its position mean, covariance matrix, color (represented by spherical harmonics) and transparency (alpha value). During the rendering process, these Gaussian volumes are projected onto the two-dimensional image plane to form a projection similar to a "snowball" shape, with energy spreading outward and weakening from the center. The methods involve technologies such as extracting sparse point clouds from images, Gaussian volume initialization, Gaussian volume optimization, and real-time rendering.

[0003] In the field of satellite observation and 3D reconstruction, traditional 3D GS technical methods often face multiple challenges:

[0004] 1. Multi-perspective image fusion problem: In satellite observation, due to changes in observation angle, distance and lighting conditions, the multiple space-based optical images obtained may have problems such as perspective differences, information overlap or missing.

[0005] 2. Point cloud data quality and noise processing: The initial point cloud data often contains a lot of noise and outliers, which seriously affect the accuracy and effect of 3D reconstruction.

[0006] 3. Optimize the matching degree between the rendered image and the real image: There may be differences between the rendered image and the real image, which will affect the accuracy of the 3D reconstruction results.

[0007] 4. Since space targets have the characteristics of limited data collection methods, complex lighting changes, and single color, and the computing performance of the space observation platform is limited, it is necessary to reduce complex calculations and memory usage, and have high requirements for rendering speed and reconstruction speed. The 3D GS method designed algorithms and rendering processes for ground targets cannot meet the requirements of three-dimensional reconstruction and real-time rendering of space targets. Summary of the invention

[0008] In order to solve the problems that 1. the 3D GS technology in the prior art uses low image efficiency data registration rate and multi-view image fusion is difficult; 2. in the process of generating the initial point cloud, due to the influence of equipment noise and the space environment, the generated point cloud has problems such as noise and outliers; 3. the amount of calculation is large and the computing efficiency is low; 4. the accuracy of the 3D reconstruction result is low, the present invention provides a method for 3D reconstruction and rendering of space target point cloud, which applies 3D GS technology to the field of space targets to achieve good 3D reconstruction and real-time rendering effects; bilateral filtering denoising is performed on the initial sparse point cloud to ensure the accuracy of the reconstruction and rendering results; the color expression in the Gaussian body is represented by grayscale values ​​to simplify color calculation and improve the running speed; the total loss function integrates the photometric loss function, the structural similarity loss function and the depth loss function, which can better measure the consistency of the depth information between the rendered image and the real image, optimize the overall rendering effect, and adapt to complex scenes. The method can achieve 3D reconstruction and rendering of space targets in special space environments.

[0009] The purpose of the present invention is specifically achieved through the following technical solutions:

[0010] The present invention discloses a method for three-dimensional reconstruction and rendering of a space target point cloud, the method comprising:

[0011] Step 1: Receive the space-based optical image obtained by the optical observation equipment to observe the space target, use the motion recovery structure algorithm SFM to generate the initial sparse point cloud of the space target, and obtain the camera pose corresponding to the space-based optical image;

[0012] Step 2: Use bilateral filtering algorithm to denoise and process outliers on the initial sparse point cloud to obtain the initial point cloud;

[0013] Step 3: assign an initial grayscale Gaussian volume to each point in the initial point cloud, wherein the attributes of the initial grayscale Gaussian volume include spatial position, covariance matrix, transparency and grayscale value;

[0014] Step 4: Convert the initial grayscale Gaussian body assigned to each point from the world coordinate system to the image coordinate system. Combined with the camera posture, use the projection formula to project each initial grayscale Gaussian body converted to the image coordinate system from a three-dimensional plane to a two-dimensional plane to obtain a two-dimensional projection image.

[0015] Step 5: Use differentiable rasterization rendering technology to render the initial grayscale Gaussian volume in each two-dimensional projection image to obtain a rendered image;

[0016] Step 6: The total loss function is constructed by integrating the photometric loss function, the structural similarity loss function and the depth loss function. The total loss value between the rendered image and the real image is calculated by the total loss function. Based on the total loss value, the properties of the initial grayscale Gaussian in the rendered image are optimized using the stochastic gradient descent method.

[0017] Step 7: Introduce an adaptive density control mechanism to automatically adjust the density of the initial point cloud according to the properties of the optimized initial grayscale Gaussian body and the spatial distribution of the spatial target, and generate a three-dimensional reconstruction structure of the spatial target point cloud.

[0018] In step 1, the steps of generating an initial sparse point cloud of a space target using the structure-from-motion algorithm SFM and obtaining the camera pose corresponding to the space-based optical image include:

[0019] The SIFT descriptor in the Structure from Motion (SFM) algorithm is used to describe the feature points in each space-based optical image.

[0020] Calculate the Euclidean distance between SIFT descriptors in any two space-based optical images, and construct feature point pairs from the feature points of the two space-based optical images with the closest Euclidean distance;

[0021] Acquire the camera pose corresponding to the space-based optical image through the feature point pair, wherein the camera pose includes a rotation matrix and a translation vector;

[0022] Based on the triangulation algorithm, the 3D point positions of the initial sparse point cloud in each space-based optical image are recovered from the feature point pairs to obtain the initial sparse point cloud of the space target.

[0023] In step 2, the bilateral filtering algorithm includes:

[0024]

[0025] Among them, I filtered (p) is the filtered image output by the bilateral filtering algorithm, and p is each point in the initial sparse point cloud; W p is a normalization constant; q is the neighborhood point of p, N(p) represents the neighborhood space of point p, I(q) and I(p) represent the attribute values ​​of q and p respectively, G s (·) is the spatial domain weight function, which is used to measure the Euclidean distance between point clouds. r (·) is the grayscale domain weight function. The more similar the grayscale attributes are, the greater the grayscale domain weight is.

[0026] In step 3, the properties of the initial grayscale Gaussian body include the spatial position, covariance matrix, transparency and grayscale value respectively:

[0027] The spatial position of the initial grayscale Gaussian body is the position coordinate of each point in the initial point cloud;

[0028] The covariance matrix of the initial grayscale Gaussian is:

[0029] Σ=R2SS T R2 T ;

[0030] Among them, Σ is the output value of the covariance matrix, R2 is the rotation matrix, S is the scaling matrix, and T represents the transpose of the matrix;

[0031] The transparency value of the initial grayscale Gaussian is in the range of [0-1], where 0 represents complete transparency and 1 represents complete opacity.

[0032] The grayscale value range of the initial grayscale Gaussian body is [0, 255]. In step 4, the method of converting the initial grayscale Gaussian body assigned to each point from the world coordinate system to the image coordinate system includes:

[0033] First, transform each initial grayscale Gaussian from the world coordinate system XYZ to the camera coordinate system xyz:

[0034]

[0035] Where R is the rotation function between the world coordinate system XYZ and the camera coordinate system xyz, and S represents the translation vector between the origin of the camera coordinate system and the origin of the world coordinate system;

[0036] Then transform the camera coordinate system xyz to the image plane coordinate system x'y':

[0037]

[0038] Where f represents the focal length of the camera;

[0039] Finally, the image plane coordinate system x'y' is transformed into the image coordinate system MN;

[0040]

[0041] Where M and N are the total number of rows and columns of pixels in the computer memory, respectively. m and O n are the row and column numbers of the central pixel in the computer memory; S x is the distance between the centers of two adjacent sensors along the x direction, S y is the distance between the centers of two adjacent sensors along the y direction, L x is the number of sensor elements in the x direction, M x is the number of pixels in a row, and μ is the uncertainty image scaling factor of the camera.

[0042] In step 4, the projection formula is:

[0043] P=K·X c ;

[0044] Among them, P is the two-dimensional coordinate of the two-dimensional projection image, K is the camera intrinsic parameter matrix in the camera pose, X c is the point in each initial grayscale Gaussian volume transformed into the image coordinate system.

[0045] In step 5, the method of rendering the initial grayscale Gaussian volume in each two-dimensional projection image using a differentiable rasterization rendering technology to obtain a rendered image includes:

[0046] The initial grayscale Gaussian body in the two-dimensional projection image is an ellipse of two-dimensional Gaussian distribution. Based on the grayscale value in the attributes of the initial grayscale Gaussian body, the distance between all target pixels to be found and the center of the ellipse is calculated to obtain an opacity set; all initial grayscale Gaussian bodies are superimposed through the opacity set to obtain a rendered image.

[0047] In step 6, the total loss function constructed is:

[0048] L=λ1L1+λ2L ssim +(1-λ1-λ2)L depth ;

[0049] Among them, L is the total loss function value, λ1 is the weight factor of the photometric loss function, λ2 is the weight factor of the structural similarity loss function, L1 is the photometric loss function of the rendered image and the real image, L ssim is the structural similarity loss function, L depth is the depth loss function.

[0050] In step 6, the calculation method of the photometric loss function is:

[0051]

[0052] Among them, L1 is the light loss value, N' is the number of pixels in the image, i is the number of pixels in the image, and I render (n i ) is the rendered image at n i The photometric value at real (n i ) is the real image in n i The photometric value at i is the pixel position in the image;

[0053] The calculation method of the structural similarity loss function is:

[0054] L ssim =1-SSIM(I render ,I real );

[0055] Among them, L ssim is the structural similarity loss value, SSIM(I render ,I real ) is the structural similarity index between the rendered image and the real image;

[0056]

[0057] In the formula, is the average pixel value of the rendered image, is the pixel average value of the real image, C1 is the pixel average value constant, which is used to avoid the situation where the denominator is 0. is the covariance of the rendered image and the real image, C2 is the standard deviation constant, which is used to avoid the situation where the denominator is 0. is the standard deviation of the rendered image, is the standard deviation of the real image;

[0058] The calculation method of the depth loss function is:

[0059]

[0060] Among them, L depth is the depth loss value, D render (n i ) is the rendering image in n i The depth value at real (n i ) The real image is in n i The depth value at .

[0061] In step 6, based on the total loss value, the update formula for optimizing the properties of the initial grayscale Gaussian in the rendered image using the stochastic gradient descent method is:

[0062]

[0063] Among them, P' is the optimized spatial position, P is the spatial position in the attributes of the initial grayscale Gaussian, α' is the optimized transparency, α is the transparency in the attributes of the initial grayscale Gaussian; Σ' is the optimized covariance, Σ is the covariance in the attributes of the initial grayscale Gaussian; G' is the optimized grayscale value, G is the grayscale value in the attributes of the initial grayscale Gaussian, η is the learning rate; ▽ P L, ▽ α L, ▽ Σ L, ▽ G L are the gradients of the total loss function value with respect to the spatial position, transparency, covariance and grayscale value in the attributes of the initial grayscale Gaussian.

[0064] In step seven, an adaptive density control mechanism is introduced to automatically adjust the density of the initial point cloud according to the properties of the optimized initial grayscale Gaussian volume and the spatial distribution of the spatial target. The method for generating a three-dimensional reconstruction structure of the spatial target point cloud includes:

[0065] Set the gradient threshold;

[0066] If the pixel gradient change of the current volume attribute of the optimized initial grayscale Gaussian in the rendered image exceeds the threshold, it is judged that the number of initial grayscale Gaussian volumes is insufficient and Gaussian volume replication is required;

[0067] If the pixel gradient change of the current attribute of the optimized initial grayscale Gaussian body in the rendered image is lower than the threshold, and the gradient change of the surrounding adjacent pixels is also lower than the threshold, it is determined that the number of initial grayscale Gaussian bodies is too large and the current initial grayscale Gaussian body needs to be replaced.

[0068] The beneficial effects of the present invention are:

[0069] 1. Aiming at the problem of multi-view image fusion: In satellite observation, due to changes in observation angle, distance and lighting conditions, the multiple space-based optical images obtained may have problems such as perspective differences, information overlap or missing. The present invention uses the SFM (Structure from Motion) algorithm to effectively integrate these multi-view image information and generate an initial sparse point cloud, which provides a basis for subsequent 3D reconstruction, applies 3D GS technology to the field of space targets, and achieves good 3D reconstruction and rendering effects;

[0070] 2. Regarding the point cloud data quality and noise processing issues: The initial point cloud data often contains a lot of noise and outliers, which will seriously affect the accuracy and effect of 3D reconstruction. The present invention performs bilateral filtering denoising on the initial sparse point cloud, denoises and processes outliers on the initial sparse point cloud, and obtains cleaner and more accurate point cloud data, thereby ensuring the accuracy of reconstruction and rendering results, and solving the problems of the prior art using images with low efficiency data registration rate, resulting in sparse initial point clouds, and the generation of initial point clouds, due to the influence of equipment noise and space environment, the generated point clouds have noise, outliers and other problems;

[0071] 3. The color expression in the Gaussian body is expressed by grayscale values, which reduces the complexity of the expression by spherical harmonic functions, simplifies color calculations, and improves the running speed;

[0072] 4. Optimizing the matching degree between rendered images and real images: There may be differences between rendered images and real images, which will affect the accuracy of the 3D reconstruction results. The total loss function constructed by the present invention integrates the photometric loss function, the structural similarity loss function and the depth loss function, which can better measure the consistency of the depth information between the rendered image and the real image, optimize the overall rendering effect, and adapt to complex scenes. This method can realize the 3D reconstruction and rendering of space targets in special space environments.

[0073] 5. Enhance the realism of rendered images: By using differentiable rasterization rendering technology and stochastic gradient descent method to optimize the properties of the initial grayscale Gaussian volume in the rendered image, a more realistic rendering result that is close to the real image can be generated, providing a more accurate data basis for subsequent image analysis and application.

[0074] 6. Improving the accuracy and efficiency of three-dimensional reconstruction: The technical solution disclosed in the present invention significantly improves the accuracy and efficiency of three-dimensional reconstruction, and provides more reliable technical support for the field of satellite observation and three-dimensional reconstruction.

[0075] 7. Improve the breadth and flexibility of technology applications: The present invention is not only applicable to the fields of satellite observation and three-dimensional reconstruction, but can also be extended to other occasions that require three-dimensional reconstruction and real-time rendering, such as aerial photogrammetry and remote sensing image processing, showing broad application prospects and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The present invention is further described in detail below based on the accompanying drawings and embodiments.

[0077] Figure 1 It is a real image among the space-based optical images obtained by the optical observation equipment observing the space target.

[0078] Figure 2 It is the rendered image obtained after optimizing the initial grayscale Gaussian volume properties in the present invention.

[0079] Figure 3 It is the three-dimensional reconstruction structure of the space target point cloud generated by the present invention. DETAILED DESCRIPTION

[0080] An embodiment of the present invention provides a method for three-dimensional reconstruction and rendering of a space target point cloud, the method comprising:

[0081] Step 1: Receive the space-based optical image obtained by the optical observation equipment to observe the space target, use the Structure from Motion (SFM) algorithm to generate the initial sparse point cloud of the space target, and obtain the camera pose corresponding to the space-based optical image;

[0082] Step 2: Use the bilateral filtering algorithm to denoise and process outliers on the initial sparse point cloud to obtain the initial point cloud. Step 2 not only avoids the problems of poor quality and low utilization of spatial target images, but also can obtain a relatively accurate initial point cloud for subsequent rendering processing.

[0083] Step 3: assign an initial grayscale Gaussian volume to each point in the initial point cloud, wherein the attributes of the initial grayscale Gaussian volume include spatial position, covariance matrix, transparency and grayscale value;

[0084] Step 4: Convert the initial grayscale Gaussian body assigned to each point from the world coordinate system to the image coordinate system. Combined with the camera posture, use the projection formula to project each initial grayscale Gaussian body converted to the image coordinate system from a three-dimensional plane to a two-dimensional plane to obtain a two-dimensional projection image.

[0085] Step 5: Use differentiable rasterization rendering technology to render the initial grayscale Gaussian volume in each two-dimensional projection image to obtain a rendered image;

[0086] Step 6: The total loss function is constructed by integrating the photometric loss function, the structural similarity loss function and the depth loss function. The total loss value between the rendered image and the real image is calculated by the total loss function. Based on the total loss value, the properties of the initial grayscale Gaussian in the rendered image are optimized using the stochastic gradient descent method.

[0087] Step 7: Introduce an adaptive density control mechanism to automatically adjust the density of the initial point cloud according to the properties of the optimized initial grayscale Gaussian body and the spatial distribution of the spatial target, and generate a three-dimensional reconstruction structure of the spatial target point cloud.

[0088] In step 1, the steps of generating an initial sparse point cloud of a space target using the structure-from-motion algorithm SFM and obtaining the camera pose corresponding to the space-based optical image include:

[0089] The SIFT descriptor in the Structure from Motion (SFM) algorithm is used to describe the feature points in each space-based optical image; the feature points include satellite sailboard corner points, body corner points, and edge points.

[0090] Calculate the Euclidean distance between SIFT descriptors in any two space-based optical images, and construct feature point pairs from the feature points of the two space-based optical images with the closest Euclidean distance;

[0091] The camera pose corresponding to the space-based optical image is obtained through feature point pairs, and the camera pose includes a rotation matrix and a translation vector. In practical applications, a special software library or algorithm is usually used to implement this process, such as the solvePnP function in OpenCV, which can directly calculate the rotation matrix and translation vector of the camera based on the world coordinates and image coordinates of the known feature point pairs and the parameter matrix in the camera, thereby obtaining the camera pose.

[0092] Based on the triangulation algorithm, the 3D point positions of the initial sparse point cloud in each space-based optical image are recovered from the feature point pairs to obtain the initial sparse point cloud of the space target.

[0093] In step 2, the bilateral filtering algorithm is a nonlinear filtering algorithm that can smooth the image while maintaining edge information. In point cloud processing, noise often appears as irregular fluctuations or outliers in point cloud data, and bilateral filtering can effectively reduce these noises while maintaining sharp features and edge information in the point cloud. The bilateral filtering algorithm updates the position and attribute value of the current point by calculating the weights of the neighboring points around the point, using the coordinates of the point as the spatial domain and the color of the point as the grayscale value domain. The bilateral filtering algorithm includes:

[0094]

[0095] Among them, I filtered (p) is the filtered image output by the bilateral filtering algorithm, and p is each point in the initial sparse point cloud; W p is a normalization constant; q is the neighborhood point of p, N(p) represents the neighborhood space of point p, I(q) and I(p) represent the attribute values ​​of q and p respectively, G s (·) is the spatial domain weight function, which is used to measure the Euclidean distance between point clouds. r (·) is the grayscale domain weight function, which is calculated based on the attribute differences of the points. The more similar the grayscale attributes are, the greater the grayscale domain weight is.

[0096] In step 3, the properties of the initial grayscale Gaussian body include the spatial position, covariance matrix, transparency and grayscale value respectively:

[0097] The spatial position of the initial grayscale Gaussian body is the position coordinate of each point in the initial point cloud;

[0098] The covariance matrix of the initial grayscale Gaussian is:

[0099] Σ=R2SS T R2 T ;

[0100] Among them, Σ is the output value of the covariance matrix, R2 is the rotation matrix, S is the scaling matrix, and T represents the transpose of the matrix;

[0101] The transparency value of the initial grayscale Gaussian is in the range of [0-1], where 0 represents complete transparency and 1 represents complete opacity.

[0102] The grayscale value range of the initial grayscale Gaussian is [0,255].

[0103] In step 4, the method of converting the initial grayscale Gaussian volume assigned to each point from the world coordinate system to the image coordinate system includes:

[0104] First, transform each initial grayscale Gaussian from the world coordinate system XYZ to the camera coordinate system xyz:

[0105]

[0106] Where R is the rotation function between the world coordinate system XYZ and the camera coordinate system xyz, and S represents the translation vector between the origin of the camera coordinate system and the origin of the world coordinate system;

[0107] Then transform the camera coordinate system xyz to the image plane coordinate system x'y':

[0108]

[0109] Where f represents the focal length of the camera;

[0110] Finally, the image plane coordinate system x'y' is transformed into the image coordinate system MN;

[0111]

[0112] Where M and N are the total number of rows and columns of pixels in the computer memory, respectively. m and O n are the row and column numbers of the central pixel in the computer memory; S x is the distance between the centers of two adjacent sensors along the x direction, S y is the distance between the centers of two adjacent sensors along the y direction, L x is the number of sensor elements in the x direction, M x is the number of pixels in a row, and μ is the uncertainty image scaling factor of the camera.

[0113] In step 4, the projection formula is:

[0114] P=K·X c ;

[0115] Among them, P is the two-dimensional coordinate of the two-dimensional projection image, K is the camera intrinsic parameter matrix in the camera pose, X c is the point in each initial grayscale Gaussian volume transformed into the image coordinate system.

[0116] In step 5, the method of rendering the initial grayscale Gaussian volume in each two-dimensional projection image using a differentiable rasterization rendering technology to obtain a rendered image includes:

[0117] The initial grayscale Gaussian body in the two-dimensional projection image is an ellipse of a two-dimensional Gaussian distribution. Based on the grayscale value in the initial grayscale Gaussian body attribute, the distance between all the target pixels to be sought and the center of the ellipse is calculated to obtain an opacity set (the closer, the more opaque), and each two-dimensional Gaussian distribution ellipse has its own grayscale value. The process of performing this calculation on all the target pixels to be sought is called differentiable rasterization. All the initial grayscale Gaussian bodies are superimposed by the opacity set, that is, all the two-dimensional Gaussian distribution ellipses are superimposed to obtain a rendered image.

[0118] In step 6, the total loss function constructed is:

[0119] L=λ1L1+λ2L ssim +(1-λ1-λ2)L depth ;

[0120] Among them, L is the total loss function value, λ1 is the weight factor of the photometric loss function, λ2 is the weight factor of the structural similarity loss function, L1 is the photometric loss function of the rendered image and the real image, L ssim is the structural similarity loss function, L depth is the depth loss function. Preferably, λ1=0.3, λ2=0.3.

[0121] In step 6, the calculation method of the photometric loss function is:

[0122]

[0123] Among them, L1 is the light loss value, N' is the number of pixels in the image, i is the number of pixels in the image, and I render (n i ) is the rendered image at n i The photometric value at real (n i ) is the real image in n i The photometric value at i is the pixel position in the image;

[0124] The calculation method of the structural similarity loss function is:

[0125] L ssim =1-SSIM(I render ,I real );

[0126] Among them, L ssimis the structural similarity loss value, SSIM(I render ,I real ) is the structural similarity index between the rendered image and the real image;

[0127]

[0128] In the formula, is the average pixel value of the rendered image, is the pixel average value of the real image, C1 is the pixel average value constant, which is used to avoid the situation where the denominator is 0. is the covariance of the rendered image and the real image, C2 is the standard deviation constant, which is used to avoid the situation where the denominator is 0. is the standard deviation of the rendered image, is the standard deviation of the real image;

[0129] The calculation method of the depth loss function is:

[0130]

[0131] Among them, L depth is the depth loss value, D render (n i ) is the rendering image in n i The depth value at real (n i ) The real image is in n i The depth value at .

[0132] In step 6, based on the total loss value, the update formula for optimizing the properties of the initial grayscale Gaussian in the rendered image using the stochastic gradient descent method is:

[0133]

[0134] Among them, P' is the optimized spatial position, P is the spatial position in the properties of the initial grayscale Gaussian, α' is the optimized transparency, α is the transparency in the properties of the initial grayscale Gaussian; Σ' is the optimized covariance, Σ is the covariance in the properties of the initial grayscale Gaussian; G' is the optimized grayscale value, G is the grayscale value in the properties of the initial grayscale Gaussian, and η is the learning rate; They are respectively the gradient of the total loss function value with respect to the spatial position, transparency, covariance and grayscale value in the attributes of the initial grayscale Gaussian.

[0135] In step seven, an adaptive density control mechanism is introduced to automatically adjust the density of the initial point cloud according to the properties of the optimized initial grayscale Gaussian volume and the spatial distribution of the spatial target. The method for generating a three-dimensional reconstruction structure of the spatial target point cloud includes:

[0136] Set the gradient threshold;

[0137] If the pixel gradient change of the current volume attribute of the optimized initial grayscale Gaussian in the rendered image exceeds the threshold, it is judged that the number of initial grayscale Gaussian volumes is insufficient and Gaussian volume replication is required;

[0138] If the pixel gradient change of the current attribute of the optimized initial grayscale Gaussian body in the rendered image is lower than the threshold, and the gradient change of the surrounding adjacent pixels is also lower than the threshold, it is determined that the number of initial grayscale Gaussian bodies is too large and the current initial grayscale Gaussian body needs to be replaced.

[0139] The adaptive density control mechanism in this step refers to adaptively controlling the density of the current initial grayscale Gaussian body during the optimization process, including increasing or decreasing the number of Gaussian ellipsoids as needed to better capture the details of the scene. For example, if the change rate of each pixel value in the rendered image relative to its surrounding pixels exceeds 30%, it means that the detail features of the area have changed significantly, the current number of Gaussian bodies is insufficient, and small Gaussian bodies need to be copied; if the pixel gradient change of the rendered image is less than 10%, and the gradient change of the surrounding pixels is also small, it indicates that there are too many Gaussian bodies, and the current large Gaussian body needs to be replaced with two small Gaussian bodies.

[0140] Verify the instance

[0141] Example 1

[0142] Take satellite A as an example. Figure 1-3 As shown:

[0143] 1) Optical observation equipment B observes satellite A and obtains multiple space-based optical images;

[0144] 2) Using the SFM algorithm, generate the initial sparse point cloud C of satellite A and the camera pose corresponding to the space-based optical image;

[0145] 3) The initial sparse point cloud C is subjected to denoising and outlier processing using bilateral filtering to obtain the processed initial point cloud C';

[0146] 4) For each point in the initial point cloud C', an initial grayscale Gaussian body is assigned, and the attributes of the initial grayscale Gaussian body include spatial position, covariance matrix, transparency and grayscale value;

[0147] 5) Perform coordinate transformation and projection operations on the initial grayscale Gaussian volume of satellite A and render it onto a two-dimensional image;

[0148] 6) Using the total loss value L between the rendered image and the real image, the properties of the initial grayscale Gaussian in the rendered image are optimized using the stochastic gradient descent method;

[0149] 7) An adaptive density control mechanism is introduced to automatically adjust the density of the initial point cloud according to the properties of the optimized initial grayscale Gaussian body and the spatial distribution of the spatial target, and generate a three-dimensional reconstruction structure of the spatial target point cloud.

[0150] Depend on Figure 1 and Figure 2 It can be seen that the rendered image is quite consistent with the real image. The rendering effect of the main part of satellite A is good, and the structure of each part can be clearly rendered. For the sailboard part in the center of the feature point, since the texture is darker and the sailboard is in motion, the shape of the sailboard can be reconstructed well, but the rendering effect of texture wrinkles is slightly lacking.

[0151] Depend on Figure 3 The three-dimensional point cloud structure of satellite A can be obtained, the sailboard and main structure can be clearly seen, and information such as the size and dimensions of the space target can be obtained, achieving a good reconstruction effect.

[0152] The beneficial effects of the embodiments of the present invention are:

[0153] 1. Aiming at the problem of multi-view image fusion: In satellite observation, due to changes in observation angle, distance and lighting conditions, the multiple space-based optical images obtained may have problems such as perspective differences, information overlap or missing. The present invention uses the SFM (Structure from Motion) algorithm to effectively integrate these multi-view image information and generate an initial sparse point cloud, which provides a basis for subsequent 3D reconstruction, applies 3D GS technology to the field of space targets, and achieves good 3D reconstruction and rendering effects;

[0154] 2. Regarding the point cloud data quality and noise processing issues: The initial point cloud data often contains a lot of noise and outliers, which will seriously affect the accuracy and effect of 3D reconstruction. The present invention performs bilateral filtering denoising on the initial sparse point cloud, denoises and processes outliers on the initial sparse point cloud, and obtains cleaner and more accurate point cloud data, thereby ensuring the accuracy of reconstruction and rendering results, and solving the problems of the prior art using images with low efficiency data registration rate, resulting in sparse initial point clouds, and the generation of initial point clouds, due to the influence of equipment noise and space environment, the generated point clouds have noise, outliers and other problems;

[0155] 3. The color expression in the Gaussian body is expressed by grayscale values, which reduces the complexity of the expression by spherical harmonic functions, simplifies color calculations, and improves the running speed;

[0156] 4. Optimizing the matching degree between rendered images and real images: There may be differences between rendered images and real images, which will affect the accuracy of the 3D reconstruction results. The total loss function constructed by the present invention integrates the photometric loss function, the structural similarity loss function and the depth loss function, which can better measure the consistency of the depth information between the rendered image and the real image, optimize the overall rendering effect, and adapt to complex scenes. This method can realize the 3D reconstruction and rendering of space targets in special space environments.

[0157] 5. Enhance the realism of rendered images: By using differentiable rasterization rendering technology and stochastic gradient descent method to optimize the properties of the initial grayscale Gaussian volume in the rendered image, a more realistic rendering result that is close to the real image can be generated, providing a more accurate data basis for subsequent image analysis and application.

[0158] 6. Improving the accuracy and efficiency of three-dimensional reconstruction: The technical solution disclosed in the present invention significantly improves the accuracy and efficiency of three-dimensional reconstruction, and provides more reliable technical support for the field of satellite observation and three-dimensional reconstruction.

[0159] 7. Improve the breadth and flexibility of technology applications: The present invention is not only applicable to the fields of satellite observation and three-dimensional reconstruction, but can also be extended to other occasions that require three-dimensional reconstruction and real-time rendering, such as aerial photogrammetry and remote sensing image processing, showing broad application prospects and flexibility.

[0160] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for three-dimensional reconstruction and rendering of a space target point cloud, characterized in that: The method includes: Step 1: Receive the space-based optical image obtained by the optical observation equipment to observe the space target, use the motion recovery structure algorithm SFM to generate the initial sparse point cloud of the space target, and obtain the camera pose corresponding to the space-based optical image; Step 2: Use bilateral filtering algorithm to denoise and process outliers on the initial sparse point cloud to obtain the initial point cloud; Step 3: assign an initial grayscale Gaussian volume to each point in the initial point cloud, wherein the attributes of the initial grayscale Gaussian volume include spatial position, covariance matrix, transparency and grayscale value; Step 4: Convert the initial grayscale Gaussian body assigned to each point from the world coordinate system to the image coordinate system. Combined with the camera posture, use the projection formula to project each initial grayscale Gaussian body converted to the image coordinate system from a three-dimensional plane to a two-dimensional plane to obtain a two-dimensional projection image. Step 5: Use differentiable rasterization rendering technology to render the initial grayscale Gaussian volume in each two-dimensional projection image to obtain a rendered image; Step 6: The total loss function is constructed by integrating the photometric loss function, the structural similarity loss function and the depth loss function. The total loss value between the rendered image and the real image is calculated by the total loss function. Based on the total loss value, the properties of the initial grayscale Gaussian in the rendered image are optimized using the stochastic gradient descent method. Step 7: Introduce an adaptive density control mechanism to automatically adjust the density of the initial point cloud according to the properties of the optimized initial grayscale Gaussian body and the spatial distribution of the spatial target, and generate a three-dimensional reconstruction structure of the spatial target point cloud.

2. The method according to claim 1, characterized in that In step 1, the steps of generating an initial sparse point cloud of a space target using the structure-from-motion algorithm SFM and obtaining the camera pose corresponding to the space-based optical image include: The SIFT descriptor in the Structure from Motion (SFM) algorithm is used to describe the feature points in each space-based optical image. Calculate the Euclidean distance between SIFT descriptors in any two space-based optical images, and construct feature point pairs from the feature points of the two space-based optical images with the closest Euclidean distance; Acquire the camera pose corresponding to the space-based optical image through the feature point pair, wherein the camera pose includes a rotation matrix and a translation vector; Based on the triangulation algorithm, the 3D point positions of the initial sparse point cloud in each space-based optical image are recovered from the feature point pairs to obtain the initial sparse point cloud of the space target.

3. The method according to claim 1, characterized in that In step 2, the bilateral filtering algorithm includes: Among them, I filtered (p) is the filtered image output by the bilateral filtering algorithm, and p is each point in the initial sparse point cloud; W p is a normalization constant; q is the neighborhood point of p, N(p) represents the neighborhood space of point p, I(q) and I(p) represent the attribute values ​​of q and p respectively, G s (·) is the spatial domain weight function, which is used to measure the Euclidean distance between point clouds. r (·) is the grayscale domain weight function. The more similar the grayscale attributes are, the greater the grayscale domain weight is.

4. The method according to claim 1, characterized in that In step 4, the method of converting the initial grayscale Gaussian volume assigned to each point from the world coordinate system to the image coordinate system includes: First, transform each initial grayscale Gaussian from the world coordinate system XYZ to the camera coordinate system xyz: Where R is the rotation function between the world coordinate system XYZ and the camera coordinate system xyz, and S represents the translation vector between the origin of the camera coordinate system and the origin of the world coordinate system; Then transform the camera coordinate system xyz to the image plane coordinate system x'y': Where f represents the focal length of the camera; Finally, the image plane coordinate system x'y' is transformed into the image coordinate system MN; Where M and N are the total number of rows and columns of pixels in the computer memory, respectively. m and O n are the row and column numbers of the central pixel in the computer memory; S x is the distance between the centers of two adjacent sensors along the x direction, S y is the distance between the centers of two adjacent sensors along the y direction, L x is the number of sensor elements in the x direction, M x is the number of pixels in a row, and μ is the uncertainty image scaling factor of the camera.

5. The method according to claim 1, characterized in that In step 4, the projection formula is: P=K·X c ; Among them, P is the two-dimensional coordinate of the two-dimensional projection image, K is the camera intrinsic parameter matrix in the camera pose, X c is the point in each initial grayscale Gaussian volume transformed into the image coordinate system.

6. The method according to claim 1, characterized in that In step 5, the method of rendering the initial grayscale Gaussian volume in each two-dimensional projection image using a differentiable rasterization rendering technology to obtain a rendered image includes: The initial grayscale Gaussian body in the two-dimensional projection image is an ellipse of two-dimensional Gaussian distribution. Based on the grayscale value in the attributes of the initial grayscale Gaussian body, the distance between all target pixels to be found and the center of the ellipse is calculated to obtain an opacity set; all initial grayscale Gaussian bodies are superimposed through the opacity set to obtain a rendered image.

7. The method according to claim 1, characterized in that In step 6, the total loss function constructed is: L=λ1L1+λ2L ssim +(1-λ1-λ2)L depth ; Among them, L is the total loss function value, λ1 is the weight factor of the photometric loss function, λ2 is the weight factor of the structural similarity loss function, L1 is the photometric loss function of the rendered image and the real image, L ssim is the structural similarity loss function, L depth is the depth loss function.

8. The method according to claim 7, characterized in that In step 6, the calculation method of the photometric loss function is: Among them, L1 is the light loss value, N' is the number of pixels in the image, i is the number of pixels in the image, and I render (n i ) is the rendered image at n i The photometric value at I real (n i ) is the real image in n i The photometric value at i is the pixel position in the image; The calculation method of the structural similarity loss function is: L ssim =1-SSIM(I render ,I real ); Among them, L ssim is the structural similarity loss value, SSIM(I render ,I real ) is the structural similarity index between the rendered image and the real image; In the formula, is the average pixel value of the rendered image, is the pixel average value of the real image, C1 is the pixel average value constant, which is used to avoid the situation where the denominator is 0. is the covariance of the rendered image and the real image, C2 is the standard deviation constant, which is used to avoid the situation where the denominator is 0. is the standard deviation of the rendered image, is the standard deviation of the real image; The calculation method of the depth loss function is: Among them, L depth is the depth loss value, D render (n i ) is the rendering image in n i The depth value at real (n i ) The real image is in n i The depth value at .

9. The method according to claim 1, characterized in that In step 6, based on the total loss value, the update formula for optimizing the properties of the initial grayscale Gaussian in the rendered image using the stochastic gradient descent method is: Among them, P' is the optimized spatial position, P is the spatial position in the properties of the initial grayscale Gaussian, α' is the optimized transparency, α is the transparency in the properties of the initial grayscale Gaussian; Σ' is the optimized covariance, Σ is the covariance in the properties of the initial grayscale Gaussian; G' is the optimized grayscale value, G is the grayscale value in the properties of the initial grayscale Gaussian, and η is the learning rate; They are respectively the gradient of the total loss function value with respect to the spatial position, transparency, covariance and grayscale value in the attributes of the initial grayscale Gaussian.

10. The method according to claim 1, characterized in that In step seven, an adaptive density control mechanism is introduced to automatically adjust the density of the initial point cloud according to the properties of the optimized initial grayscale Gaussian volume and the spatial distribution of the spatial target. The method for generating a three-dimensional reconstruction structure of the spatial target point cloud includes: Set the gradient threshold; If the pixel gradient change of the current volume attribute of the optimized initial grayscale Gaussian in the rendered image exceeds the threshold, it is judged that the number of initial grayscale Gaussian volumes is insufficient and Gaussian volume replication is required; If the pixel gradient change of the current attribute of the optimized initial grayscale Gaussian body in the rendered image is lower than the threshold, and the gradient change of the surrounding adjacent pixels is also lower than the threshold, it is determined that the number of initial grayscale Gaussian bodies is too large and the current initial grayscale Gaussian body needs to be replaced.

Images