Cutter three-dimensional scene reconstruction method and system based on 3DGS

Through the 3DGS-based tool three-dimensional scene reconstruction method, high-quality three-dimensional scenes are generated, which solves the problem of relying on manual or high-end equipment in the reconstruction of digital twin tool models of CNC machine tools, and achieves efficient three-dimensional reconstruction and rendering effects.

CN118918237APending Publication Date: 2024-11-08QIQIHAR UNIVERSITY
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202410927370.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art relies on manual reconstruction or high-end equipment in the reconstruction of digital twin tool models of CNC machine tools, resulting in high cost of labor and equipment and long reconstruction time, and failure to effectively apply three-dimensional reconstruction technology.

Method used

Using the 3DGS-based tool three-dimensional scene reconstruction method, a three-dimensional point cloud is generated by obtaining tool model image information, expanding into a Gaussian ellipsoid, using spherical harmonic function to record color and opacity information, and rendering high-quality through depth and scale loss optimization Three-dimensional scene.

Benefits of technology

The end-to-end three-dimensional reconstruction method is realized, which reduces labor and equipment costs, shortens the reconstruction time, and improves the quality of three-dimensional scene rendering.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918237A_ABST
    Figure CN118918237A_ABST
Patent Text Reader

Abstract

The invention discloses a cutter three-dimensional scene reconstruction method and system based on 3DGS. The method comprises the steps that cutter model picture information is acquired; generating a three-dimensional point cloud of the tool model according to the tool model picture information; expanding the three-dimensional point cloud of the tool model into a Gaussian ellipsoid; recording color and opacity information of the Gaussian ellipsoid by using a spherical harmonic function; through depth loss, Gaussian ellipsoid depth is obtained through point cloud projection, supervised training is carried out on the depth model, and the position of a Gaussian ellipsoid near the surface of the to-be-rendered object is obtained; according to the method, more 3D Gaussian ellipsoids are split and cloned by using scale loss, and the split and cloned 3D Gaussian ellipsoids are more tightly attached to the surface of the original ellipsoid, so that a cutter three-dimensional scene with higher quality is rendered. According to the method, three-dimensional reconstruction can be completed by inputting the picture, an end-to-end reconstruction mode is realized, and the labor and equipment cost of three-dimensional reconstruction of the cutter is effectively reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of three-dimensional reconstruction of CNC machine tool tools, and in particular relates to a three-dimensional scene reconstruction method and system for tools based on 3DGS. Background Art

[0002] With the development of simulation technology and sensor monitoring technology, according to the literature (Tao F, Zhang M, Nee A YC. Digital twin driven smart manufacturing [M]. Academic Press, 2019), CNC machine tools and digital twins are the core equipment and key enabling technologies of intelligent manufacturing. The application research of digital twin technology in CNC machine tools has far-reaching significance for promoting the development and standardization of intelligent manufacturing in my country. Through this technology, the performance, efficiency and intelligence level of machine tools can be effectively improved, thereby providing strong support for the transformation and upgrading of the entire manufacturing industry. According to the literature (XUE Ruijuan, HUANG Zuguang, WANG Jinjiang, TAO Fei, ZHANG Peisen, WU Yiran. Research on digital twin standards system of CNC machine tool [J]. Manufacturing Technology & Machine Tool, 2023, (3): 39-50.), the digital twin model is the core basic element in the digital twin system and the basis for realizing the digital twin function. Visualization, reconfigurability and interactivity are important model performance requirements. A 3D Gaussian Splatting technology is proposed in the literature (Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. 3D Gaussian Splatting for Real-Time Radiance Field Rendering [J]. ACM Transactions on Graphics, 2023, 42 (4): 1-14). As an efficient real-time rendering technology, this technology can be applied to digital twins to improve the visualization and interactivity of digital twins.

[0003] Although three-dimensional scene reconstruction technology has been widely studied at home and abroad, its application is mainly concentrated in medical, architectural science and indoor scenes. For example, the literature (Chen Pei, Fan Shengming, Li Shugang. Application value of three-dimensional reconstruction technology in thoracoscopic radical surgery for lung cancer [J]. Zhejiang Trauma Surgery, 2023 (04)) The application of three-dimensional reconstruction technology in thoracoscopic lung segment resection in patients with lung cancer helps to shorten the time and intraoperative blood loss of lung cancer surgery. The literature (Yang Junsheng, Wang Zijian, Tang Yu. Research on the determination method of the rebound rate of tunnel shotcrete based on image three-dimensional reconstruction [J]. Journal of Building Science and Engineering. 2024, 41 (02)) Based on the structure from motion (SFM) algorithm, the patch-based dense matching (PMVS) algorithm and the triangulation principle, the three-dimensional surface mesh model of the actual tunnel excavation contour and the inner contour of the primary branch is established; through the grid ray method, the spatial difference between the actual tunnel excavation contour and the three-dimensional mesh model of the inner contour of the primary branch is solved to obtain the actual effective shotcrete volume of the primary branch. In the field application, the construction quality control suggestions for reducing the rebound rate were proposed in combination with the test results of the rebound rate of shotcrete. After being adopted on site, the construction was good and the quality of tunnel construction was guaranteed; this determination method has good engineering application value. The literature (Jiaming Sun, Yiming Xie, Linghao Chen. Neural Recon: Real-Time Coherent 3D Reconstruction from Monocular Video. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.) directly reconstructs the local surface of each video clip through a neural network, which is represented as a sparse TSDF volume. A learning TSDF fusion module based on a gated recurrent unit is used to guide the network to fuse the features of previous clips. Accurate, coherent and real-time reconstruction of indoor scene surfaces is achieved. In the field of CNC machine tool tools, the generation of digital twin models often relies on workers to use modeling software for manual modeling, and then reconstruct the three-dimensional scene through operations such as UV mapping, which requires them to not only master modeling skills but also have professional knowledge of CNC machine tools. Another method is to use instruments such as 3D scanners to measure and achieve reverse modeling, however, these devices are usually expensive. At present, 3D reconstruction technology has not been applied and verified in the field of CNC machine tools.

[0004] After the above analysis, the existing technology still mainly relies on manual reconstruction or the use of high-end equipment in the current reconstruction process of digital twin tool scenes for CNC machine tools. Summary of the invention

[0005] In view of the existing technology, in the current CNC machine tool digital twin tool model reconstruction process, it mainly relies on manual reconstruction or the use of expensive scanning equipment. In order to reduce labor and equipment costs and shorten the reconstruction time, the present invention proposes a tool three-dimensional scene reconstruction method and system based on 3DGS. Three-dimensional reconstruction can be completed by inputting pictures, realizing an end-to-end reconstruction method, and effectively reducing the labor and equipment costs of tool three-dimensional reconstruction.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The 3DGS-based tool 3D scene reconstruction method includes the following steps:

[0008] Get tool model image information;

[0009] Generate a three-dimensional point cloud of the tool model according to the tool model image information;

[0010] Expanding the three-dimensional point cloud of the tool model into a Gaussian ellipsoid;

[0011] Use spherical harmonics to record the color and opacity information of the Gaussian ellipsoid;

[0012] The Gaussian ellipsoid that records color and opacity information is loss-optimized to render a higher-quality three-dimensional tool scene.

[0013] Preferably, according to the tool model image information, the method for generating a tool model point cloud comprises:

[0014] Extract feature points from multi-view images according to the process of converting the two-dimensional image space into a three-dimensional world coordinate system;

[0015] Based on the feature points, the external parameters R and T of the initial camera are estimated using a five-point method;

[0016] For the same-name points x1 and x2 in the two images, the three-dimensional space point coordinates X of the same-name points x1 and x2 are solved by triangulation to obtain the three-dimensional point cloud of the tool model;

[0017] Among them, the process of converting the two-dimensional image space to the three-dimensional world coordinate system is:

[0018] sm=M(RX+T),

[0019] Where s is a scaling factor and m = (u, v, 1) T are two-dimensional homogeneous coordinates, M is the camera intrinsic parameter, X is the three-dimensional space point coordinate, and R and T are the camera extrinsic rotation and translation matrices.

[0020] Preferably, the method of expanding the three-dimensional point cloud of the tool model into a Gaussian ellipsoid comprises:

[0021]

[0022] Where Σ is the covariance matrix, x=(x1,x2,x3) T is a random variable vector in three-dimensional space, u=(u1,u2,u3) T is the mean vector, (xu) T Σ -1 When (xu) is a constant, an ellipsoid is defined, G(x,u,Σ)=[0,1], thus constructing a solid Gaussian ellipsoid.

[0023] Preferably, the method of using spherical harmonics to record the color and opacity information of the Gaussian ellipsoid includes:

[0024] For color and opacity information, spherical harmonic coefficients are calculated separately for each color channel;

[0025] Among them, the spherical harmonic coefficients can be obtained by integration:

[0026]

[0027] Where Ω is the integration area on the sphere, yes The complex conjugate of is the basis function, θ and φ are the polar angle and azimuth angle on the sphere respectively, and f(θ,φ) is the color function.

[0028] Preferably, the method for performing loss optimization on the Gaussian ellipsoid recording color and opacity information comprises:

[0029] Through depth loss, the depth of the Gaussian ellipsoid is obtained by point cloud projection to perform supervised training on the depth model, and the position of the Gaussian ellipsoid near the surface of the object to be rendered is obtained;

[0030] Using scale loss, more 3D Gaussian ellipsoids are split and cloned, and the split and cloned 3D Gaussian ellipsoids are made to adhere more closely to the surface of the original ellipsoid.

[0031] Preferably, the depth of the Gaussian sphere obtained by point cloud projection is used to perform supervised training on the depth model, so that the position of the Gaussian sphere near the surface of the object specifically includes:

[0032] Use the depth loss function to control the position of the Gaussian ellipsoid:

[0033] L depth =||DD * dense ||1,

[0034] Among them, D is the depth map obtained by projecting the point cloud, D *dense is the depth map estimated by the MLP depth model; Among them, d i Represents the depth of each Gaussian sphere read from the point cloud, a i =(1-exp(-σ i δ i )) represents opacity, σ represents the probability density of light being blocked by particles, δ represents the interval between Gaussian balls, It represents the probability that the light is not blocked before point i.

[0035] Preferably, scale loss is used to make the three-dimensional Gaussian more closely attached to the surface of the ellipse, and the three-dimensional Gaussian is split and cloned into more 3D Gaussian balls, specifically including:

[0036]

[0037] Among them, L scale represents the Gaussian scale loss, that is, the size loss of the Gaussian ball, N scale represents the scale of all Gaussian balls, N is the number of all Gaussian balls, represents the scale of the i-th Gaussian ball, represents the Gaussian scaled mean.

[0038] The present invention also provides a tool three-dimensional scene reconstruction system based on 3DGS, including: an image acquisition module, a point cloud generation module, an expansion module, a recording module and a rendering module;

[0039] The image acquisition module is used to acquire tool model image information;

[0040] The point cloud generation module is used to generate a three-dimensional point cloud of the tool model according to the tool model image information;

[0041] The expansion module is used to expand the three-dimensional point cloud of the tool model into a Gaussian ellipsoid;

[0042] The recording module is used to record the color and opacity information of the Gaussian ellipsoid using spherical harmonics;

[0043] The rendering module is used to perform loss optimization on the Gaussian ellipsoid that records color and opacity information, and render a higher quality three-dimensional tool scene.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention proposes a 3DGS-based tool 3D scene reconstruction method and system. The depth map generated by MLP is supervised by projecting a 3D point cloud to generate a depth map, optimize the position of the 3D Gaussian ball, use scale loss to control the size and number of the 3D Gaussian ball, make the 3D Gaussian more closely attached to the surface of the ellipse, and split and clone more 3D Gaussian balls to improve the quality of 3D scene rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 It is a schematic flow chart of a method for reconstructing a three-dimensional scene of a tool based on 3DGS according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram showing the comparison of average Gaussian sizes before and after optimization in an embodiment of the present invention;

[0049] Figure 3 Schematic diagram showing the comparison of the number of Gaussian spheres before and after optimization according to an embodiment of the present invention;

[0050] Figure 4 1 is a comparison diagram of the similarity of the learned perceived image blocks before and after optimization according to an embodiment of the present invention, (a) is before optimization, and (b) is after optimization.

[0051] Figure 5 1 is a peak signal-to-noise ratio comparison diagram before and after optimization of an embodiment of the present invention, (a) is before optimization, (b) is after optimization;

[0052] Figure 6 1 is a structural similarity comparison diagram before and after optimization of an embodiment of the present invention, (a) is before optimization, (b) is after optimization;

[0053] Figure 7 This is a comparison diagram of the effects before and after optimization of the tool 3D scene reconstruction of Case 1 of an embodiment of the present invention, (a) is before optimization, and (b) is after optimization. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Embodiment 1

[0057] like Figure 1 As shown, the present invention provides a tool three-dimensional scene reconstruction method based on 3DGS (3D Gaussian Splatting), comprising the following steps:

[0058] Get tool model image information;

[0059] Generate a three-dimensional point cloud of the tool model based on the tool model image information;

[0060] Expand the three-dimensional point cloud of the tool model into a Gaussian ellipsoid;

[0061] Use spherical harmonics to record the color and opacity information of the Gaussian ellipsoid;

[0062] The loss of the Gaussian ellipsoid that records color and opacity information is optimized to render a higher quality three-dimensional tool scene.

[0063] In this embodiment, the image information of the tool model is obtained by taking pictures around the model with a camera.

[0064] In this embodiment, a model point cloud is generated based on an image. The transformation from pixel coordinates to three-dimensional coordinates is a common problem in computer vision and image processing, which involves the process of converting from a two-dimensional image space to a three-dimensional world coordinate system. The transformation can be performed using the following formula:

[0065] sm=M(RX+T) (1)

[0066] s is a scaling factor, m = (u, v, 1) T are two-dimensional homogeneous coordinates, M is the camera intrinsic parameter, X is the three-dimensional space point coordinate, and R and T are the camera extrinsic rotation and translation matrices.

[0067] According to formula (1), feature points are extracted from multi-view images, and the five-point method is used to estimate the external parameters R and T of the initial camera. For the points x1 and x2 with the same name in the two images, their three-dimensional coordinates X are solved by triangulation. Based on this process, the three-dimensional point cloud of the model can be obtained.

[0068] In this embodiment, the 3D ellipsoid set is reconstructed. The point cloud is expanded into a Gaussian ellipsoid, because the Gaussian kernel is still closed after the Gaussian ellipsoid is affine transformed, and it is still Gaussian after integration along a certain axis, that is, from 3D to 2D. The Gaussian ellipsoid formula is as follows:

[0069]

[0070] Where Σ is the covariance matrix, x=(x1,x2,x3) T is a random variable vector in three-dimensional space, u=(u1,u2,u3) T is the mean vector, (xu) T Σ -1 When (xu) is a constant, an ellipsoid is defined, G(x,u,Σ)=[0,1], thus constructing a solid Gaussian ellipsoid.

[0071] In this embodiment, spherical harmonics record color and opacity information. Any function of spherical coordinates can be approximated by multiple spherical harmonics.

[0072]

[0073] are the coefficients, is the basis function. L is the order of the spherical harmonics, l and m are the reading and imaginary number of the spherical harmonics. θ and φ are the polar angle and azimuth angle on the sphere, respectively.

[0074] For color and opacity information, spherical harmonic coefficients are calculated for each color channel (such as red, green, and blue). For example, if we have a color function f r (θ,φ),f g (θ,φ),f b (θ, φ), respectively represents the red, green, and blue values ​​of the color at the polar angle θ and the azimuth angle φ, and their spherical harmonic coefficients are calculated respectively. The spherical harmonic coefficients can be obtained by integration:

[0075]

[0076] Ω is the integration area on the sphere, yes The complex conjugate of .

[0077] In this embodiment, the loss is optimized. First, through the depth loss, the depth model is supervisedly trained using the Gaussian sphere depth obtained by point cloud projection, so that the position of the Gaussian sphere near the surface of the object (specifically, using point cloud projection to obtain the depth information of the Gaussian ellipsoid and conducting supervised training is an effective method that can be used to estimate the three-dimensional shape of the object surface). Using scale loss, more 3D Gaussian ellipsoids are split and cloned, and the split and cloned three-dimensional Gaussian ellipsoids are more closely attached to the surface of the original ellipsoid (specifically, in three-dimensional space, Gaussian distribution (or ellipsoid Gaussian) can be used to describe the position distribution of objects. When multiple similar 3D Gaussian ellipsoids need to be generated, a method called "split and clone" can be used to copy the existing Gaussian distribution. This process can be achieved by adding a scale loss term to the objective function to ensure that the generated ellipsoid is more closely attached to the surface of the original ellipsoid). In 3DGS, L1 and SSIM are used to superimpose and calculate the loss and gradient descent is used to update the parameters.

[0078] L=(1-λ)L1+λL SSIM (5)

[0079] λ = 0.2, L1 loss function, also known as mean absolute error (MAE), refers to the average of the absolute differences between the model prediction value and the true value, L SSIM It is a structural similarity index, which is an index used to measure the similarity between two digital images.

[0080] In order to improve the rendering quality, this paper proposes a new loss function:

[0081] L=L depth +L scale (6)

[0082] L depth =||DD * dense ||1 means calculating the L1 norm between two vectors, that is, the sum of the absolute value errors. * dense is the depth map estimated by the MLP depth model, where d i Indicates the depth of each Gaussian sphere read from the point cloud; a i =(1-exp(-σ i δ i )) represents opacity, σ represents the probability density of light being blocked by particles, and δ represents the interval between Gaussian spheres; It represents the probability that the light is not blocked before point i.

[0083] D is the depth map obtained by projecting the point cloud, which is obtained by the following transformation:

[0084]

[0085] R, T represent the rotation matrix and translation matrix. (X w ,Y w ,Z w ) is a point in world coordinates, (X c ,Y c ,Z c ) is a point in camera coordinates. Once the point cloud data is in camera coordinates, the 3D points can be mapped to the 2D image plane using perspective projection. This can be achieved using the camera's intrinsic matrix (K):

[0086]

[0087] f x 、f y are the x and y components of the focal length. x 、c y is the principal point coordinate. For each point projected onto the image plane, its depth value is the Z component of its camera coordinate. That is, Z c

[0088]

[0089] For each pixel (u, v) on the two-dimensional image plane, find the nearest point in the corresponding three-dimensional point cloud and calculate its depth value Z c Assigned to the pixel. In this way, a two-dimensional depth map can be generated.

[0090] Represents Gaussian scale loss, that is, the size of the Gaussian ball, N scale represents the scale of all Gaussian balls, N is the number of all Gaussian balls, represents the scale of the i-th Gaussian ball, represents the Gaussian scaled mean.

[0091] In this embodiment, forward propagation is performed and a forward propagation function is defined as:

[0092] In 3DGS, the Gaussian spheres are sorted by depth Z to determine the order of rendering. The rendering is shown in formula (11):

[0093]

[0094] C represents the rendering color, T i represents the probability that the light is not blocked before point i, c i is the coefficient of the spherical harmonics, which is the coefficient of step 4.

[0095] The final rendering order is determined by the MLP depth model. The present invention adds depth propagation and generates a depth map through MLP as follows:

[0096]

[0097] The depth D is given by the volume rendering along the ray.

[0098] d i Indicates the depth read from the point cloud;

[0099] a i =(1-exp(-σ i δ i )) represents opacity, σ represents the probability density of light being blocked by particles, and δ represents the interval between Gaussian spheres;

[0100] It represents the probability that the light is not blocked before point i.

[0101] In this embodiment, the optimization parameters are propagated backwards. The backward propagation function is as follows:

[0102]

[0103] L represents the total loss, t z represents the depth position of the Gaussian sphere, represents the gradient of the total loss to the depth loss of each pixel block, L D Represents the depth loss, which is updated using formula (13) in each round of backward propagation

[0104] The center position, covariance matrix, spherical harmonic function coefficients, and transparency ɑ of each Gaussian sphere are optimized by back propagation.

[0105] In this embodiment, the model is trained. The training mainly includes the following steps:

[0106] Every 1000 iterations, increase the order of the spherical harmonic coefficients.

[0107] Randomly select a camera view.

[0108] Render an image, getting information such as the view space point, visibility filter, and radius.

[0109] Calculate the loss to optimize the position and size of the Gaussian ball and perform backpropagation.

[0110] Subsequent operations are performed through the context without gradients:

[0111] Perform point cloud density operations based on the number of iterations

[0112] Update maximum radius information.

[0113] Point cloud density is increased and trimmed according to conditions.

[0114] Update the optimizer parameters

[0115] Output the rendering result.

[0116] The present invention uses a camera to collect tool data sets to experimentally verify the method. After collecting the tool image, the results obtained by running the source code and the improved version are as follows: Figures 2 to 7 As shown in the figure, the results show that the algorithm can effectively reduce the average size of Gaussian spheres by 40%, increase the number of Gaussian spheres by 36%, and improve the structural similarity index from 0.94 to 0.96, thereby rendering a higher quality three-dimensional tool scene.

[0117] Embodiment 2

[0118] The present invention also provides a tool three-dimensional scene reconstruction system based on 3DGS, including: an image acquisition module, a point cloud generation module, an expansion module, a recording module and a rendering module;

[0119] The image acquisition module is used to obtain the image information of the tool model;

[0120] The point cloud generation module is used to generate a three-dimensional point cloud of the tool model according to the tool model image information;

[0121] The expansion module is used to expand the three-dimensional point cloud of the tool model into a Gaussian ellipsoid;

[0122] The recording module is used to record the color and opacity information of the Gaussian ellipsoid using spherical harmonics;

[0123] The rendering module is used to perform loss optimization on the Gaussian ellipsoid that records color and opacity information, and render a higher quality three-dimensional tool scene.

[0124] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A 3DGS-based tool 3D scene reconstruction method, characterized in that: The following steps are involved: Get tool model image information; Generate a three-dimensional point cloud of the tool model according to the tool model image information; Expanding the three-dimensional point cloud of the tool model into a Gaussian ellipsoid; Use spherical harmonics to record the color and opacity information of the Gaussian ellipsoid; The loss of the Gaussian ellipsoid that records color and opacity information is optimized to render a higher quality three-dimensional tool scene.

2. The 3DGS-based tool 3D scene reconstruction method according to claim 1, characterized in that: According to the tool model image information, the method for generating a tool model point cloud includes: Extract feature points from multi-view images according to the process of converting the two-dimensional image space into a three-dimensional world coordinate system; Based on the feature points, the external parameters R and T of the initial camera are estimated using a five-point method; For the same-name points x1 and x2 in the two images, the three-dimensional space point coordinates X of the same-name points x1 and x2 are solved by triangulation to obtain the three-dimensional point cloud of the tool model; Among them, the process of converting the two-dimensional image space to the three-dimensional world coordinate system is: sm=M(RX+T), Where s is a scaling factor and m = (u, v, 1) T are two-dimensional homogeneous coordinates, M is the camera intrinsic parameter, X is the three-dimensional space point coordinate, and R and T are the camera extrinsic rotation and translation matrices.

3. The 3DGS-based tool 3D scene reconstruction method according to claim 1, characterized in that: The method of expanding the three-dimensional point cloud of the tool model into a Gaussian ellipsoid comprises: Where Σ is the covariance matrix, x=(x1,x2,x3) T is a random variable vector in three-dimensional space, u=(u1,u2,u3) T is the mean vector, (xu) T Σ -1 When (xu) is a constant, an ellipsoid is defined, G(x,u,Σ)=[0,1], thus constructing a solid Gaussian ellipsoid.

4. The 3DGS-based tool 3D scene reconstruction method according to claim 1, characterized in that: Methods for using spherical harmonics to record Gaussian ellipsoid color and opacity information include: For color and opacity information, spherical harmonic coefficients are calculated separately for each color channel; Among them, the spherical harmonic coefficients can be obtained by integration: Where Ω is the integration area on the sphere, yes The complex conjugate of is the basis function, θ and φ are the polar angle and azimuth angle on the sphere respectively, and f(θ,φ) is the color function.

5. The 3DGS-based tool 3D scene reconstruction method according to claim 1, characterized in that: Methods for loss optimization of the Gaussian ellipsoid that records color and opacity information include: Through depth loss, the depth of the Gaussian ellipsoid is obtained by point cloud projection to perform supervised training on the depth model, and the position of the Gaussian ellipsoid near the surface of the object to be rendered is obtained; Using scale loss, more 3D Gaussian ellipsoids are split and cloned, and the split and cloned 3D Gaussian ellipsoids are made to adhere more closely to the surface of the original ellipsoid.

6. The 3DGS-based tool 3D scene reconstruction method according to claim 5, characterized in that: The depth of the Gaussian ball is obtained by projecting the point cloud to perform supervised training on the depth model, so that the position of the Gaussian ball near the surface of the object specifically includes: Use the depth loss function to control the position of the Gaussian ellipsoid: L depth =||D-D * dense ||1, Among them, D is the depth map obtained by projecting the point cloud, D * dense is the depth map estimated by the MLP depth model; Among them, d i Represents the depth of each Gaussian sphere read from the point cloud, a i =(1-exp(-σ i δ i )) represents opacity, σ represents the probability density of light being blocked by particles, δ represents the interval between Gaussian balls, It represents the probability that the light is not blocked before point i.

7. The 3DGS-based tool 3D scene reconstruction method according to claim 5, characterized in that: Use scale loss to make the 3D Gaussian more closely attached to the surface of the ellipse, and split and clone more 3D Gaussian balls. Specifically include: Among them, L scale represents the Gaussian scale loss, that is, the size loss of the Gaussian ball, N is the number of all Gaussian balls, represents the scale of the i-th Gaussian ball, represents the Gaussian scaled mean.

8. The tool three-dimensional scene reconstruction system based on 3DGS is characterized by: include: Image acquisition module, point cloud generation module, expansion module, recording module and rendering module; The image acquisition module is used to acquire tool model image information; The point cloud generation module is used to generate a three-dimensional point cloud of the tool model according to the tool model image information; The expansion module is used to expand the three-dimensional point cloud of the tool model into a Gaussian ellipsoid; The recording module is used to record the color and opacity information of the Gaussian ellipsoid using spherical harmonics; The rendering module is used to perform loss optimization on the Gaussian ellipsoid that records color and opacity information, and render a higher quality three-dimensional tool scene.

Citation Information

Cited By

  • Turnout three-dimensional visualization method, device, equipment and medium

    CN119152134A

  • Three-dimensional scene novel view synthesis method based on matched light

    CN119478173A

  • A novel view synthesis method for 3D scenes based on matching light

    CN119478173B

  • Three-dimensional reconstruction method and device for hinged object, electronic equipment and storage medium

    CN119832158A

  • Plant three-dimensional reconstruction method and related device

    CN119991962A