Generalized rapid Mesh extraction method based on Gaussian

Abstract

The invention discloses a generalized rapid Mesh extraction method based on Gaussian, and the method comprises the steps: configuring a training data collection system, and carrying out the marking and preprocessing of data; the method comprises the following steps: constructing a generalized rapid Mesh extraction network based on Gaussian, and training the rapid Mesh extraction network; a training model of a generalized rapid Mesh extraction network based on Gaussian is used, testing is carried out under multiple data sets and different scenes, index evaluation is carried out on a result, the performance of the Mesh extraction method based on the Gaussian model can be further improved through a self-adaptive adjustment strategy and an optimized parameter estimation method, and the performance of the Mesh extraction method based on the Gaussian model is improved. The method has important research significance in theory, particularly has wide practical application prospects in scenes needing to process large-scale and high-dimensional complex data, aims to overcome the defects in the prior art and improve the adaptability and efficiency in different data sets and application scenes, and provides an innovative solution for the field of three-dimensional data processing. And further development and application expansion of the Mesh extraction technology are promoted.

Description

Technical Field

[0001] The invention relates to the technical field of artificial intelligence, and in particular to a generalizable fast Mesh extraction method based on Gaussian. Background Art

[0002] With the continuous progress of computer graphics, virtual reality and artificial intelligence technology, the reconstruction and representation of 3D models have become the core technology in various applications. Mesh structure, as the main form of 3D model, is widely used in industrial design, medical image processing, geographic information system (GIS), virtual reality (VR) and augmented reality (AR). Mesh extraction technology aims to automatically generate 3D mesh structure from point cloud, voxel data or image sequence, providing a basis for subsequent 3D reconstruction, simulation and analysis. With the diversification and complexity of 3D data sets, how to extract high-quality and stable Mesh structure under the challenging conditions of noise, irregularity and data loss has become a hot issue in current research. Traditional Mesh extraction methods such as Marching Cubes algorithm, Delaunay triangulation, Poisson Surface Reconstruction, etc. have solved the 3D reconstruction problem to a certain extent, but when faced with complex and noisy data, these methods usually produce reconstruction errors, surface discontinuities or geometric distortions. In addition, these methods often face bottlenecks of high computational complexity and large memory usage when processing large-scale 3D data or scenarios requiring real-time applications, which limits their widespread promotion in practical applications.

[0003] However, in order to overcome these challenges, Mesh extraction methods based on probability models have gradually entered the field of vision of researchers. As a probability distribution model with good expressive power, the Gaussian model has significant advantages in describing the spatial distribution characteristics of data. By introducing the Gaussian distribution, the probability density of point cloud data can be effectively estimated, thereby enhancing the robustness of the Mesh extraction process. This Gaussian-based Mesh extraction method has demonstrated good performance in dealing with scenes containing noise, complex geometric shapes, and partial data missing. Traditional Gaussian-based Mesh extraction methods usually require the preset of complex parameters and make strict assumptions on the prior distribution of the data, which increases the complexity and limitations of the method in practical applications. At the same time, with the increase in data scale and the complexity of geometric structure, these methods also face challenges in computational efficiency and adaptability. Summary of the invention

[0004] The present invention provides a generalizable fast Mesh extraction method based on Gaussian, which can effectively solve the problem proposed in the above background technology that the traditional Mesh extraction method based on Gaussian usually needs to preset complex parameters and make strict assumptions on the prior distribution of data, which increases the complexity and limitations of the method in practical applications. At the same time, with the increase of data scale and the complexity of geometric structure, these methods also face challenges in computational efficiency and adaptability.

[0005] To achieve the above object, the present invention provides the following technical solution: a generalizable fast Mesh extraction method based on Gaussian, comprising the following steps:

[0006] S1. Configure the training data collection system to label and preprocess the data;

[0007] S2. Build a generalizable fast Mesh extraction network based on Gaussian and train it;

[0008] S3, use the training model of the generalizable fast mesh extraction network based on Gaussian, test it in multiple data sets and different scenarios, and evaluate the results with indicators;

[0009] In S1, configuring the training data acquisition system, annotating and preprocessing the data specifically includes: installing and configuring camera equipment to collect data, obtaining multi-view and surround images, capturing information of the training scene from different angles, and performing registration, correction, annotation and data enhancement processing on the collected data;

[0010] In S2, a generalizable fast mesh extraction network based on Gaussian is constructed, and the network consists of an FPN feature extraction module, a cost volume, a volume rendering module and a stereo matching part;

[0011] In S3, the proposed method is extensively tested in wild scenes using smartphones, demonstrating its excellent reconstruction capability. In addition, the method is tested on the Tanks and Temples and DTU benchmarks and achieves state-of-the-art results.

[0012] According to the above technical solution, in S2, specifically, the multi-view image captured by the acquisition device is used to extract features by using the FPN pyramid network to obtain 2D features of the input image, and the 2D image features are projected to different depth plane spaces through homography transformation, and the features of different depth planes are aggregated to form a 3D feature cost volume, and the feature cost volume is regularized by 3D-CNN to generate a depth map. Based on the depth map, each pixel-aligned 3D point is feature encoded by aggregating multiple views and spatial information, and the encoded features are decoded for rendering.

[0013] According to the above technical solution, in S2, a simple depth-aware volume rendering module is integrated, in which each ray samples only one point, and a method combining Gaussian sputtering and volume rendering is used to render two views, and then the two rendered views are averaged into the final view, which is constructed in a cascade structure to propagate the depth map and the rendered view in a coarse-to-fine manner;

[0014] We bridge the gap between noisy 3DGS representation and smooth 3D mesh representation by injecting real-world priors into the depth extraction process, extracting geometry through a pre-trained stereo matching model, rendering stereo aligned image pairs corresponding to the original training poses, inputting these image pairs into the stereo model to obtain depth profiles, and finally fusion all profiles together to obtain a single mesh.

[0015] According to the above technical solution, the 3DGS represents the scene as a point cloud of 3D Gaussians, where each Gaussian has the attributes of opacity, rotation, scale, position and spherical harmonics, and the scaling of Gaussians is anisotropic, which enables them to represent smaller and finer structures in the scene;

[0016] The Gaussian algorithm uses the Structure from Motion (SFM) algorithm to initialize the input image, which extracts the camera pose and provides an initial guess for the position of the Gaussian. 3DGS has a short capture time and can be rendered in real time. 3D-GS is used to generate a generalizable Mesh extraction method.

[0017] For data collection, it is required to shoot different scenes under different lighting conditions. The data collection system uses multiple sets of RGB-D camera arrays to cover the panoramic view around the object in front, behind, left front, left rear, right front and right rear. The obtained images need to be processed with Colmap software to obtain relevant parameters of the image.

[0018] According to the above technical solution, in the overall network process, it is first necessary to construct a cost volume based on the features of the input image, and infer the scene depth through 3D-CNN regularization. Given a set of input images, the 2D features of the images are obtained by extracting multi-scale features from the input view by the FPN pyramid feature extraction network. Following the MVS-based method, multiple parallel planes are first established on the target view, and then the features of the source view are mapped to these parallel planes using homography transformation, as shown below:

[0019]

[0020] Among them, [K i ,R i ,t i ] and [R t ,K t ,t t ] are the intrinsic, rotation, and translation matrices of the source and target views, respectively, a represents the principal axis of the view camera, I represents the identity matrix, and Z represents the sampling depth;

[0021] Construct the cost volume C by calculating its variance;

[0022]

[0023] Where N represents the number of input views, V i Represents multi-scale features, Represents the average value of multi-scale features;

[0024] The obtained cost volume C is input into the 3D-CNN for regularization to obtain the probability volume. Each depth hypothesis plane is weighted by the depth probability distribution to obtain the final depth. Through the obtained depth, each pixel can be projected to a 3D point, that is, the position of the 3D Gaussian. These 3D points are then feature encoded to establish a pixel-aligned 3D Gaussian representation.

[0025] According to the above technical solution, in the process of feature encoding of 3D points, the features in the source view are first mapped to the target view using the homography transformation matrix, and then these multi-scale features are aggregated using the pooling network ρ.

[0026] Due to the characteristics of Gaussian, each Gaussian contributes to the pixel color value in a specific area of ​​the image. The 2D-Unet network is used for spatial enhancement, and then the features are decoded to obtain the Gaussian parameters for rendering. Each Gaussian can be represented by {μ, s, r, α, c}. For the position μ, it can be obtained based on the estimated depth back projection, where s, r, α, and c represent the scaling, rotation matrix, opacity, and color, respectively.

[0027] According to the above technical solution, when the Gaussian parameters are obtained to render a new view, a simple one-to-one relationship is used between the 3D Gaussian and the pixel to predict the color for segmentation. In this case, the sputtering is degenerated into a volume rendering with a single depth-aware sampling point. Specifically, the volume density is obtained by decoding fv, and then the volume rendering is performed to obtain the rendered view.

[0028] In order to make the model have good generalization performance and show excellent performance in unseen scene reconstruction, a multi-view based geometric consistency strategy is introduced. Specifically, the predicted depth of the same 3D point in different viewpoints remains consistent. This geometric consistency can be measured by calculating the reprojection error between different views. Given a reference depth map D0 to be checked and a depth map D1 from a nearby viewpoint, first project the pixel p in D0 to a nearby view to obtain the projection point q. The calculation method is as follows:

[0029]

[0030] Among them, π 0-1 represents the transformation matrix from D0 to D1, d is the projected depth, and conversely, the obtained pixel q is back-projected with the estimated depth D1(q) onto the reference view to obtain the re-projected point p';

[0031]

[0032] Among them, π 0-1 represents the transformation matrix from D1 to D0, d' is the depth of the projection, and the calculation formula for its projection error can be:

[0033] ξ p =||p-p'||2,

[0034] The estimated depth is reliable when it has very low reprojection error in a few views or relatively small error in most views, which can be formulated as:

[0035] ξ p <θ p (n),ξ d <θ d (n);

[0036] Among them, θ p (n) and θ d (n) represents a predefined threshold whose value increases as the number of views n increases. When there are n nearby views that meet the corresponding threshold θ p (n) and θ d(n), the depth is reliable, the noise points that do not meet the conditions are filtered out, and the correct and reliable points are stored, so that the generated new viewpoints can effectively avoid noise interference.

[0037] According to the above technical solution, in order to obtain the geometric shape of the scene from the new view, in the process of rendering the new view, the rendered new view needs to keep the internal and external parameters of the original input image consistent, so that an image pair can be formed with the original input image, and then the pre-trained stereo matching model is used to obtain the depth of the image. In order to further enhance the generated reconstruction result, a mask is applied to the result generated by stereo matching;

[0038] The specific approach is: the relationship between the stereo matching errors can be described as:

[0039]

[0040] Where ∈(d) represents the disparity output error, Z is the true depth, ε(Z) is the error of depth estimation, fx represents the horizontal focal length of the camera, and B is the baseline;

[0041] Due to limitations of the matching algorithm, estimating the depth of objects that are too close to the camera may result in errors, while estimating the depth of objects that are farther away may result in quadratic errors. Considering depths in the range 4B≤Z≤20B improves the overall accuracy and reliability of the depth estimation process, ensuring a more consistent geometric reconstruction.

[0042] According to the above technical solution, in order to further enhance geometric consistency and eliminate noise and errors that may originate from each depth map, the truncated distance function (TSDF) algorithm is used to aggregate all extracted depths, and then the Marching-Cubes meshing algorithm is used to obtain the geometric shape of the scene;

[0043] The network model generated based on the above method was evaluated and measured on different datasets.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention combines deep learning technology, and through adaptive adjustment strategies and optimized parameter estimation methods, can further improve the performance of the Mesh extraction method based on the Gaussian model. It has important research significance in theory, especially in scenarios where large-scale, high-dimensional and complex data need to be processed. It has broad practical application prospects and aims to solve the deficiencies in the existing technology and improve the adaptability and efficiency in different data sets and application scenarios. It will provide an innovative solution for the field of three-dimensional data processing and promote the further development and application expansion of Mesh extraction technology.

[0046] 2. MVS is used to encode the geometrically-aware Gaussian representation and decode it into Gaussian parameters. A mixed Gaussian rendering method is proposed, which integrates an efficient volume rendering design for new view synthesis. The rendered new perspective should correspond to the camera pose of the training image to form a stereo aligned image pair, which is then input into the stereo matching model to obtain the depth map. Finally, all the depth maps are fused to obtain the Mesh. In order to support fast fine-tuning of specific scenes, a multi-view geometrically consistent aggregation strategy is introduced to effectively aggregate the point cloud generated by the generalizable model as the initialization for the optimization of each scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0048] In the attached picture:

[0049] Figure 1 It is a schematic diagram of the structure of the present invention;

[0050] Figure 2 It is a schematic diagram of the basic network process of the present invention;

[0051] Figure 3 It is a schematic diagram of the process of constructing the cost volume of the present invention;

[0052] Figure 4 is a schematic diagram of the process of synthesizing the new view of the present invention;

[0053] Figure 5 It is a schematic diagram of the process of the multi-view geometric consistency strategy of the present invention;

[0054] Figure 6 It is a schematic diagram of generating scene geometry through a stereo matching model in the present invention. DETAILED DESCRIPTION

[0055] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0056] Example: Figure 1 As shown, the present invention provides a technical solution, a generalizable fast Mesh extraction method based on Gaussian, comprising the following steps:

[0057] S1. Configure the training data collection system to label and preprocess the data;

[0058] S2. Build a generalizable fast Mesh extraction network based on Gaussian and train it;

[0059] S3, use the training model of the generalizable fast mesh extraction network based on Gaussian, test it in multiple data sets and different scenarios, and evaluate the results with indicators;

[0060] In S1, the training data collection system is configured, and the data is labeled and preprocessed, including: installing and configuring camera equipment to collect data, obtaining multi-view and surround images, capturing information of the training scene from different angles, and performing registration, correction, labeling and data enhancement processing on the collected data;

[0061] In S2, a generalizable fast mesh extraction network based on Gaussian is constructed, which consists of an FPN feature extraction module, a cost volume, a volume rendering module, and a stereo matching part;

[0062] In S3, the proposed method is extensively tested in wild scenes using smartphones, demonstrating its excellent reconstruction capability. In addition, the method is tested on the Tanks and Temples and DTU benchmarks and achieves state-of-the-art results.

[0063] Based on the above technical solution, in S2, specifically, the multi-view images captured by the acquisition device are used to extract features by using the FPN pyramid network to obtain 2D features of the input image, the 2D image features are projected to different depth plane spaces through homography transformation, the features of different depth planes are aggregated to form a 3D feature cost volume, the feature cost volume is regularized by 3D-CNN to generate a depth map, based on the depth map, feature encoding is performed on each pixel-aligned 3D point by aggregating multiple views and spatial information, and the encoded features are decoded for rendering.

[0064] Based on the above technical solutions, in S2, Gaussian expansion is an explicit region-based representation designed for tile-based rendering, involving complex many-to-many mappings between Gaussians and pixels, which poses a challenge to the generalization learning of generalizable learning. In order to solve this problem, an efficient hybrid rendering method is proposed, which integrates a simple depth-aware volume rendering module, in which each ray samples only one point, and uses a combination of Gaussian sputtering and volume rendering to render two views. Then, the two rendered views are averaged into the final view, which is built in a cascade structure to propagate the depth map and the rendered view in a coarse-to-fine manner;

[0065] Although very good new views can be obtained, it is still a challenge to extract the geometry of the scene directly from the above Gaussian attributes, because these attributes are optimized based on photometric loss. Although some concurrent models try to add geometric constraints in the Gaussian optimization process, they still produce noisy and unrealistic surfaces. Based on this, by injecting real-world priors into the depth extraction process, the gap between the noisy 3DGS method representation and the smooth 3D mesh representation is bridged. Compared with other methods, instead of directly extracting the geometry of the scene from the Gaussian attributes, the geometry is extracted through a pre-trained stereo matching model;

[0066] We render stereo-aligned image pairs corresponding to the original training poses, feed these into a stereo model to obtain depth profiles, and finally fuse all the profiles together to obtain a single mesh. The resulting reconstructions are smoother, more accurate, and show more intricate details than other surface reconstruction methods such as Gaussian sputtering, while incurring only a small overhead on top of a fairly short 3DGS optimization process.

[0067] Based on the above technical solutions, the emergence of the 3DGS method has made significant progress in the field of new viewpoint synthesis of images. This is a faster and more accurate scene capture method. 3DGS represents the scene as a point cloud of 3D Gaussians, where each Gaussian has the properties of opacity, rotation, scale, position and spherical harmonics. The scaling of Gaussians is anisotropic, which enables them to represent smaller and finer structures in the scene.

[0068] The Gaussian algorithm uses the Structure from Motion (SFM) algorithm to initialize the input image, which extracts the camera pose and provides an initial guess for the location of the Gaussian. Compared with other MLP-based methods, the 3DGS-based work has a short capture time and can be rendered in real time. Therefore, inspired by the 3D-GS work, a generalizable Mesh extraction method is generated using 3D-GS.

[0069] For data collection, it is required to shoot different scenes under different lighting conditions. The data collection system uses multiple sets of RGB-D camera arrays to cover the panoramic view around the object in front, behind, left front, left rear, right front and right rear. The obtained images need to be processed with Colmap software to obtain relevant parameters of the image.

[0070] Based on the above technical solution, in the overall network process, such as Figure 2As shown, we first need to build a cost volume based on the features of the input image, and infer the scene depth through 3D-CNN regularization. Given a set of input images, the 2D features of the image are obtained by extracting multi-scale features from the input view by the FPN pyramid feature extraction network. The depth map is an important part of our process, which connects the 2D image and 3D scene representation. Following the MVS-based method, we first establish multiple parallel planes on the target view, and then use the homography transformation to map the features of the source view to these parallel planes, as shown below:

[0071]

[0072] Among them, [K i ,R i ,t i ] and [R t ,K t ,t t ] are the intrinsic, rotation, and translation matrices of the source and target views, respectively, a represents the principal axis of the view camera, I represents the identity matrix, and Z represents the sampling depth;

[0073] Construct the cost volume C by calculating its variance;

[0074]

[0075] Where N represents the number of input views, V i Represents multi-scale features, Represents the average value of multi-scale features;

[0076] The obtained cost volume C is input into 3D-CNN for regularization to obtain the probability volume. Through the depth probability distribution, each depth hypothesis plane is weighted to obtain the final depth. The specific implementation process is as follows Figure 3 As shown, through the obtained depth, each pixel can be projected to a 3D point, i.e., the location of the 3D Gaussian, and then these 3D points are feature encoded to build a pixel-aligned 3D Gaussian representation.

[0077] Based on the above technical solution, in the process of feature encoding of 3D points, the features in the source view are first mapped to the target view using the homography transformation matrix, and then these multi-scale features are aggregated using the pooling network ρ.

[0078] Due to the characteristics of Gaussian, each Gaussian contributes to the pixel color value in a specific area of ​​the image. However, the aggregated features only encode the multi-view information of a single pixel and lack spatial perception. Therefore, a 2D-Unet network is used for spatial enhancement, and then the features are decoded to obtain Gaussian parameters for rendering.

[0079] Each Gaussian can be represented by {μ, s, r, α, c}. For the position μ, it can be obtained based on the estimated depth back-projection, where s, r, α, c represent the scale, rotation matrix, opacity and color respectively.

[0080] Based on the above technical solution, when obtaining Gaussian parameters to render a new view, the conventional method is to render the new view through Gaussian sputtering technology. However, the view obtained by this method lacks fine details, and the generalization performance of this method is very limited. Therefore, it is recommended to use a simple one-to-one relationship between 3D Gaussian and pixels to predict color for segmentation. Specifically, Figure 4 As shown, in this case, sputtering degenerates into volume rendering with a single depth-aware sampling point, specifically, the volume density is obtained by decoding the fv, and then volume rendering is performed to obtain the rendered view;

[0081] In order to make the model have good generalization performance and show excellent performance in reconstructing unseen scenes, a multi-view geometric consistency strategy is introduced. The specific implementation process is as follows: Figure 5 As shown, the predicted depth of the same 3D point at different viewpoints should be consistent, otherwise, the predicted depth is considered unreliable. This geometric consistency can be measured by calculating the reprojection error between different views, given a reference depth map D0 to be checked and a depth map D1 from a nearby viewpoint;

[0082] First, project the pixel p in D0 to a nearby view to get the projection point q. The calculation is as follows:

[0083]

[0084] Among them, π 0-1 represents the transformation matrix from D0 to D1, d is the projected depth, and conversely, the obtained pixel q is back-projected with the estimated depth D1(q) onto the reference view to obtain the re-projected point p';

[0085]

[0086] Among them, π 0-1 represents the transformation matrix from D1 to D0, d' is the depth of the projection, and the calculation formula for its projection error can be:

[0087] ξ p =||p-p'||2,

[0088] The estimated depth is reliable when it has very low reprojection error in a few views or relatively small error in most views, which can be formulated as:

[0089] ξ p <θ p (n),ξ d <θ d (n);

[0090] Among them, θ p (n) and θ d (n) represents a predefined threshold whose value increases as the number of views n increases. When there are n nearby views that meet the corresponding threshold θ p (n) and θ d (n), the depth is reliable, the noise points that do not meet the conditions are filtered out, and the correct and reliable points are stored, so that the generated new viewpoints can effectively avoid noise interference.

[0091] Based on the above technical solution, in order to obtain the geometric shape of the scene from the new view, in the process of rendering the new view, the rendered new view needs to keep the internal and external parameters of the original input image consistent, so that it can form an image pair with the original input image, and then use the pre-trained stereo matching model to obtain the depth of the image, such as Figure 6 As shown, in order to further enhance the generated reconstruction results, a mask is applied to the results generated by stereo matching;

[0092] The specific approach is: the relationship between the stereo matching errors can be described as:

[0093]

[0094] Where ∈(d) represents the disparity output error, Z is the true depth, ε(Z) is the error of depth estimation, fx represents the horizontal focal length of the camera, and B is the baseline;

[0095] On the contrary, if the disparity between the matching pixels in the two images of an object located at a shorter distance from the camera may exceed the maximum disparity limit produced by the stereo matching algorithm, therefore, estimating the depth of objects that are too close to the camera may result in errors due to the limitations of the matching algorithm, while estimating the depth of objects that are farther away may result in quadratic errors. Considering the depth in the range of 4B≤Z≤20B improves the overall accuracy and reliability of the depth estimation process and ensures a more consistent geometric reconstruction.

[0096] Based on the above technical solutions, in order to further enhance geometric consistency and eliminate noise and errors that may originate from individual depth maps, the truncated distance function (TSDF) algorithm is used to aggregate all extracted depths, and then the Marching-Cubes meshing algorithm is used to obtain the geometric shape of the scene;

[0097] Based on the network model generated by the above method, it was evaluated and measured on different data sets. The results show that compared with the most advanced Gaussian sputtering model surface reconstruction method, this method has superiority and good generalization performance. Compared with previous neural radiation field-related methods, it not only improves the accuracy, but also greatly shortens the calculation time.

[0098] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A generalizable fast mesh extraction method based on Gaussian, characterized by: The steps include: S1. Configure the training data collection system to label and preprocess the data; S2. Build a generalizable fast Mesh extraction network based on Gaussian and train it; S3, use the training model of the generalizable fast mesh extraction network based on Gaussian, test it in multiple data sets and different scenarios, and evaluate the results with indicators; In S1, configuring the training data acquisition system, annotating and preprocessing the data specifically includes: installing and configuring camera equipment to collect data, obtaining multi-view and surround images, capturing information of the training scene from different angles, and performing registration, correction, annotation and data enhancement processing on the collected data; In S2, a generalizable fast mesh extraction network based on Gaussian is constructed, and the network consists of an FPN feature extraction module, a cost volume, a volume rendering module and a stereo matching part; In S3, the proposed method is extensively tested in wild scenes using smartphones, demonstrating its excellent reconstruction capability. In addition, the method is tested on the Tanks and Temples and DTU benchmarks and achieves state-of-the-art results.

2. The Gaussian-based generalizable fast Mesh extraction method according to claim 1, characterized in that: Specifically, in S2, the multi-view images captured by the acquisition device are used to extract features by using the FPN pyramid network to obtain 2D features of the input image, the 2D image features are projected to different depth plane spaces through homography transformation, the features of different depth planes are aggregated to form a 3D feature cost volume, the feature cost volume is regularized by 3D-CNN to generate a depth map, based on the depth map, feature encoding is performed on each pixel-aligned 3D point by aggregating multiple views and spatial information, and the encoded features are decoded for rendering.

3. The Gaussian-based generalizable fast Mesh extraction method according to claim 2, characterized in that: In S2, a simple depth-aware volume rendering module is integrated, where each ray samples only one point, two views are rendered using a combination of Gaussian sputtering and volume rendering, and then the two rendered views are averaged into the final view, built in a cascade structure to propagate the depth map and rendered views in a coarse-to-fine manner; We bridge the gap between noisy 3DGS representation and smooth 3D mesh representation by injecting real-world priors into the depth extraction process, extracting geometry through a pre-trained stereo matching model, rendering stereo aligned image pairs corresponding to the original training poses, inputting these image pairs into the stereo model to obtain depth profiles, and finally fusion all profiles together to obtain a single mesh.

4. The Gaussian-based generalizable fast Mesh extraction method according to claim 3, characterized in that: The 3DGS represents the scene as a point cloud of 3D Gaussians, where each Gaussian has properties of opacity, rotation, scale, position, and spherical harmonics. The scaling of Gaussians is anisotropic, which enables them to represent smaller and finer structures in the scene. The Gaussian algorithm uses the Structure from Motion (SFM) algorithm to initialize the input image, which extracts the camera pose and provides an initial guess for the position of the Gaussian. 3DGS has a short capture time and can be rendered in real time. 3D-GS is used to generate a generalizable Mesh extraction method. For data collection, it is required to shoot different scenes under different lighting conditions. The data collection system uses multiple sets of RGB-D camera arrays to cover the panoramic view around the object in front, behind, left front, left rear, right front and right rear. The obtained images need to be processed with Colmap software to obtain relevant parameters of the image.

5. The Gaussian-based generalizable fast Mesh extraction method according to claim 4, characterized in that: In the overall network process, we first need to build a cost volume based on the features of the input image, and infer the scene depth through 3D-CNN regularization. Given a set of input images, the 2D features of the images are obtained by extracting multi-scale features from the input view using the FPN pyramid feature extraction network. Following the MVS-based method, we first establish multiple parallel planes on the target view, and then use the homography transformation to map the features of the source view to these parallel planes, as shown below: Among them, [K i ,R i ,t i ] and [R t ,K t ,t t ] are the intrinsic, rotation, and translation matrices of the source and target views, respectively, a represents the principal axis of the view camera, I represents the identity matrix, and Z represents the sampling depth; Construct the cost volume C by calculating its variance; Where N represents the number of input views, V i Represents multi-scale features, Represents the average value of multi-scale features; The obtained cost volume C is input into the 3D-CNN for regularization to obtain the probability volume. Each depth hypothesis plane is weighted by the depth probability distribution to obtain the final depth. Through the obtained depth, each pixel can be projected to a 3D point, that is, the position of the 3D Gaussian. These 3D points are then feature encoded to establish a pixel-aligned 3D Gaussian representation.

6. The Gaussian-based generalizable fast Mesh extraction method according to claim 4, characterized in that: In the process of feature encoding of 3D points, the features in the source view are first mapped to the target view using the homography transformation matrix, and then these multi-scale features are aggregated using the pooling network ρ. Due to the characteristics of Gaussian, each Gaussian contributes to the pixel color value in a specific area of ​​the image. The 2D-Unet network is used for spatial enhancement, and then the features are decoded to obtain the Gaussian parameters for rendering. Each Gaussian can be represented by {μ, s, r, α, c}. For the position μ, it can be obtained based on the estimated depth back projection, where s, r, α, and c represent the scaling, rotation matrix, opacity, and color, respectively.

7. The Gaussian-based generalizable fast Mesh extraction method according to claim 4, characterized in that: When rendering a new view with Gaussian parameters, a simple one-to-one relationship between 3D Gaussian and pixels is used to predict the color for segmentation. In this case, sputtering degenerates into volume rendering with a single depth-aware sampling point. Specifically, the volume density is obtained by decoding fv, and then volume rendering is performed to obtain the rendered view. In order to make the model have good generalization performance and show excellent performance in unseen scene reconstruction, a multi-view based geometric consistency strategy is introduced. Specifically, the predicted depth of the same 3D point in different viewpoints remains consistent. This geometric consistency can be measured by calculating the reprojection error between different views. Given a reference depth map D0 to be checked and a depth map D1 from a nearby viewpoint, first project the pixel p in D0 to a nearby view to obtain the projection point q. The calculation method is as follows: Among them, π 0-1 represents the transformation matrix from D0 to D1, d is the projected depth, and conversely, the obtained pixel q is back-projected with the estimated depth D1(q) onto the reference view to obtain the re-projected point p'; Among them, π 0-1 represents the transformation matrix from D1 to D0, d' is the depth of the projection, and the calculation formula for its projection error can be: x p =||p-p'||2, The estimated depth is reliable when it has very low reprojection error in a few views or relatively small error in most views, which can be formulated as: x p <θ p (n),ξ d <θ d (n); Among them, θ p (n) and θ d (n) represents a predefined threshold whose value increases as the number of views n increases. When there are n nearby views that meet the corresponding threshold θ p (n) and θ d (n), the depth is reliable, the noise points that do not meet the conditions are filtered out, and the correct and reliable points are stored, so that the generated new viewpoints can effectively avoid noise interference.

8. The Gaussian-based generalizable fast Mesh extraction method according to claim 4, characterized in that: In order to obtain the geometry of the scene from the new view, the new view needs to keep the original input image’s internal and external parameters consistent during the rendering process, so that it can form an image pair with the original input image. The depth of the image is then obtained using the pre-trained stereo matching model. To further enhance the generated reconstruction result, a mask is applied to the result generated by stereo matching. The specific approach is: the relationship between the stereo matching errors can be described as: Where ∈(d) represents the disparity output error, Z is the true depth, ε(Z) is the error of depth estimation, fx represents the horizontal focal length of the camera, and B is the baseline; Due to limitations of the matching algorithm, estimating the depth of objects that are too close to the camera may result in errors, while estimating the depth of objects that are farther away may result in quadratic errors. Considering depths in the range 4B≤Z≤20B improves the overall accuracy and reliability of the depth estimation process, ensuring a more consistent geometric reconstruction.

9. The Gaussian-based generalizable fast Mesh extraction method according to claim 4, characterized in that: To further enhance geometric consistency and eliminate noise and errors that may originate from individual depth maps, all extracted depths are aggregated using the truncated distance function (TSDF) algorithm, and then the Marching-Cubes meshing algorithm is used to obtain the geometry of the scene; The network model generated based on the above method was evaluated and measured on different datasets.

Images