A three-dimensional Gaussian point cloud mesh construction method and system based on multimodal embedding

Abstract

The present application discloses a method and system for constructing a three-dimensional Gaussian point cloud mesh based on multimodal embedding, the method comprising: obtaining three-dimensional Gaussian point cloud data and performing sparse sampling and adaptive sampling processing to obtain a point cloud embedding vector; introducing an attention mechanism to perform Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a mesh embedding vector; based on the point cloud embedding vector and the mesh embedding vector, performing Transformer network training and autoregressive sampling to generate a three-dimensional Gaussian point cloud mesh. The embodiments of the present application can fully capture the global structure and local detail features of the point cloud, improve the compactness of the mesh generated by autoregressive sampling, and thereby improve the quality of the generated image or data. The present application can be widely used in the field of three-dimensional point cloud reconstruction technology.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional point cloud reconstruction, and in particular to a three-dimensional Gaussian point cloud mesh construction method and system based on multimodal embedding. Background Art

[0002] In the field of computer vision and graphics, 3D Gaussian point cloud is an efficient three-dimensional data representation method with fast training speed and the ability to generate high-quality rendering effects. Specifically, by introducing Gaussian distribution parameters, it can accurately describe the location and density information of points and is widely used in scenes such as scene reconstruction, object recognition, and virtual reality. Mesh is a common three-dimensional representation in downstream fields. It provides the topological structure and surface details of objects, supports complex geometric operations and rendering tasks, and plays a key role in game development, film and television special effects, and industrial design.

[0003] However, it should be noted that 3D Gaussian point cloud data is inherently disordered and lacks clear structural information, making it difficult to generate high-quality meshes directly from point clouds. In addition, the uneven density distribution of Gaussian point clouds greatly increases the difficulty of mesh generation. The point cloud mesh generation algorithm in related technologies usually relies on isosurface sampling of density functions. The mesh generated by this method is often too dense and smooth, thereby losing important global information and fine details in the original Gaussian point cloud. It only uses the point cloud as a modality and fails to effectively capture and learn the information of each mesh facet, resulting in the generated mesh being deficient in compactness and detail expression, which reduces the visual effect of the generated mesh.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a three-dimensional Gaussian point cloud mesh construction method and system based on multimodal embedding, which can comprehensively capture the global structure and local detail features of the point cloud, improve the compactness of the autoregressive sampling grid generation, and thus improve the quality of the generated image or data.

[0006] To achieve the above purpose, an embodiment of the present application proposes a method for constructing a three-dimensional Gaussian point cloud mesh based on multimodal embedding, the method comprising:

[0007] Obtain three-dimensional Gaussian point cloud data and perform sparse sampling and adaptive sampling processing to obtain the point cloud embedding vector;

[0008] An attention mechanism is introduced to perform Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a grid embedding vector;

[0009] Based on the point cloud embedding vector and the grid embedding vector, Transformer network training and autoregressive sampling are performed to generate a three-dimensional Gaussian point cloud grid.

[0010] In some embodiments, the step of acquiring three-dimensional Gaussian point cloud data and performing sparse sampling and adaptive sampling processing to obtain a point cloud embedding vector includes:

[0011] Acquire the three-dimensional Gaussian point cloud data;

[0012] Sparsely sample the three-dimensional Gaussian point cloud data using a voxel grid downsampling algorithm to obtain sparsely sampled point cloud data;

[0013] Adaptively sampling the three-dimensional Gaussian point cloud data by using an adaptive sampling algorithm to obtain adaptive sampling point cloud data;

[0014] Based on a multi-layer perceptron module, vector embedding is performed on the sparse sampling point cloud data and the adaptive sampling point cloud data to obtain the point cloud embedding vector.

[0015] In some embodiments, sparse sampling of the three-dimensional Gaussian point cloud data by a voxel grid downsampling algorithm to obtain sparse sampled point cloud data includes:

[0016] Determine the voxel side length according to the distribution of the three-dimensional Gaussian point cloud data;

[0017] Dividing the three-dimensional space into voxel grids according to the voxel side length and mapping the three-dimensional Gaussian point cloud data to obtain three-dimensional Gaussian point cloud coordinate data;

[0018] Constructing a voxel list according to the three-dimensional Gaussian point cloud coordinate data;

[0019] The voxel list is subjected to a center point calculation and selection combination process to obtain the sparse sampling point cloud data.

[0020] In some embodiments, the adaptive sampling of the three-dimensional Gaussian point cloud data by an adaptive sampling algorithm to obtain the adaptive sampling point cloud data includes:

[0021] Acquire local density data of the three-dimensional Gaussian point cloud data by using a K-nearest neighbor algorithm and set a preset sampling density threshold;

[0022] The sampling rate is determined according to the local density data and the three-dimensional Gaussian point cloud data is adaptively sampled in combination with the preset sampling density threshold to obtain adaptively sampled point cloud data.

[0023] In some embodiments, the multi-layer perceptron module is used to perform vector embedding on the sparse sampling point cloud data and the adaptive sampling point cloud data to obtain the point cloud embedding vector, including:

[0024] The sparse sampling point cloud data and the adaptive sampling point cloud data are respectively encoded by a point cloud encoder to obtain a sparse sampling point cloud embedding vector and an adaptive sampling point cloud embedding vector;

[0025] Concatenating the sparse sampling point cloud embedding vector with the adaptive sampling point cloud embedding vector to obtain a concatenated point cloud embedding vector;

[0026] Based on the multi-layer perceptron module, feature extraction and application embedding are performed on the spliced ​​point cloud embedding vector to obtain the point cloud embedding vector.

[0027] In some embodiments, the introducing of the attention mechanism to perform Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a grid embedding vector includes:

[0028] Performing Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a Poisson reconstructed grid;

[0029] Discretize and represent the Poisson-reconstructed meshes in sequence to obtain triangular face sequence data;

[0030] The attention mechanism is introduced to construct an improved three-dimensional generative model MeshGPT, wherein the improved three-dimensional generative model MeshGPT includes a graph convolution encoder, an attention mechanism and a ResNet decoder;

[0031] The triangular face sequence data is input into the improved three-dimensional generative model MeshGPT for reconstruction to obtain the mesh embedding vector.

[0032] In some embodiments, inputting the triangular face sequence data into the improved three-dimensional generative model MeshGPT for reconstruction to obtain the mesh embedding vector includes:

[0033] Based on the graph convolution encoder of the improved three-dimensional generative model MeshGPT, feature extraction processing is performed on the triangular face sequence data to obtain a face encoding feature vector;

[0034] Based on the attention mechanism of the improved three-dimensional generative model MeshGPT, weight information is generated according to the triangular face sequence data;

[0035] Based on the ResNet decoder of the improved three-dimensional generative model MeshGPT, the encoded feature vector of the surface is decoded in combination with the weight information to obtain the mesh embedding vector.

[0036] In some embodiments, the performing of Transformer network training and autoregressive sampling based on the point cloud embedding vector and the mesh embedding vector to generate a three-dimensional Gaussian point cloud mesh includes:

[0037] Concatenating the point cloud embedding vector with the grid embedding vector to obtain a multimodal embedding concatenated vector;

[0038] Training the Transformer network using the multimodal embedding concatenation vector to obtain a trained Transformer network;

[0039] Autoregressive sampling is performed on the trained Transformer network to generate the three-dimensional Gaussian point cloud grid.

[0040] In some embodiments, the performing autoregressive sampling on the trained Transformer network to generate the three-dimensional Gaussian point cloud grid includes:

[0041] Inputting the point cloud embedding vector into the trained Transformer network, and generating a plurality of embeddings through autoregression of the trained Transformer network at each time step to obtain embedding sequence data;

[0042] The embedded sequence data is decoded by a decoder to obtain the three-dimensional Gaussian point cloud grid.

[0043] To achieve the above object, another aspect of the embodiment of the present application proposes a three-dimensional Gaussian point cloud mesh construction system based on multimodal embedding, the system comprising:

[0044] The first module is used to obtain three-dimensional Gaussian point cloud data and perform sparse sampling and adaptive sampling processing to obtain point cloud embedding vectors;

[0045] The second module is used to introduce an attention mechanism to perform Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a grid embedding vector;

[0046] The third module is used to perform Transformer network training and autoregressive sampling based on the point cloud embedding vector and the grid embedding vector to generate a three-dimensional Gaussian point cloud grid.

[0047] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for constructing a three-dimensional Gaussian point cloud mesh based on multimodal embedding. The scheme obtains three-dimensional Gaussian point cloud data and performs sparse sampling and adaptive sampling processing to obtain a point cloud embedding vector, implements sparse sampling and adaptive sampling strategies on the 3D Gaussian point cloud, and encodes the sampling results into a point cloud embedding vector to comprehensively capture the global structure and local detail features of the point cloud. The attention mechanism is further introduced to perform Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a mesh embedding vector, and embedding learning is performed on the mesh extracted by the Poisson reconstruction to obtain structured information at the mesh level. Finally, based on the point cloud embedding vector and the mesh embedding vector, Transformer network training and autoregressive sampling are performed to generate a three-dimensional Gaussian point cloud mesh. The point cloud embedding and the Poisson reconstructed mesh embedding jointly guide the training of the Transformer network, so that the model makes full use of the complementary information between different modalities, thereby improving the compactness and quality of the mesh generated by autoregressive sampling, capturing more details, improving the quality of the generated image or data, and making the visual effect more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of a method for constructing a three-dimensional Gaussian point cloud mesh based on multimodal embedding provided in an embodiment of the present application;

[0049] Figure 2 It is a structural schematic diagram of a three-dimensional Gaussian point cloud mesh construction system based on multimodal embedding provided in an embodiment of the present application;

[0050] Figure 3 It is a schematic diagram of three-dimensional Gaussian point cloud mesh reconstruction provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0052] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0053] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] Reference Figure 1 , Figure 1 A flowchart of a method for constructing a three-dimensional Gaussian point cloud mesh based on multimodal embedding provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0056] S100, acquiring three-dimensional Gaussian point cloud data and performing sparse sampling and adaptive sampling processing to obtain a point cloud embedding vector;

[0057] It should be noted that, in some embodiments, step S100 may include: S110, obtaining three-dimensional Gaussian point cloud data; S120, sparsely sampling the three-dimensional Gaussian point cloud data through a voxel grid downsampling algorithm to obtain sparsely sampled point cloud data; S130, adaptively sampling the three-dimensional Gaussian point cloud data through an adaptive sampling algorithm to obtain adaptively sampled point cloud data; S140, based on a multi-layer perceptron module, vector embedding the sparsely sampled point cloud data and the adaptively sampled point cloud data to obtain a point cloud embedding vector.

[0058] In some specific embodiments, the 3D Gaussian point cloud The space is divided into voxels, and sparse sampling is performed using the voxel grid downsampling algorithm to finally obtain a sparsely sampled point cloud. Adaptive sampling dynamically adjusts the sampling rate according to the local density and Gaussian distribution parameters of the point cloud, reducing the sampling points in areas with low density and uniform distribution, and increasing the sampling points in areas with high density or large curvature changes, and finally obtaining an adaptively sampled point cloud. .

[0059] Further, it should be noted that step S120 may include: S121, determining the voxel side length according to the distribution of the three-dimensional Gaussian point cloud data; S122, dividing the three-dimensional space into voxel grids according to the voxel side length and mapping the three-dimensional Gaussian point cloud data to obtain the three-dimensional Gaussian point cloud coordinate data; S123, constructing a voxel list according to the three-dimensional Gaussian point cloud coordinate data; S124, performing a center point calculation and selection combination processing on the voxel list to obtain sparse sampling point cloud data.

[0060] In some specific embodiments, the basic idea of ​​the voxel grid downsampling algorithm is to divide the three-dimensional space into voxels of uniform size and then retain a certain number of points in each voxel.

[0061] The specific steps are as follows:

[0062] 1) Set the voxel size:

[0063] Determine the side length of the voxel (voxel_size) and choose an appropriate voxel size based on the distribution of the point cloud.

[0064] 2) Constructing the voxel grid:

[0065] The 3D space is divided into a voxel grid according to the voxel size, and the point cloud is mapped into the voxel grid using its coordinates.

[0066] 3) Point attribution and statistics:

[0067] According to the coordinates of each point, calculate the voxel it is in and add it to the corresponding voxel list.

[0068] 4) Selection of sampling points:

[0069] For each voxel, one or more points are selected as representative points. Common methods include calculating the center point of all points in the voxel and randomly selecting points in the voxel.

[0070] 5) Generate sparse sampling point cloud:

[0071] The representative points selected in all voxels are combined to form a sparsely sampled point cloud.

[0072] Further, it should be noted that step S130 may include: S131, obtaining local density data of three-dimensional Gaussian point cloud data through the K-nearest neighbor algorithm and setting a preset sampling density threshold; S132, determining the sampling rate according to the local density data and adaptively sampling the three-dimensional Gaussian point cloud data in combination with the preset sampling density threshold to obtain adaptively sampled point cloud data.

[0073] In some specific embodiments:

[0074] 1) Calculate local density:

[0075] The local density of each point is calculated by the K-nearest neighbor algorithm, and the neighborhood points of each point can be accessed by using, for example, a kD tree.

[0076] 2) Set the sampling strategy:

[0077] Set density thresholds and sampling scales, such as increasing the number of sampling points in high-density areas and reducing them in low-density areas.

[0078] 3) Perform adaptive sampling:

[0079] Traverse the point cloud and determine the sampling rate based on the local density. Increase the number of samples in areas with high density or large curvature; reduce the number of samples in areas with low density and uniform distribution.

[0080] 4) Generate adaptive sampling point cloud:

[0081] The final adaptive sparse point cloud is obtained by applying the filtered sampling strategy.

[0082] Further, it should be noted that step S140 may include: S141, encoding the sparse sampling point cloud data and the adaptive sampling point cloud data respectively through a point cloud encoder to obtain a sparse sampling point cloud embedding vector and an adaptive sampling point cloud embedding vector; S142, splicing the sparse sampling point cloud embedding vector and the adaptive sampling point cloud embedding vector to obtain a spliced ​​point cloud embedding vector; S143, based on a multi-layer perceptron module, performing feature extraction and application embedding on the spliced ​​point cloud embedding vector to obtain a point cloud embedding vector.

[0083] In some specific embodiments, a point cloud encoder is used to convert the sparsely sampled point cloud and adaptive sampling point cloud Encoded as an embedding vector and ,Will and Concatenate to form a new vector , and input it into the multi-layer perceptron to finally obtain the embedding vector of the sampled point cloud , whose expression is as follows:

[0084] ;

[0085] ;

[0086] In the above formula, represents the concatenated point cloud embedding vector, represents the embedding vector of the sparsely sampled point cloud, represents the adaptive sampling point cloud embedding vector, represents the point cloud embedding vector, Represents a multilayer perceptron operation.

[0087] S200, introduce the attention mechanism, perform Poisson reconstruction on the three-dimensional Gaussian point cloud data, and obtain the grid embedding vector;

[0088] It should be noted that, in some embodiments, step S200 may include: S210, performing Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a Poisson reconstructed mesh; S220, discretizing and processing the Poisson reconstructed mesh in sequence to obtain triangular surface sequence data; S230, introducing an attention mechanism to construct an improved three-dimensional generative model MeshGPT, the improved three-dimensional generative model MeshGPT including a graph convolutional encoder, an attention mechanism and a ResNet decoder; S240, inputting the triangular surface sequence data into the improved three-dimensional generative model MeshGPT for reconstruction to obtain a mesh embedding vector.

[0089] Further, it should be noted that step S240 may include: S241, a graph convolutional encoder based on an improved three-dimensional generation model MeshGPT, performing feature extraction processing on the triangular face sequence data to obtain a face encoding feature vector; S242, an attention mechanism based on the improved three-dimensional generation model MeshGPT, generating weight information according to the triangular face sequence data; S243, a ResNet decoder based on the improved three-dimensional generation model MeshGPT, decoding the face encoding feature vector in combination with the weight information to obtain a mesh embedding vector.

[0090] In some specific embodiments, the mesh embedding part of Poisson reconstruction is an improvement of MeshGPT. The embedding process is shown in the figure. The process is as follows: first, the Poisson reconstruction mesh is discretized and input as a triangle face sequence. , the encoding part uses a graph convolutional encoder Encoded mesh surface To extract the feature vector of the face , learn the codebook through the residual quantization module .

[0091] Here, the attention mechanism is introduced to improve the 1D ResNet of the MeshGPT decoding part. The output of each time step will adjust its weight according to the entire sequence information that has been generated, thereby improving the decoder's ability to recognize and process key features. The formula is as follows:

[0092] ;

[0093] ;

[0094] In the above formula, The encoding vector representing each face includes the nine position encoding coordinates of each face vertex, the face normal, the angle between edges, and the area. express The number of faces, represents the features of the face extracted by the encoder, represents the introduced attention mechanism, Represents the decoder pair using the attention mechanism The decoded reconstructed mesh.

[0095] In addition, it should be noted that Poisson Surface Reconstruction is a commonly used point cloud processing method that converts dense point clouds into smooth three-dimensional surfaces. Combined with the MeshGPT method, the mesh embedding process encodes and extracts features from triangular meshes through a series of steps.

[0096] S300, based on the point cloud embedding vector and the mesh embedding vector, perform Transformer network training and autoregressive sampling to generate a three-dimensional Gaussian point cloud mesh;

[0097] It should be noted that, in some embodiments, step S300 may include: S310, splicing the point cloud embedding vector and the grid embedding vector to obtain a multimodal embedding splicing vector; S320, training the Transformer network through the multimodal embedding splicing vector to obtain a trained Transformer network; S330, performing autoregressive sampling on the trained Transformer network to generate a three-dimensional Gaussian point cloud grid.

[0098] In some specific embodiments, the point cloud is embedded into a vector and the grid embedding vector concatenate to form a comprehensive multimodal embedding representation , whose expression is:

[0099] ;

[0100] In the above formula, represents the multimodal embedding concatenation vector, represents the point cloud embedding vector, represents the grid embedding vector.

[0101] Embedding multimodal The embedded features are transmitted through the multi-head self-attention layer inside the Transformer, and the codebook The next token prediction loss generated in is used to train the Transformer network, and finally the next embedded token index in the sequence is predicted.

[0102] Further, it should be noted that step S330 may include: S331, inputting the point cloud embedding vector into the trained Transformer network, and based on each time step, generating a number of embeddings through autoregression of the trained Transformer network to obtain embedded sequence data; S332, decoding the embedded sequence data through a decoder to obtain a three-dimensional Gaussian point cloud mesh.

[0103] In some specific embodiments, the final trained Transformer network can generate a grid by autoregressive sampling: only embedding the point cloud into vectors The input is input into the trained Transformer network. At each time step, the next embedding is generated through autoregression of the Transformer network and the current embedding state is updated. Finally, the decoder is used to decode the generated embedding sequence into the final mesh, realizing the complete process of generating a mesh from a 3D Gaussian point cloud.

[0104] In summary, in practical engineering applications, Figure 3 As shown in the figure, after using 3D Gaussian Splatting to generate a 3D Gaussian point cloud scene representation from a picture or video, Poisson reconstruction is used to generate a mesh from the 3D Gaussian point cloud. The embedding vector of the 3D Gaussian point cloud and the mesh embedding vector of the Poisson reconstruction jointly guide the training of the Transformer network. After a large amount of multimodal embedding guides the training of the Transformer network, it is able to learn the geometric and topological features of different 3D Gaussian scenes, and the generative relationship between point clouds and meshes. Finally, when a new 3D Gaussian point cloud scene is input, it is possible to generate a compact mesh using only the autoregressive sampling of the decoding part of the Transformer network.

[0105] The generated mesh can be bound to 3D Gaussian points and use Gaussian scattering rendering technology to achieve higher rendering quality than traditional mesh-based rendering methods. In addition, by converting 3D Gaussian point clouds into meshes, the bound meshes can be modified and edited using traditional mesh editing tools, bringing value to the development of downstream fields such as animation production.

[0106] See also Figure 2 The embodiment of the present application further provides a three-dimensional Gaussian point cloud mesh construction system based on multimodal embedding, which can implement the above-mentioned three-dimensional Gaussian point cloud mesh construction method based on multimodal embedding, and the system includes:

[0107] The first module 201 is used to obtain three-dimensional Gaussian point cloud data and perform sparse sampling and adaptive sampling processing to obtain a point cloud embedding vector;

[0108] The second module 202 is used to introduce an attention mechanism to perform Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a grid embedding vector;

[0109] The third module 203 is used to perform Transformer network training and autoregressive sampling based on the point cloud embedding vector and the grid embedding vector to generate a three-dimensional Gaussian point cloud grid.

[0110] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A three-dimensional Gaussian point cloud mesh construction method based on multimodal embedding, characterized in that: The method comprises the following steps: Acquire the three-dimensional Gaussian point cloud data; Sparsely sample the three-dimensional Gaussian point cloud data using a voxel grid downsampling algorithm to obtain sparsely sampled point cloud data; Adaptively sampling the three-dimensional Gaussian point cloud data by using an adaptive sampling algorithm to obtain adaptive sampling point cloud data; Based on a multi-layer perceptron module, vector embedding is performed on the sparse sampling point cloud data and the adaptive sampling point cloud data to obtain a point cloud embedding vector; Performing Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a Poisson reconstructed grid; Discretize and represent the Poisson-reconstructed meshes in sequence to obtain triangular face sequence data; Introducing the attention mechanism, constructing an improved three-dimensional generative model MeshGPT, wherein the improved three-dimensional generative model MeshGPT includes a graph convolution encoder, an attention mechanism and a ResNet decoder; Inputting the triangular face sequence data into the improved three-dimensional generative model MeshGPT for reconstruction to obtain a mesh embedding vector; Concatenating the point cloud embedding vector with the grid embedding vector to obtain a multimodal embedding concatenated vector; Training a Transformer network using the multimodal embedding concatenation vector to obtain a trained Transformer network; Autoregressive sampling is performed on the trained Transformer network to generate a three-dimensional Gaussian point cloud grid.

2. The method according to claim 1, characterized in that The sparse sampling of the three-dimensional Gaussian point cloud data by a voxel grid downsampling algorithm to obtain sparse sampled point cloud data includes: Determine the voxel side length according to the distribution of the three-dimensional Gaussian point cloud data; Dividing the three-dimensional space into voxel grids according to the voxel side length and mapping the three-dimensional Gaussian point cloud data to obtain three-dimensional Gaussian point cloud coordinate data; Constructing a voxel list according to the three-dimensional Gaussian point cloud coordinate data; The voxel list is subjected to a center point calculation and selection combination process to obtain the sparse sampling point cloud data.

3. The method according to claim 1, characterized in that The method of adaptively sampling the three-dimensional Gaussian point cloud data by an adaptive sampling algorithm to obtain adaptive sampling point cloud data includes: Acquire local density data of the three-dimensional Gaussian point cloud data by using a K-nearest neighbor algorithm and set a preset sampling density threshold; The sampling rate is determined according to the local density data and the three-dimensional Gaussian point cloud data is adaptively sampled in combination with the preset sampling density threshold to obtain adaptively sampled point cloud data.

4. The method according to claim 1, characterized in that: The method of performing vector embedding on the sparse sampling point cloud data and the adaptive sampling point cloud data based on the multi-layer perceptron module to obtain a point cloud embedding vector includes: The sparse sampling point cloud data and the adaptive sampling point cloud data are respectively encoded by a point cloud encoder to obtain a sparse sampling point cloud embedding vector and an adaptive sampling point cloud embedding vector; Concatenating the sparse sampling point cloud embedding vector with the adaptive sampling point cloud embedding vector to obtain a concatenated point cloud embedding vector; Based on the multi-layer perceptron module, feature extraction and application embedding are performed on the spliced ​​point cloud embedding vector to obtain a point cloud embedding vector.

5. The method according to claim 1, characterized in that The step of inputting the triangular face sequence data into the improved three-dimensional generative model MeshGPT for reconstruction to obtain a mesh embedding vector comprises: Based on the graph convolution encoder of the improved three-dimensional generative model MeshGPT, feature extraction processing is performed on the triangular face sequence data to obtain a face encoding feature vector; Based on the attention mechanism of the improved three-dimensional generative model MeshGPT, weight information is generated according to the triangular face sequence data; Based on the ResNet decoder of the improved three-dimensional generative model MeshGPT, the encoded feature vector of the surface is decoded in combination with the weight information to obtain the mesh embedding vector.

6. The method according to claim 5, characterized in that The step of performing autoregressive sampling on the trained Transformer network to generate the three-dimensional Gaussian point cloud grid includes: Inputting the point cloud embedding vector into the trained Transformer network, and generating a plurality of embeddings through autoregression of the trained Transformer network at each time step to obtain embedding sequence data; The embedded sequence data is decoded by a decoder to obtain the three-dimensional Gaussian point cloud grid.

7. A three-dimensional Gaussian point cloud mesh construction system based on multimodal embedding, characterized in that: The system comprises: The first module is used to obtain the three-dimensional Gaussian point cloud data; Sparsely sample the three-dimensional Gaussian point cloud data using a voxel grid downsampling algorithm to obtain sparsely sampled point cloud data; Adaptively sampling the three-dimensional Gaussian point cloud data by using an adaptive sampling algorithm to obtain adaptive sampling point cloud data; Based on a multi-layer perceptron module, vector embedding is performed on the sparse sampling point cloud data and the adaptive sampling point cloud data to obtain a point cloud embedding vector; The second module is used to perform Poisson reconstruction on the three-dimensional Gaussian point cloud data to obtain a Poisson reconstructed grid; Discretize and represent the Poisson-reconstructed meshes in sequence to obtain triangular face sequence data; Introducing the attention mechanism, constructing an improved three-dimensional generative model MeshGPT, wherein the improved three-dimensional generative model MeshGPT includes a graph convolution encoder, an attention mechanism and a ResNet decoder; Inputting the triangular face sequence data into the improved three-dimensional generative model MeshGPT for reconstruction to obtain a mesh embedding vector; A third module is used to splice the point cloud embedding vector with the grid embedding vector to obtain a multimodal embedding splicing vector; Training a Transformer network using the multimodal embedding concatenation vector to obtain a trained Transformer network; Autoregressive sampling is performed on the trained Transformer network to generate a three-dimensional Gaussian point cloud grid.

Images