A Point Cloud to Mesh Generation Method and System Based on Autoregressive Model and Reinforcement Learning Optimization

By using an autoregressive model and reinforcement learning optimization method, the problems of redundant face count and topological defects in point cloud to mesh generation are solved, generating high-quality 3D mesh models suitable for game engines, animation rigging, physical simulation and 3D printing.

CN123118958APending Publication Date: 2026-11-17FUDAN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611113606.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-27
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing point cloud to mesh generation methods suffer from problems such as redundant face count, topological defects, high computational complexity, and invalid generation results, making it difficult to generate high-quality 3D mesh models.

Method used

We employ an autoregressive model and reinforcement learning optimization method to divide the 3D point cloud into local mesh blocks. We extract geometric details and global shape semantics through a point cloud encoder, perform geometric alignment by combining a cross-attention mechanism, and introduce watertightness, manifold constraints, and geometric consistency rewards to construct a composite geometric reward function for optimization.

Benefits of technology

It significantly improves the topological quality and geometric fit accuracy of the generated mesh, reduces computational costs, and the generated mesh is suitable for game engines, animation rigging, physics simulation, and 3D printing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118958A_ABST
    Figure CN123118958A_ABST
Patent Text Reader

Abstract

This invention relates to a point cloud-to-mesh generation method and system based on autoregressive models and reinforcement learning optimization. The method includes: dividing the 3D space corresponding to a 3D point cloud into multiple local mesh blocks; extracting point cloud feature sequences from the 3D point cloud; inputting the point cloud feature sequences into an autoregressive generation model for processing, and predicting the mesh features of each local mesh block based on the point cloud feature sequences to reconstruct a 3D mesh model; during model training: for each input point cloud feature sequence, generating a set of mesh feature sequences, and generating a parameter-frozen reference model based on the autoregressive generation model, calculating compound geometric rewards for each mesh feature sequence; calculating KL divergence based on the outputs of the autoregressive generation model and the reference model to construct an overall objective function to guide the parameter updates of the model. Compared with existing technologies, this invention has advantages such as low resource consumption, high stability, and high-quality generated mesh topology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of mesh generation technology, and in particular to a point cloud to mesh generation method and system based on autoregressive models and reinforcement learning optimization. Background Technology

[0002] 3D content generation technology has enormous application demands in fields such as game development, film production, virtual reality, and industrial manufacturing. Among all 3D data representations, polygonal meshes have become the most universal and valuable 3D asset standard in industry due to their explicit topological structure, extremely high rendering efficiency, and native support for physical simulation and skeletal rigging. In the entire 3D generation pipeline, converting disordered, discrete point cloud data into high-quality continuous mesh models has always been a hot research topic and a challenging task.

[0003] Currently, methods for generating point clouds into meshes can be mainly categorized as follows, but all have certain limitations: 1. Traditional Geometric Reconstruction and Isosurface Extraction Methods: Early point cloud to mesh conversion relied primarily on traditional geometric algorithms, such as Poisson surface reconstruction or rolling sphere methods. These methods depend on local geometric priors and are prone to generating large-area holes and incorrect topological connections when dealing with noisy, sparse, or occluded point clouds. With the development of deep learning, methods based on neural implicit fields and 3D Gaussians have gradually emerged. However, these methods are essentially continuous spatial fields or unstructured representations, ultimately requiring contour surface extraction techniques such as Marching Cubes to derive the mesh. This method has two main problems. First, it results in redundant facets and messy wiring, often producing results with obvious over-voxelation staircase artifacts. Not only is the number of facets enormous, but the topological wiring also deviates significantly from the norms of manual modeling. Second, topological defects are frequent, easily generating non-manifold edges, self-intersecting surfaces, and isolated geometric fragments during extraction. This makes the model difficult to directly apply to industrial engines such as Unity and UE5, and also hinders subsequent editing and optimization.

[0004] 2. Existing Autoregressive Mesh Generation Methods: Recent research has attempted to discretize the vertex and face information of 3D meshes into tokens and use autoregressive models to generate mesh sequences end-to-end. This type of method can directly output explicit meshes, avoiding some defects in the post-processing extraction process. Although this feedforward network method can directly output meshes, due to the complexity of 3D data, existing autoregressive generation methods still face the following serious bottlenecks, failing to meet the needs of high-quality generation. Their drawbacks are: existing mesh tokenization schemes typically encode absolute vertex coordinates and connectivity relationships directly. Due to the complexity of 3D mesh structures, this easily leads to excessively long input sequences, causing the computational complexity of the autoregressive model to increase quadratically during processing. This not only makes it difficult to capture global geometric dependencies but also severely limits the generation capability of high-resolution, fine meshes. Furthermore, existing generation algorithms ignore the global physical properties that a mesh, as a 3D entity, must possess, such as watertightness and manifold structure. The difficulty of long-range modeling, coupled with the lack of explicit global topological constraints, causes the model to frequently output invalid topological indices when predicting complex curvature regions. As a result, the generated mesh is prone to serious "geometric illusion" phenomena such as local holes, surface fragmentation, and self-intersection. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a point cloud to mesh generation method and system based on autoregressive model and reinforcement learning optimization, so as to realize the reconstruction of high-fidelity 3D mesh with industrial-grade topological quality based on sparse point cloud.

[0006] The objective of this invention can be achieved through the following technical solutions: A point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization includes the following steps: Obtain the 3D point cloud to be processed and divide the corresponding 3D space into multiple local mesh blocks; The absolute coordinates of each vertex in the 3D point cloud are converted into the offset index of its local mesh block, and the local geometric details and global shape semantics of the 3D point cloud are extracted by a point cloud encoder to output the point cloud feature sequence. The point cloud feature sequence and the corresponding offset index are input into a pre-constructed and trained autoregressive generation model for processing. The processing of the autoregressive generation model includes: based on the offset index, autoregressively predicting the mesh features of each local mesh block according to the point cloud feature sequence, and geometrically aligning the point cloud feature sequence and the mesh feature sequence through a cross-attention mechanism during the autoregressive prediction process; and restoring the mesh feature sequence output by the autoregressive prediction into a three-dimensional mesh model through a dequantization and decoding process. The training process of the autoregressive generative model includes: for each input point cloud feature sequence, the autoregressive generative model generates a set of different grid feature sequences, and generates a parameter-frozen reference model based on the autoregressive generative model; the composite geometric reward is calculated for each grid feature sequence in the current group to calculate the relative advantage of each grid feature sequence within the group; the KL divergence is calculated based on the output of the autoregressive generative model and the reference model; and the overall objective function is constructed by combining the relative advantage within the group and the calculation results of the KL divergence to guide the parameter update of the autoregressive generative model.

[0007] Furthermore, the method further includes the following steps before outputting the point cloud feature sequence: Based on the offset index of each vertex in the corresponding local mesh block in the 3D point cloud, the vertices are reordered based on the adjacency of the local mesh block, and only the offset index of the first occurrence of vertices repeatedly shared by adjacent patches is retained. The final vertex set is obtained based on the retained offset index to construct the point cloud feature sequence.

[0008] Furthermore, the cross-attention mechanism uses point cloud features as keys and values, and grid features as queries, to perform geometric alignment between the point cloud feature sequence and the grid feature sequence.

[0009] Furthermore, the composite geometric reward includes a watertightness reward, a manifold constraint reward, and a geometric consistency reward. The watertightness reward is used to systematically traverse the boundary structure of the generated 3D mesh model, retrieve the number of unpaired boundary edges, thereby penalizing non-watertight structures and encouraging the generation of closed surfaces. The manifold constraint reward is used to analyze the connection relationship between vertices and edges in the generated 3D mesh model, and to give negative feedback to structures that violate manifold constraints in order to suppress the generation of non-manifold topologies; The geometric consistency reward is used to calculate the distance metric between the sampled points on the mesh surface of the generated 3D mesh model and the input point cloud.

[0010] Furthermore, the watertightness reward The calculation expression is: In the formula, This represents the total number of unpaired boundary edges retrieved by the system during traversal. The preset boundary penalty weight coefficient; The structures that violate manifold constraints include non-manifold edges and non-manifold vertices. A non-manifold edge is an edge that is shared by three or more triangular facets at the same time. A non-manifold vertex is a vertex whose adjacent facets cannot form a single closed sector or open disk. manifold constraint reward The calculation expression is: In the formula, This represents the total number of non-manifold edges present in the grid. This represents the total number of non-manifold vertices. and These are the preset penalty coefficients for the corresponding topological defects.

[0011] Furthermore, the expression for calculating the relative advantage within the group is as follows: In the formula, The relative advantage within the group for the i-th grid feature sequence. The composite geometric reward value is the i-th grid feature sequence within the group. The composite geometric reward value of the grid feature sequence within the group The average value, The composite geometric reward value of the grid feature sequence within the group The standard deviation.

[0012] Furthermore, the calculation expression for the overall objective function is as follows: In the formula, To generate model parameters in autoregressive model The corresponding overall objective function is as follows: This represents the number of grid feature sequences within the current group. For the first i A grid feature sequence, To generate model parameters in autoregressive model Next i The importance sampling ratio of grid feature t in a grid feature sequence The relative advantage within the group for the i-th grid feature sequence. To truncate parameters, Let KL divergence penalty coefficient be denoted as . This represents the KL divergence between the autoregressive generative model and the reference model.

[0013] Furthermore, the method is used to convert 3D point clouds in the fields of game engines, animation rigging, physical simulation, or 3D printing into 3D mesh models.

[0014] This invention also provides a point cloud to mesh generation system based on an autoregressive model and reinforcement learning optimization method, as described above, comprising: An autoregressive mesh generation module based on point cloud conditions is used to acquire the 3D point cloud to be processed, divide the corresponding 3D space into multiple local mesh blocks, convert the absolute coordinates of each vertex in the 3D point cloud into the offset index of its local mesh block, and use a point cloud encoder to extract the local geometric details and global shape semantics of the 3D point cloud, outputting a point cloud feature sequence. The point cloud feature sequence and the corresponding offset index are input into a pre-constructed and trained autoregressive generation model for processing. The processing of the autoregressive generation model includes: based on the offset index, autoregressively predicting the mesh features of each local mesh block according to the point cloud feature sequence, and using a cross-attention mechanism to geometrically align the point cloud feature sequence and the mesh feature sequence during the autoregressive prediction process; and restoring the mesh feature sequence output by the autoregressive prediction into a 3D mesh model through dequantization and decoding processes. Topological quality post-optimization based on reinforcement learning is used to train the autoregressive generative model. The corresponding training process includes: for each input point cloud feature sequence, the autoregressive generative model generates a set of different grid feature sequences, and generates a parameter-frozen reference model based on the autoregressive generative model; the composite geometric reward is calculated for each grid feature sequence in the current group to calculate the relative advantage of each grid feature sequence within the group; the KL divergence is calculated based on the outputs of the autoregressive generative model and the reference model; and the overall objective function is constructed by combining the relative advantage within the group and the KL divergence calculation results to guide the parameter update of the autoregressive generative model.

[0015] The present invention also provides a point cloud to mesh generation system based on autoregressive model and reinforcement learning optimization, including a memory and a processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of the method described above.

[0016] Compared with the prior art, the present invention has the following advantages: (1) A progressive optimization mechanism is constructed that connects autoregressive generation with reinforcement learning-based post-training optimization: Unlike existing point cloud to mesh conversion methods that typically rely on a single supervised learning objective for end-to-end training, this invention adopts a phased optimization strategy: First, supervised learning is used to enable the model to master the basic generation capability from point clouds to explicit meshes; on this basis, for higher-order quality attributes such as watertightness, manifold, and overall topological rationality, which are difficult to characterize directly by the supervised objective, reinforcement learning is used to explicitly inject non-differentiable geometric constraints into the model optimization process, thereby balancing generation capability and geometric legality. This approach avoids the training instability problem caused by directly introducing complex topological constraints in the initial training stage, and realizes a progressive modeling process of learning generation first and then optimizing quality. As a result, the model can maintain good geometric reconstruction capability while further improving the topological quality and overall usability of the generated mesh.

[0017] (2) Significantly shortens the length of the mesh sequence, reducing training and inference costs: Existing autoregressive mesh generation typically directly encodes absolute vertex coordinates and connectivity, resulting in excessively long token sequences that severely increase the difficulty and computational burden of Transformer modeling. To address this issue, this invention does not employ traditional VQ-VAE-style discrete encoding. Instead, it converts absolute coordinates into relative offset indices within local mesh blocks and combines local face traversal, vertex rearrangement, and adjacent index deduplication to reduce spatial and topological redundancy in the sequence from the source. This achieves a sequence compression rate of approximately 72% while preserving geometric details. This effect helps improve training throughput, reduce memory usage, and support the generation of meshes with higher resolution and more complex topologies, while maintaining the reversible recovery of geometric details.

[0018] (3) Improve the geometric fit accuracy of point cloud to mesh: Unlike existing autoregressive methods that rely solely on the context of a single mesh sequence for prediction, this invention introduces a point cloud encoder to extract local geometric and global semantic features of the input point cloud. It also utilizes a cross-attention mechanism to continuously query prior information of the point cloud during the progressive generation of the mesh token, achieving a dynamic correspondence between the generated position and the original observed geometry. This not only leverages global shape semantics but also achieves more accurate geometric alignment in local detail areas. This mechanism allows the model to move beyond relying solely on sequence statistical patterns and instead be constrained by the input point cloud at each prediction step. The generated results of this invention exhibit higher surface fit and smaller shape offset in edges, curvature variation areas, and areas with fine structures, thus better restoring the true geometric contour of the input point cloud.

[0019] (4) Significantly improves the topological legitimacy and engineering usability of the mesh: Existing supervised learning objectives typically only optimize the prediction probability of the next token, failing to directly characterize the overall geometric quality of the mesh. This invention transforms watertightness, manifold structure, and geometric consistency into computable reward terms, forming a composite geometric reward function: it constrains closed surfaces through boundary edge detection, suppresses non-manifold topology through vertex / edge connection analysis, and improves shape fit through distance measurement between mesh surface sampling points and the input point cloud. This explicitly incorporates mesh quality constraints, which are traditionally difficult to differentiate directly, into the training objective, expanding the optimization objective from merely predicting correct sequences to predicting geometrically more usable and legitimate meshes. This effectively reduces problems such as unpaired boundary edges, non-manifold connections, self-intersecting surfaces, and local holes, thereby improving the closure, continuity, and topological correctness of the output mesh. Compared to existing generation schemes that rely solely on supervised learning, the mesh output by this invention is more suitable for direct application in game engines, animation rigging, physical simulation, 3D printing, and downstream geometric editing processes, reducing the workload of manual model repair, hole patching, retopology, and debris cleanup in the later stages.

[0020] (5) Reduce resource consumption in the post-optimization stage and improve training stability: Unlike existing point cloud to mesh generation methods that mainly rely on supervised learning and are difficult to directly optimize non-differentiable topological indicators such as watertightness and manifold, this invention does not propose a new reinforcement learning algorithm, but innovatively applies the Group Relative Policy Optimization (GRPO) mechanism to the point cloud to mesh generation task. In this process, multiple candidate mesh results are generated for the same input point cloud, and relative rewards within the group are compared based on quality indicators such as watertightness, manifold, and geometric consistency, thereby achieving posterior optimization of the generation policy; at the same time, the policy update magnitude is constrained by the reference model to suppress policy drift during training. By using GRPO for explicit mesh quality improvement, this invention enables reinforcement learning to effectively serve the topology correction and quality optimization problems in 3D mesh generation.

[0021] Furthermore, compared to traditional Actor-Critic type reinforcement learning methods, the proposed solution does not require additional training of the value network, resulting in a simpler network structure, smaller parameter size, and a more stable post-training process. By incorporating the KL divergence constraint of the reference model, catastrophic forgetting or excessive distribution shifts can be avoided when the model pursues local rewards, thereby improving the geometric quality of the output mesh while maintaining its original generative capabilities.

[0022] (6) Reduce reliance on manual rule repair and complex post-processing: Traditional implicit field / isosurface extraction schemes often require additional post-processing steps such as smoothing, hole filling, self-intersection removal, fragmentation removal, and reconstruction of normal consistency; this invention, through explicit mesh direct generation and reinforcement learning topology correction, moves some of the work that originally relied on manual experience and post-processing software to the model itself. This reduces manual intervention, shortens the 3D asset production chain, and improves the degree of automation, making it particularly suitable for large-scale 3D asset generation scenarios. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of a point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of an autoregressive mesh generation process based on point cloud conditions provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a reinforcement learning optimization process based on GRPO provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0026] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0027] Example 1 like Figure 1 As shown, this embodiment provides a point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization, which is divided into the following two stages: Phase 1: Autoregressive Mesh Generation Based on Point Cloud Conditions The mesh reconstruction task is refactored into a sequence prediction problem in a discrete latent space, utilizing Transformer to learn geometric topological dependencies. Specifically, this stage abandons the traditional VQ-VAE scheme and innovatively introduces a mesh segmentation algorithm based on hierarchical block offset coordinate quantization and local face traversal, effectively removing spatial redundancy and achieving a sequence compression rate of approximately 72%. During the generation process, a point cloud encoder is used to extract multi-scale geometric features, and the cross-attention mechanism in the Decoder-only Transformer is used to dynamically align the point cloud features with the mesh sequence geometrically, ultimately dequantizing the autoregressive predicted discrete tokens into a high-quality 3D mesh.

[0028] Phase Two: Post-Optimization of Topology Quality Based on Reinforcement Learning The GRPO algorithm is introduced to transform non-differentiable geometric constraints into reward signals, explicitly correcting illegal mesh structures. This stage constructs a composite reward system including watertightness detection (penalizing unpaired boundary edges), manifold constraints (suppressing illegal vertex / edge connections), and geometric consistency (surface fit measurement). Simultaneously, a parameter-frozen reference model is introduced for distribution consistency constraints to prevent catastrophic forgetting. The GRPO algorithm calculates the advantage function through intra-group relative comparisons of inputs from the same source, completely avoiding the training of an additional value network, significantly reducing computational and memory overhead, and effectively solving the geometric illusions such as local holes and self-intersections that are prone to occur in autoregressive models.

[0029] Specifically, the following steps are included: S1: Obtain the 3D point cloud to be processed and divide the corresponding 3D space into multiple local mesh blocks; S2: As Figure 2 As shown, the absolute coordinates of each vertex in the 3D point cloud are converted into the offset index of the local mesh block in which it is located, and the point cloud encoder is used to extract the local geometric details and global shape semantics of the 3D point cloud, and output the point cloud feature sequence. S3: Input the point cloud feature sequence and the corresponding offset index into the pre-built and trained autoregressive generation model for processing. The processing of the autoregressive generation model includes: based on the offset index, autoregressively predicting the mesh features of each local mesh block according to the point cloud feature sequence, and geometrically aligning the point cloud feature sequence and the mesh feature sequence through a cross-attention mechanism during the autoregressive prediction process; and restoring the mesh feature sequence output by the autoregressive prediction into a 3D mesh model through dequantization and decoding processes. S4: As Figure 3 As shown, the training process of the autoregressive generative model includes: for each input point cloud feature sequence, the autoregressive generative model generates a set of different grid feature sequences, and generates a parameter-frozen reference model based on the autoregressive generative model. The composite geometric reward is calculated for each grid feature sequence in the current group to calculate the relative advantage of each grid feature sequence within the group. The KL divergence is calculated based on the output of the autoregressive generative model and the reference model. The overall objective function is constructed by combining the relative advantage within the group and the calculation results of the KL divergence to guide the parameter update of the autoregressive generative model.

[0030] Preferably, step S2 further includes the following before outputting the point cloud feature sequence: Based on the offset index of each vertex in the corresponding local mesh block in the 3D point cloud, the vertices are reordered based on the adjacency of the local mesh blocks, and vertices expressed by adjacent offset indices in the same local mesh block are merged to obtain the final vertex set, so as to construct the point cloud feature sequence.

[0031] Step S2 includes the following aspects: a) Hierarchical block offset coordinate quantization: Divide the 3D space into local mesh blocks and convert the absolute coordinates of vertices into offset indices relative to their local mesh blocks.

[0032] b) Local face traversal and deduplication. A local face traversal mechanism is used to reorder vertices based on the adjacency of the mesh topology and merge the index representations of adjacent vertices. This method effectively removes spatial redundancy in the sequence, achieving a sequence compression rate of approximately 72% while preserving accurate geometric details, significantly reducing the computational complexity of long sequence modeling.

[0033] The merging of adjacent vertex indices here does not refer to forcibly merging different vertices in space into a single vertex. The underlying logic of this scheme is lossless spatial sequence compression. In 3D mesh data, since multiple adjacent faces share the same vertices, traditional serialization schemes would cause the coordinates and indices of the same vertex to be repeatedly written in the token sequence, resulting in severe sequence redundancy.

[0034] This scheme first converts absolute coordinates into relative offset indices within local blocks through hierarchical block offset coordinate quantization. Then, it utilizes a local face traversal mechanism to ensure that adjacent faces sharing the same vertex are arranged consecutively in the generated sequence. At this point, for the same vertex repeatedly shared by adjacent faces, this invention only retains its first occurrence index token, or merges its redundant index representations in adjacent sequences. In summary, this step essentially involves deduplication and compression of redundant markers in topological connectivity, rather than fuzzing or simplifying the merging of different spatial vertices.

[0035] c) Point cloud feature encoding. A jointly trained point cloud encoder is used to extract local geometric details and global shape semantics from the input sparse point cloud, and output a point cloud feature sequence as a conditional prior for subsequent autoregressive generation.

[0036] The cross-attention mechanism in step S3 uses point cloud features as keys and values, and grid features as queries, to perform geometric alignment between the point cloud feature sequence and the grid feature sequence.

[0037] Step S3 includes the following aspects: The autoregressive generative model employs a Decoder-only Transformer architecture, which includes stacked self-attention layers and cross-attention layers.

[0038] a) Geometric alignment query: The cross-attention mechanism serves as a geometric alignment module connecting the point cloud representation and the mesh generation process, using the encoded point cloud features as keys and values, and the mesh sequence features as queries.

[0039] b) Autoregressive sequence prediction: When the model makes an autoregressive prediction of each grid token (i.e., the quantized vertex offset or topological index), it dynamically queries the corresponding geometric region in the point cloud and learns the topological dependency relationship between grid vertices and faces.

[0040] The guidance process here is as follows: A point cloud encoder transforms the input 3D point cloud into a continuous vector containing multi-scale geometric features and global shape semantics. In the cross-attention layer of the autoregressive large model, these point cloud geometric features are used as keys and values. Simultaneously, the currently predicted grid token sequence features from the autoregressive model are used as queries. By calculating adaptive attention weights between the query and the key, the model can dynamically query and focus on the corresponding 3D spatial geometric region in the original point cloud when predicting each grid token (i.e., the quantized vertex offset or topological index) in the autoregressive model. This geometric alignment mechanism explicitly imposes constraints on the spatial distribution of the point cloud on the generation of the grid topology sequence, thereby guiding the model to output a grid topology that highly conforms to the surface of the input point cloud.

[0041] c) Dequantization and Decoding: The discrete token sequence output by the model is directly restored into a high-quality 3D mesh model with complete topology and smooth surface through a deterministic dequantization and decoding process.

[0042] The discrete token sequence output by the autoregressive model consists of alternating quantized vertex offset tokens and topological patch index tokens. The decoding and reconstruction process specifically includes: First, dequantizing the vertex offset tokens in the sequence by multiplying their discrete index values ​​by a preset quantization step size and superimposing them onto the center reference coordinates of the local mesh block where they reside, thereby losslessly reconstructing the 3D absolute spatial coordinates [x, y, z] of each vertex and constructing the vertex set of the 3D mesh; Second, parsing the topological patch index tokens in the sequence, and according to the local face traversal rules, combining every three adjacent vertex indices into a triangular patch definition, constructing a topological face set representing the vertex connection relationship; Finally, combining and encapsulating the reconstructed vertex set and topological face set to directly reconstruct a topologically complete, smooth, high-quality 3D mesh model.

[0043] In step S4, since the optimization objective of the autoregressive generative model under supervised learning mainly focuses on maximizing the conditional likelihood of the next label, it lacks the ability to directly model the overall geometric structure and global topological properties, and is prone to generating meshes with problems such as non-manifold structures, local holes, or self-intersections. This module introduces the group GRPO reinforcement learning algorithm for post-training, transforming non-differentiable geometric constraints into continuous reward signals.

[0044] a) Compound geometric reward calculation: The core driving force for model optimization comes from the carefully designed compound geometric reward function r, which includes constraints in three dimensions: 1. Watertightness Reward: By systematically traversing the mesh boundary structure and detecting the number of unpaired boundary edges, a penalty is imposed on non-watertight structures, thereby encouraging the generation of closed surfaces.

[0045] Specifically, by globally traversing all edges of the generated 3D mesh model, the number of adjacent triangular faces for each edge is counted. In a standard polygonal mesh topology, any internal edge of a watertight closed surface should be shared by exactly two triangular faces; if an edge is adjacent to only one triangular face, that edge is defined as an unpaired boundary edge. This scheme constructs a watertightness reward... The specific calculation expression is as follows: In the formula, This represents the total number of unpaired boundary edges retrieved by the system during traversal. The default boundary penalty weight coefficient is set (e.g., 0.5). Through this negative feedback penalty mechanism, the policy model can adaptively reduce the probability of outputting non-closed topological sequences during post-training, explicitly guiding the model to fill in local holes, thereby ensuring the generation of a closed surface with good watertightness.

[0046] 2. Manifold Constraint Reward: Analyze the connectivity between vertices and edges, and provide negative feedback to structures that violate manifold constraints to suppress the generation of non-manifold topologies.

[0047] Specifically, by analyzing the legality of connections between vertices and edges in the generated 3D mesh model, non-manifold edges and vertices are detected and negative feedback is provided. A non-manifold edge is an edge shared by three or more triangular faces; a non-manifold vertex is a vertex whose adjacent faces cannot form a single closed sector or open disk. This invention constructs a manifold constraint reward... The specific expression is as follows: In the formula, This represents the total number of non-manifold edges present in the grid. This represents the total number of non-manifold vertices. and These are the preset penalty coefficients for the corresponding topological defects. By calculating the connection relationship between edges and vertices and implementing scalarized penalties, this reward term successfully introduces the traditional non-differentiable manifold geometric constraints into the objective function of reinforcement learning, thereby fundamentally suppressing the generation of non-manifold cluttered topologies and spatially self-intersecting patches.

[0048] 3. Geometric Consistency Bonus: Calculate the distance metric between the sampled points on the generated mesh surface and the input point cloud to ensure that the optimized mesh closely fits the original geometric contour in terms of overall shape.

[0049] b) Value-free architecture: The GPRO algorithm adopts a group-relative strategy optimization method, which completely avoids the introduction of value networks, greatly reduces memory usage and computational complexity, and makes reinforcement learning on large-scale 3D generative models highly scalable.

[0050] c) Intra-group relative advantage and objective function: For each input point cloud condition The strategy model generates a set of different grid sequence outputs. By normalizing individual samples using the statistical characteristics of within-group rewards, the relative advantage is calculated as follows: At the same time, a parameter-frozen model is introduced as a reference distribution. (i.e., the initial model without reinforcement learning training), the overall objective function The KL divergence is incorporated to limit the offset magnitude and avoid catastrophic forgetting, as shown in the following formula: In the formula, To generate model parameters in autoregressive model The corresponding overall objective function is as follows: This represents the number of grid feature sequences within the current group. For the first i A grid feature sequence, To generate model parameters in autoregressive model Next i The importance sampling ratio of grid feature t in a grid feature sequence , The relative advantage within the group for the i-th grid feature sequence. To truncate parameters, Let KL divergence penalty coefficient be denoted as . This represents the KL divergence between the autoregressive generative model and the reference model.

[0051] The above method can be used in fields such as game engines, animation rigging, physical simulation, and 3D printing to convert 3D point clouds into 3D mesh models.

[0052] This embodiment also provides a point cloud to mesh generation system based on the point cloud to mesh generation method optimized by autoregressive model and reinforcement learning, as described above, including: The autoregressive mesh generation module based on point cloud conditions is used to acquire the 3D point cloud to be processed, divide the corresponding 3D space into multiple local mesh blocks, convert the absolute coordinates of each vertex in the 3D point cloud into the offset index of its local mesh block, and use a point cloud encoder to extract the local geometric details and global shape semantics of the 3D point cloud, outputting a point cloud feature sequence. The point cloud feature sequence and the corresponding offset index are input into a pre-constructed and trained autoregressive generation model for processing. The processing of the autoregressive generation model includes: based on the offset index, autoregressively predicting the mesh features of each local mesh block according to the point cloud feature sequence, and using a cross-attention mechanism to geometrically align the point cloud feature sequence and the mesh feature sequence during the autoregressive prediction process; and restoring the mesh feature sequence output by the autoregressive prediction into a 3D mesh model through dequantization and decoding. Topological quality post-optimization based on reinforcement learning is used to train an autoregressive generative model. The corresponding training process includes: for each input point cloud feature sequence, the autoregressive generative model generates a set of different grid feature sequences, and a reference model with frozen parameters is generated based on the autoregressive generative model. The composite geometric reward is calculated for each grid feature sequence in the current group to calculate the relative advantage of each grid feature sequence within the group. The KL divergence is calculated based on the outputs of the autoregressive generative model and the reference model. The overall objective function is constructed by combining the relative advantage within the group and the KL divergence calculation results to guide the parameter update of the autoregressive generative model.

[0053] This embodiment also provides a point cloud to mesh generation system based on autoregressive model and reinforcement learning optimization, including a memory and a processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of the point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization as described above.

[0054] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization, characterized in that, Includes the following steps: Obtain the 3D point cloud to be processed, and divide the corresponding 3D space into multiple local mesh blocks; The absolute coordinates of each vertex in the 3D point cloud are converted into the offset index of its local mesh block, and the local geometric details and global shape semantics of the 3D point cloud are extracted by a point cloud encoder to output the point cloud feature sequence. The point cloud feature sequence and the corresponding offset index are input into a pre-constructed and trained autoregressive generation model for processing. The processing of the autoregressive generation model includes: based on the offset index, autoregressively predicting the mesh features of each local mesh block according to the point cloud feature sequence, and geometrically aligning the point cloud feature sequence and the mesh feature sequence through a cross-attention mechanism during the autoregressive prediction process; and restoring the mesh feature sequence output by the autoregressive prediction into a three-dimensional mesh model through a dequantization and decoding process. The training process of the autoregressive generative model includes: for each input point cloud feature sequence, the autoregressive generative model generates a set of different grid feature sequences, and generates a parameter-frozen reference model based on the autoregressive generative model; the composite geometric reward is calculated for each grid feature sequence in the current group to calculate the relative advantage of each grid feature sequence within the group; the KL divergence is calculated based on the output of the autoregressive generative model and the reference model; and the overall objective function is constructed by combining the relative advantage within the group and the calculation results of the KL divergence to guide the parameter update of the autoregressive generative model.

2. The point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization according to claim 1, characterized in that, The method further includes the following before outputting the point cloud feature sequence: Based on the offset index of each vertex in the corresponding local mesh block in the 3D point cloud, the vertices are reordered based on the adjacency of the local mesh block, and only the offset index of the first occurrence of vertices repeatedly shared by adjacent patches is retained. The final vertex set is obtained based on the retained offset index to construct the point cloud feature sequence.

3. The point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization according to claim 1, characterized in that, The cross-attention mechanism uses point cloud features as keys and values, and grid features as queries, to perform geometric alignment between the point cloud feature sequence and the grid feature sequence.

4. The point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization according to claim 1, characterized in that, The composite geometric reward includes a watertightness reward, a manifold constraint reward, and a geometric consistency reward. The watertightness reward is used to systematically traverse the boundary structure of the generated 3D mesh model, retrieve the number of unpaired boundary edges, thereby penalizing non-watertight structures and encouraging the generation of closed surfaces. The manifold constraint reward is used to analyze the connection relationship between vertices and edges in the generated 3D mesh model, and to give negative feedback to structures that violate manifold constraints in order to suppress the generation of non-manifold topologies; The geometric consistency reward is used to calculate the distance metric between the sampled points on the mesh surface of the generated 3D mesh model and the input point cloud.

5. The point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization according to claim 4, characterized in that, The watertightness reward The computational expression is: In the formula, is the total number of the retrieved unpaired boundary edges traversed by the system, is a preset boundary penalty weight coefficient; The structures that violate manifold constraints include non-manifold edges and non-manifold vertices. A non-manifold edge is an edge that is shared by three or more triangular facets at the same time. A non-manifold vertex is a vertex whose adjacent facets cannot form a single closed sector or open disk. The manifold constraint reward The computational expression is: wherein is the total number of non-manifold edges present in the mesh, is the total number of non-manifold vertices, and are preset penalty coefficients corresponding to the topological defects, respectively.

6. The method of claim 1, wherein, The formula for calculating the relative advantage within the group is: wherein is the in-group relative advantage of the i-th grid feature sequence within the group, is the composite geometric reward value of the i-th grid feature sequence within the group, is the composite geometric reward value of the grid feature sequence within the group is the average of the composite geometric reward values of the grid feature sequence within the group, is the standard deviation of the composite geometric reward values of the grid feature sequence within the group is the standard deviation of the composite geometric reward values of the grid feature sequence within the group 7. The method of claim 1, wherein, The overall objective function is calculated as follows: In the formula, To generate model parameters in autoregressive model The corresponding overall objective function is as follows: This represents the number of grid feature sequences within the current group. For the first i A grid feature sequence, To generate model parameters in autoregressive model Next i The importance sampling ratio of grid feature t in a grid feature sequence The relative advantage within the group for the i-th grid feature sequence. To truncate parameters, Let KL divergence penalty coefficient be denoted as . This represents the KL divergence between the autoregressive generative model and the reference model.

8. The point cloud to mesh generation method based on autoregressive model and reinforcement learning optimization according to claim 1, characterized in that, The method is used to convert 3D point clouds into 3D mesh models in the fields of game engines, animation rigging, physical simulation, or 3D printing.

9. A point cloud to mesh generation system based on an autoregressive model and reinforcement learning optimization method as described in any one of claims 1-8, characterized in that, include: An autoregressive mesh generation module based on point cloud conditions is used to acquire the 3D point cloud to be processed, divide the corresponding 3D space into multiple local mesh blocks, convert the absolute coordinates of each vertex in the 3D point cloud into the offset index of its local mesh block, and use a point cloud encoder to extract the local geometric details and global shape semantics of the 3D point cloud, outputting a point cloud feature sequence. The point cloud feature sequence and the corresponding offset index are input into a pre-constructed and trained autoregressive generation model for processing. The processing of the autoregressive generation model includes: based on the offset index, autoregressively predicting the mesh features of each local mesh block according to the point cloud feature sequence, and using a cross-attention mechanism to geometrically align the point cloud feature sequence and the mesh feature sequence during the autoregressive prediction process; and restoring the mesh feature sequence output by the autoregressive prediction into a 3D mesh model through dequantization and decoding processes. Topological quality post-optimization based on reinforcement learning is used to train the autoregressive generative model. The corresponding training process includes: for each input point cloud feature sequence, the autoregressive generative model generates a set of different grid feature sequences, and generates a parameter-frozen reference model based on the autoregressive generative model; the composite geometric reward is calculated for each grid feature sequence in the current group to calculate the relative advantage of each grid feature sequence within the group; the KL divergence is calculated based on the outputs of the autoregressive generative model and the reference model; and the overall objective function is constructed by combining the relative advantage within the group and the KL divergence calculation results to guide the parameter update of the autoregressive generative model.

10. A point cloud to mesh generation system based on autoregressive model and reinforcement learning optimization, characterized in that, It includes a memory and a processor, the memory storing a computer program, the processor invoking the computer program to perform the steps of the method as described in any one of claims 1 to 8.